Ofqual wants to make sure that public debate about our report on this year’s GCSE English awarding is well informed. On this page, we set out and discuss some of the claims that have been made in recent days.

“There has been a big drop in the numbers getting top grades this year”

At national level, there was a small drop (1.5 percentage points) in the proportion of UK students gaining GCSE English at grades A*-C in 2012 compared to 2011. As explained in our report, this fall can be explained by changes in the profile of students who took GCSE English in 2012. Had the cohort been identical in 2012 to that in 2011, our approach to securing standards (known as ‘comparable outcomes’) would have produced virtually identical results.

Some schools have seen results in 2012 that are lower than expected and in some cases lower than in previous years. Given that national results remain the same, other schools and colleges will have seen increases in their results, as happens every year. The evidence in our report demonstrates that students awarded GCSE English in June 2012 got the results they deserve.

“Ofqual should have acted in January, when the unit awards were too generous”

Our report sets out how we reviewed January awarding. Our conclusion is that the right processes were followed and the outcomes were reasonable given the data available at the time.

There was not sufficient evidence when the January awards were made for Ofqual or the exam boards to conclude that the unit awards were generous. Indeed, as set out in our report, the data at the time suggested that the boundaries on the AQA foundation tier paper should have been even lower than they were.

This demonstrates one of the challenges with awarding graded modular qualifications. We will be moving away from a modular system at GCSE in England after the 2012/13 school year.

“The goalposts were moved mid-year”

Grade boundaries are not goalposts. They are not something to aim at. They are set after an assessment has been taken to reflect the demands of the assessment, using the professional judgement of examiners, checks against standards in previous assessments, and data on candidate performance and prior attainment. Sometimes grade boundaries will be set differently for different cohorts taking similar assessments, because the demands of the assessments are different or because more evidence becomes available about the performance of candidates.

It was regrettable that the publication of grade boundaries for the January assessments could have led schools to assume that the boundaries would remain constant, and we will review with the exam boards any lessons from this.

“Ofqual could solve the problem easily by regrading June assessments using January boundaries”

If we were to do this, there would be a big increase in the proportion of candidates getting grades A*-C at English which would not be justified by the evidence. There would be a serious risk of undermining confidence in these qualifications: given the publicity the issue has had, employers and others may be wary of trusting any grades achieved by GCSE English candidates in 2012. It would not be fair to candidates in other years, because the standard would have been set at a different level from other years. It would also not be in line with Ofqual’s statutory standards objective, set by Parliament.

We do not believe that these disadvantages would be acceptable given the issue we face, which is that a relatively small number of students benefitted from generous grading in the January 2012 assessments.

“The results are not fair”

Those students who took the assessments in June 2012 can be confident that the grades they were awarded were right.

It is true that some of those students may have got better grades had they taken the assessments in January 2012. That is because, in retrospect, it is clear that the January awards (to a relatively small number of students) were generous.

It has been suggested that we should revisit the January 2012 awards, so that they were in line with the June awards. Taking away grades that had already been awarded would have been unfair. Students and teachers made decisions about resits for the summer based on these outcomes and may have made different choices if they had been given lower grades. Candidates from January may have wanted to re-sit units if they’d known their results were lower; regrading their units in July meant they were too late to re-sit.

If we were to revisit the June 2012 awards and bring them into line with the generous January standards, it would compound the unfairness. It would mean that, rather than just the relatively small number of January awards benefitting from generous grading, the entire 2012 cohort of students would benefit. That would not be fair to students in past years or future years. Nor would it be fair to the employers, colleges or universities, who will be looking to use these qualifications to make recruitment and selection decisions.

“Ofqual knew in 2009 that there would be problems with awarding modular GCSEs and did nothing about it”

It is no secret that awarding graded modular qualifications presents some technical challenges. When modular GCSEs were introduced, it was Ofqual’s responsibility, as the then qualifications regulator, to consider and decide how to meet those challenges, drawing on the advice of assessment experts.

As part of considering what approach to take, Ofqual’s former Chief Executive, Isabel Nisbet, gave a presentation to a conference in 2009 discussing some of the challenges and presenting some options for meeting them (see http://www.cambridgeassessment.org.uk/ca/digitalAssets/182118_isabel-nisbet-seminar.pdf and the accompanying paper http://www.ofqual.gov.uk/files/the-ping-factor.pdf). Ms Nisbet’s paper does not present a preferred approach or suggest that there is an easy answer. In particular, contrary to some media reports, it does not imply that the option of not disclosing any unit outcomes until the qualification is complete is an ideal solution – indeed, that option (4d) is ruled out in the paper.

Following this presentation, Ofqual continued its work, in discussion with the exam boards and others, to develop its arrangements for managing standards in modular GCSEs. These arrangements enabled successful awards of modular GCSEs in 2011, and there have not been significant concerns about awards of subjects other than English in 2012.

The difficulties with awarding the January 2012 English units at the right standard highlight the types of challenges that were discussed in the 2009 paper. We will not face these difficulties in future because we have already announced that we will be moving away from modular GCSEs in England after the 2012/13 school year.

“Ofqual’s approach is a return to norm referencing”

A norm referencing approach would mean that in each exam board there would be, say, 5% getting A*, 10% getting A, and so on. That takes no account of differences in the cohort between years or between exam boards. Candidates entering with an exam board with a higher ability entry would be less likely to get into the top 5%, or the top 10%.

Our comparable outcomes approach is explained in detail in the following document relating to the summer 2012 awards, http://www.ofqual.gov.uk/files/2012-05-09-maintaining-standards-in-summer-2012.pdf. This approach is based on a number of assumptions, including the assumption that if the cohort is similar, in terms of ability, to the cohorts in previous years then the outcomes should be comparable. But if the cohort is different, then the outcomes will be different. This is a fairer approach than a simple norm referencing approach.

“The Welsh Education Minister has ordered a regrade of WJEC papers. Doesn’t this demonstrate that Ofqual should order a regrade in England?”

No. The Welsh regulator has made regulatory decisions about qualifications in Wales, taking into account particular considerations in Wales – Ofqual’s focus is on the standards of qualifications taken in England. We do not agree that the change to the standard required by the Welsh regulator was appropriate for England.

Our interim report, published on 31 August, sets out why we have come to the conclusion that the awards made in June were made correctly and standards were comparable with previous years and across exam boards.

We understand why it is suggested that papers should be regraded. At first sight, that seems fair. We have thought about that very carefully, but we do not think that is the fairest thing to do for students in England. 

Those students who took the assessments in June 2012 can be confident that the grades they were awarded were right. It is true that some of those students may have got better grades had they taken the assessments in January 2012. That is because, in retrospect, it is clear that the January awards (to a relatively small number of students) were generous. But if we were to revisit the June 2012 awards and bring them into line with the generous January standards, it would compound the unfairness. It would mean that, rather than just the relatively small number of January awards benefitting from generous grading, all the students who took the qualifications would benefit.

That would not be fair to GCSE students in other subjects, or students in past years, or future years. Nor would it be fair to the employers, colleges or universities, who will be looking to use these qualifications to make recruitment and selection decisions.

We think that maintaining the results for students in England is the fairest thing to do. And it is our job to maintain standards. We would not be doing the job we are set up to do if we altered standards in this way.

“Ofqual should have realised there was a problem in January”

We have found that the grade boundaries for some units in January were generous, even though the examiners were using their best professional judgement based on the information they had available to them at the time. Setting standards in the subject of English in particular is not easy. In January it proved to be very difficult.  

It is important to understand that grade boundaries do move, sometimes significantly. There is nothing unusual in grade boundary movement.  However, these were new qualifications, with no data or information from previous years. And examiners were assessing speaking and listening skills which are notoriously difficult to assess. These factors, along with a high percentage of controlled assessment in the written unit added to the difficulties. We look at this in a bit more detail, below.

Usually, examiners are helped when they are deciding grade boundaries by a wealth of data and information.  So for example they can generally rely to some extent on how standards were set in previous years, and make comparisons. But this time, examiners were dealing with new qualifications and with a different sort of assessment – controlled assessment. That made it harder for examiners to rely on how standards were set in the past.

Examiners also use data and other information to help them set grade boundaries. Exam boards use the Key Stage 2 scores for a year group to predict that year group’s achievement at GCSE. They do this by looking back at previous year’s results and using the relationship between KS2 scores and those results to predict the percentages of candidates at each grade for the current year. This has proved to be a reliable way of gauging standards. It helps examiners as they come to a final judgement about grade boundaries.

But:

• The predictions are predicting the grades students could be expected to get at the end of a two year course, when they have benefitted from two years teaching and maturity. Using this information to predict how well they will do in early units, such as January, means that ‘over-predictions’ are likely, but it is not possible to quantify by how much. When a qualification is new, it is also difficult to say how the pupils will perform across the different units, which will be their strongest and which their weakest, which adds to the uncertainty.
• Predictions work well for the cohort as a whole, but they are less reliable for a small and unrepresentative sample. Not every student will follow the predicted pattern exactly, due to natural variation caused by many other factors, but this variation is evened out across the whole year group. When a qualification is new, and the entry routes are different as they were for GCSE English, is it also difficult to identify exactly how unrepresentative the group of students taking the unit is, and therefore to make judgements accordingly.

All this means that standards setting was difficult in January. But we know that examiners recognised how difficult it was, and did their level best to get it right.

“The grade boundaries in June had to be harsh to compensate for the generous grade boundaries in January”

This is not the case. We have found that the June grade boundaries were awarded correctly. Examiners used their professional judgement to come to grade boundaries that reflected the performance of the students at unit level and made sure the overall qualification grades were comparable.

“Harsh grade boundaries in June resulted in a big drop in the pass rates for GCSE English”

At national level, there was a small drop (1.5 percentage points) in the proportion of UK students gaining GCSE English at grades A*-C in 2012 compared to 2011. As explained in our report, this fall is because of changes in the cohort that took GCSE English in 2012. Evidence about prior attainment at Key Stage 2 of the students who took the exams shows that this small drop could be expected.

Had the cohort been identical in 2012 to that in 2011, our approach to securing standards (known as ‘comparable outcomes’) would have produced a slight increase in attainment, within the tolerances we set.

We have looked carefully at the grade boundaries in June and found they were awarded correctly. We did not see a big drop in the pass rates for GCSE English this year. The concerns that have been expressed have been from schools that did not get the results they expected, and we are doing more work now to understand that better. There are variations, school by school, and we are doing more work to understand that better.

“Ofqual should have told schools that the grade boundaries would be different sooner”

Grade boundaries are set by exam boards after the exams have been taken, during the process of awarding grades. Grade boundaries do change between units and exam boards do tell schools and colleges this.

As we look at the reasons behind the variability of results at school level, we will consider whether an undue focus on published grade boundaries from past papers and assessments may have been one of the factors behind unexpectedly poor results in some schools.

“Ofqual only cares about statistics, not pupils”

We recognise the difficulties and confusion that these events have caused pupils, parents and teachers. There is, though, an important job – that Ofqual is charged with – to make sure that qualification standards are maintained over time and across exam boards. This is the most important thing Ofqual can do for students, because it means that the qualifications they work hard for carry meaning and credibility over time.

The use of statistics to help inform examiner judgements is an established part of the exam system and is one of the tools exam boards and examiners use to make sure standards are right.

 ”Ofqual forced the exam boards to make their grade boundaries tougher”

For any individual qualification – GCSE English for example – results need to be comparable between one exam board and another.  If they are not, then standards slip, and we know this has happened in the past. Where individual exam boards propose results that differ significantly from expectations, we will challenge them and intervene where necessary to make sure standards are correct. This is the job Parliament set us up to do.

We have the legal power to direct exam boards to make changes, and we are prepared to use that power if needed to make sure standards are maintained. We did not need to issue any directions to exam boards over the awarding of GCSEs or A levels in summer 2012.

“Ofqual disregarded its ‘comparable outcomes’ approach with GCSE English 2012″

This is not the case. The first part of our approach is to consider whether  the student mix for a subject is similar, in terms of ability, to previous years. As our initial report into this year’s English GCSE said, there were some changes to the cohort which explain the 1.5 percentage point drop in outcomes in GCSE English.  But they were not big changes – the cohort was similar enough that the comparable outcomes approach remained valid.       

We apply our comparable outcomes approach if the exams are ‘fit for purpose’ – that is, that that they are a valid and reliable assessment of individual students’ achievements.  It is right that the awarding in English this year proved more challenging than expected – but the important point is that there were no problems with the assessments themselves, beyond what might be expected for a new qualification.

Our approach also says that the purpose, requirements and nature of the qualification should be the same.  Even though there were changes to the structure and assessment the new qualifications were assessing the same National Curriculum Programme of Study for English.

A recent Times Education Supplement article referred to one of the conclusions of our report – that English GCSE grade boundaries were “too generous” in January 2012. It went on to argue that a fourth component of our comparable outcomes approach – that previous grade standards were appropriate – was therefore not met.  But the purpose of comparable outcomes is to maintain qualification level standards year on year.  The issues with January unit awards did not affect previous qualification level grade standards. 

“Ofqual’s letter to Edexcel proves you were secretly downgrading results”

Our job is to make sure standards are right. This is the job that Parliament set us up to do, and it gave us powers to intervene in the awarding process by ordering exam boards to change grade boundaries if necessary to make sure grades are comparable over time and across exam boards.

There is nothing secret about it. We have been up-front about the fact that we challenge exam boards when proposed results are out of line with expectations – exam boards need to use evidence (for example, demonstrating that student work is of a higher quality than expected) to justify such results. The exchanges with the exam board were part of that process and were entirely proper.

“Students who were predicted to get certain grades have been cheated out of them”

GCSEs are awarded by exam boards and it is they – not teachers – who need to make the judgement about whether a student has demonstrated the standard required for a particular grade.

We need to do more to understand how schools predict grades and we are continuing to talk to the representatives of teachers and schools to look into why some schools got results that differed from what they expected.

However, predicted grades are not guaranteed. We have heard from many schools where teachers have been disappointed in the results achieved this summer. We also know that there are schools whose results exceeded their expectations and predictions. We have looked carefully into the awarding of GCSE English this summer and found that the grades awarded are correct.