Friday, 13 April 2018

Missing the mark at GCSE English: the costly consequences of just failing to get a grade C

School students who narrowly fail to achieve a grade C in their GCSE English exam pay a high price, according to new research by Stephen Machin, Sandra McNally and Jenifer Ruiz-Valenzuela.

Getting above or failing to reach thresholds in high-stakes public examinations is an important feature of success or failure in many people's lives. One well-known example is the need to obtain a grade C in English and maths in the age 16 school-leaving exams in England (or Grade 4 in the new system).

This is in part because achievement of good literacy and numeracy skills is recognised as an important output of the education system. It is also because achieving a ‘good pass’ (grade C or better) in these exams has long been recognised as a key requirement for employment. In fact, this level of achievement is deemed so important that since 2015, it has become mandatory for students to repeat the exams if they fail to get a C grade in English or maths and wish to continue in some form of publicly funded education thereafter.

New research by the Centre for Vocational Education Research (CVER) [Discussion Paper 014] analyses the benefits (or costs) for students who just pass (or fail) to meet a key threshold in these exams. More specifically, evidence is presented on the importance of just obtaining a grade C in GCSE English Language (which is the form of English exam undertaken by 72% of students in the cohort under study).

The administrative data that we use follow the cohort that took the GCSE exam in 2013 over the next three years of their lives. Comparing students on the threshold of success and failure makes it possible to explore whether just passing or just failing has consequences for them in relation to their probability of early drop-out from education (and employment) and their probability of accessing higher-level courses, which are known to have a positive wage return in the labour market. The analysis also looks at the effect on the probability of entering higher education.

The question is not so much whether it is important to perform well in English, as whether it is important to get past the specific threshold of a grade C. In other words, the focus is on isolating the effects of good or bad luck, which lead a student to end up on either side of the C threshold. Up to now this has not been evaluated empirically, even though getting a grade C in English is given great weight within English institutions and in public conversation.

Our study makes use of the distribution of exact marks around the important threshold of grade C, using data provided by one of the four national awarding bodies (the AQA). One key feature of English exams is the right to appeal, and while the administrative data contain final (post-appeal) grades (i.e. from A* to G), we have also obtained access to student-level data on the pre-appeal and post-appeal marks. Marks range from 0 to 300, where the C threshold lies at 180 marks.

This is important since we can use these data to ascertain whether or not what looks like manipulation in the data is actually due to the re-grading process through appeals. Our research is unique in having the ‘pre-manipulation’ and ‘post-manipulation’ distribution of marks for the same students.

The findings we report show that just failing to achieve a grade C in English has a large associated cost. Put another way, the marginal student would have performed significantly better in the longer term had he or she not been so unlucky at this point. The results show that narrowly missing the C grade in English language decreases the probability of enrolling in a higher-level qualification by at least 9 percentage points (illustrated in Figure 1). There is a similarly large effect on the probability of achieving a higher (‘full level 3’) academic or vocational qualification by age 19 – which is a pre-requisite for university or getting a job with good wage prospects. There is also an effect on the probability of entering tertiary or higher education.

Perhaps most surprisingly, narrowly missing a grade C increases the probability of dropping out of education at age 18 by about 4 percentage points (in a context where the national average is 12%) – illustrated in Figure 2. It increases the probability of becoming ‘not in education, training or employment’ by about 2 percentage points. Those entering employment at this age (and without a grade C in English) are unlikely to be in jobs with good progression possibilities. If they are ‘not in education, employment or training’, this puts them at a high risk of wage scarring effects and crime participation resulting from youth unemployment in the longer term.



We show some evidence on the mechanisms through which failing to obtain a grade C in English leads to poor outcomes. These involve a narrowing of opportunities that arise within the educational system on the choice of post-16 institution and course the year after failing to get a C grade in GCSE English: students end up in institutions with less well performing peers.

In a well-functioning education system, there would be ladders for the marginal student – or at least alternative educational options with good prospects. The CVER study suggests that the marginal student who is unlucky pays a high price.

Our analysis does not suggest that having pass/fail thresholds are undesirable. Achievement of a minimum level of literacy and numeracy in the population is an important social and economic objective. But the fact that there are such big consequences from narrowly missing out on a C grade suggests that there is something going wrong within the system. It suggests that young people are not getting the support they need if they fail to make the grade (even narrowly).

It also suggests that other educational options available to people who cannot immediately enter higher academic or vocational education are failing to help a significant proportion of young people make progress up the educational ladder. Thus, it is symptomatic of an important source of inequality in education, with associated negative long-term economic consequences for young people who just fail to pass such an important high-stakes national exam taken at the end of compulsory schooling.

Wednesday, 11 April 2018

Choosing the best counterfactual for assessing the returns to qualifications

Sophie Hedges of London Economics summarises a new paper which finds that, for both males and females, non-achievers are generally closer in their observable characteristics to the achievers, than are individuals who only complete the qualification at the level below.”

What’s new?

The labour market returns to qualifications have typically been estimated by comparing the wages of individuals who achieve a particular qualification with the wages of two contrasting counterfactual groups: either level-below (i.e. similar individuals in possession of the qualification at the level below); or non-achievers (i.e. similar individuals who start studying the qualification but then fail to achieve).

The choice of counterfactual used has typically been driven by the data available. Achievement at level-below has generally been used when working with survey data such as the LFS since these surveys only gather information on qualifications achieved. In contrast, non-achievers have been utilised as the main counterfactual group when using administrative data as (until recently) this only covered learners with some interaction with the Further Education system (and reported no, or very limited, information on school and higher education attainment). However, using information from the new Longitudinal Education Outcomes (LEO) administrative data, it is now possible to directly compare the estimates of returns to qualifications from both types of counterfactual using a common dataset.


What do we do?

The objective of CVER Discussion Paper 013 is to understand which counterfactual group (level-below or non-achievers) is relatively more suited for estimating the returns to qualifications in that it comprises individuals who are most similar to those who successfully complete the qualification of interest. In order to do that, we match each individual who achieved the qualification (the treatment group) with their most similar counterpart within the pooled counterfactual group encompassing both non-achievers and achievers at the level below. We then compare the composition of the combined counterfactual group pre-match and post-match (i.e. the sub-sample of the pre-match pooled counterfactual group who were paired with treated individuals). If one counterfactual group (non-achievers or level-below) is over-represented in the post-match sample, then there is a relative preference for that particular control group.

Whilst this process identifies the most similar individuals for the counterfactual in terms of their observable characteristics, it is important to note that it still cannot address the problem of unobservable differences between the treatment and control groups (e.g. innate ability, motivation etc.).

What do we find?

For both males and females, non-achievers are generally closer in their observable characteristics to the achievers, than are individuals who only complete the qualification at the level below. This is particularly true for apprenticeships (both Advanced and Intermediate), NVQs at Levels 2 and 3, and BTECs at Level 3.

How does the counterfactual affect the earnings differentials for vocational qualifications?

We then explored whether estimates of earnings differentials for vocational qualifications vary significantly across the two different counterfactuals. The findings indicate that estimates of earnings differentials using the non-achievers counterfactual group are positive for men achieving vocational qualifications at Levels 3 and 2, although the magnitude is (often considerably) smaller than the earnings differentials estimated using achievement at the level below as the counterfactual (the exception are individuals holding BTECs). For females, the estimated differentials are also positive for all vocational qualifications at Levels 3 and 2, but there is not such a strong pattern in terms of magnitude relative to the estimates using the level-below group as the counterfactual.

Wednesday, 28 March 2018

Apprenticeships are changing: The Levy a year in

Since the introduction of the Apprenticeship Levy a year ago there has been a shift from Intermediate Apprenticeships to the Advanced and Higher Apprenticeships, which are vital for increasing productivity. NIESR's Matt Bursnall and Stefan Speckesser discuss the changes.


Data from the Institute for Apprenticeships on approved and planned Apprenticeship Standards suggest that Higher Level Apprenticeships, in particular degree apprenticeships, will grow further. 

If successful, the new Standards could help build a second, professional route of high-level education to deliver qualifications for associate professional roles, benefiting workplace productivity and social mobility. However, there are also risks, for example if the Levy-funded apprenticeships simply replaced existing under and post-graduate studies, which would have also been undertaken in the absence of the Levy, and did not result in a genuine increase in skills investment.

With many of the emerging Apprenticeships taking up to four years to achieve, we will begin to be able to estimate the full impact of the Levy in a couple of years.

Initial drop of apprenticeship starts

Since 6 April 2017, a UK employer with a pay bill exceeding £3 million per annum pays the Apprenticeship Levy, which is charged at 0.5% of their annual pay bill. Each Levy-paying employer has an account on the Digital Apprenticeship Service to give them more control over how their Levy contribution is spent on apprenticeships, which combine paid employment with study. They can top up their Levy if their apprenticeship training requirements exceed what the Levy can purchase and can transfer up to 10% of their contribution to other employers in their supply chain.

Initially, the introduction of the Levy in the final quarter of the 2017/18 academic year reduced the number of apprenticeship starts markedly: apprenticeship starts were down 59% compared the same period in the previous academic year. In the first quarter of this academic year, the difference declined, but is still 26.5% lower than the period before the Levy. While starts are likely to increase further, we cannot yet fully anticipate the extent to which the quality of the apprenticeships is changing under the Levy.

However, some broader trends become apparent from learner data and data of emerging Apprenticeship Standards, i.e. the apprenticeships currently in the process of approval by the Institute for Apprenticeships.

Changes to higher level apprenticeships

While many apprenticeships are still aiming for Level 2 qualifications with intermediate knowledge, skills and competences of specific occupations, the shares of Advanced and Higher Apprenticeships with qualifications at Level 3 and above have grown. Overall, the proportion of Advanced Apprenticeships in the first quarter of the 2017/18 academic year is now about the same (44%) as the share for Level 2 Intermediate Apprenticeships (45%), after it had been 53% for Intermediate Apprenticeships in 2016/17 and typically around 60% in other recent years. In some subjects, such as Engineering, Heath and Leisure/Tourism, Advanced Apprenticeships dominate, and there are now considerable shares of Higher Level Apprenticeships in Science (50%), ICT (25%) and Business (20%) (Figure 1).

Source: Department for Education (based on 114,380 starts of Q1 2017/18)

When comparing the overall 495,000 apprenticeship starts of 2016/17 with the starts of 2017/18 by levels (Figure 2), the growth of Higher Apprenticeships in almost all subject areas stands out, and especially in science/mathematics, ICT and construction. In contrast, the shares of Intermediate Apprenticeships declined in all subjects apart from education.


Source: Department for Education (494,900 of Q1-Q4 2016/17 compared to 114,380 starts of Q1 2017/18)


Many higher level apprenticeships and degree apprenticeships planned

Based on a look into the Institute for Apprenticeships’ database (taken a snapshot earlier this week on 26 March 2018), we can expect strong growth of apprenticeships with higher level qualifications, including undergraduate or postgraduate degrees (Figure 3). The database currently holds 542 existing or planned Apprenticeship Standards. While two thirds of the 246 existing Standards are at Intermediate and Advanced Apprenticeship level (Levels 2 and 3) which existed for many years, there are now many apprenticeships aiming for higher level qualifications.

Before the introduction of Apprenticeship Standards, the Higher Apprenticeships included mainly Foundation Degrees from a university (a Level 4 qualification). With the new Standards coming in over the next few years, there will be many apprenticeships aiming to include qualifications equivalent to a university degree or even a Master’s degree (Levels 6 and 7), e.g. 17% of the 296 Standards, currently going into be approvals process, aim for Level 6 qualifications.

Source: Institute for Apprenticeship standards, downloaded 26 March 2018

Along with the move to apprenticeships with higher level qualifications will be further change: In sectors traditionally operating an Apprenticeship system, e.g. Construction, Apprenticeships took between one and three years to achieve. Compared to this, many of the incoming apprenticeships will take more than three years to complete (Figure 4), e.g. 40% of the Level 6 apprenticeship will last for more than four years.


Source: Institute for Apprenticeship standards, downloaded 26 March 2018 (based on all approved Standards)


Since higher level apprenticeships are also more costly to deliver than Intermediate and Advanced Apprenticeships, most will apply the maximum amount of funding: 20 (75%) of the approved Level 6 apprenticeships are associated with the maximum of £27,000 of funding, while there are 51 further Standards, where approval of the funding band is pending. In summary, we can expect that a large segment of degree-type education (at the costs of a degree for people with satisfactory pre-existing qualifications) will be undertaken through the apprenticeships route and funded by the Levy.

Source: Institute for Apprenticeship standards, downloaded 26 March 2018 (based on all approved Standards)

Higher level apprenticeships, but overall impact remains unclear

It is positive that under the Levy, more apprenticeships are aiming for higher level qualifications, which are associated with significant earnings improvements compared to Level 3 technical or academic qualifications. This tendency is already emerging from reported apprenticeship starts, but will happen even more often once the incoming Standards are operational.

Since apprenticeships are organised by employers, incoming Standards could help create a second, professional route of high-level education for associate professional roles. Using the Levy funding in this way could create new opportunities for high-level education, e.g. for those already working in firms, and help improve social mobility. However, there are also significant risks to the success of this policy, for example if the Levy-funded apprenticeships simply replaced existing under and post-graduate studies, which would have also been undertaken in the absence of the Levy, and did not result in a genuine increase in skills investment.

Monday, 26 February 2018

Do BTEC qualifications pay?

London Economics' Pietro Patrignani with a closer look at BTEC qualifications


BTEC qualifications are career-based qualifications and can be taken at different levels of the Regulated Qualification Framework (RQF) between Level 1 to Level 5 and above. BTECs at Levels 1 to 3 are normally taken at Further Education Colleges or schools, while BTEC qualifications at higher levels are available at FE colleges and Higher Education Institutions. While BTECs are specialist work-related qualifications, they are college based and not work-based learning qualifications (such as NVQs and Apprenticeships). BTECs account for a significant share of those with Level 3 vocational qualifications: for example, they account for 47% of young men and 30% of young women who have Level 3 vocational qualifications as their highest qualification (excluding Apprenticeships). 

Our recent research published in the CVER Discussion Paper 007 used the Longitudinal Education Outcomes (LEO) data to estimate the earnings differentials associated with a range of technical and vocational qualifications for young people. This showed the earnings differential associated with qualifications when they are the highest level attained by a young person. A positive earnings differential can reflect both higher wages and higher hours of work. Our analysis showed that, while for NVQs and Apprenticeships there were positive and strong earnings differentials for men and women, for BTECs this was only true for women. For men, there was no significant effect from having BTECs as the highest qualification. This is in contrast with typical evidence from the Labour Force Survey: findings using historical data have usually shown positive wage differentials for BTECs for both males and females at all levels of the Regulated Qualification Framework (RQF). While the LFS uses survey data referring to a sample of working age individuals (16-64), analysis using LEO is based on information from matched administrative data sources for a group of young learners who recently went through the English educational system, with labour market outcomes measured at the age of 26. As a result, part of the difference in the estimates is driven by age, as shown in CVER Discussion Paper 009.

We have followed on from this work by investigating BTECs in far more detail. This is the subject of our most recent briefing note.



Personal characteristics and labour market outcomes

We first investigate the characteristics and labour market outcomes for the group of learners in possession of BTECs and other vocational qualifications, with a particular focus on Level 2 and 3 qualifications. 

In terms of personal characteristics and further education attainment the key findings show that, compared to NVQs and other vocational qualifications:

  • The average level of prior attainment (measured at the ages of 11 and 16) is substantially higher for both men and women undertaking BTECs compared to other vocational routes. 
  • A higher proportion of learners holding BTECs are from BAME backgrounds (non-white British), and this is true for both males and females across all levels.

When looking at further study and attainment at higher levels, BTEC qualifications at Level 3 and Level 2 often act as a stepping stone for further study and education for both males and females:

  • Between 40% and 45% of learners with BTECs at Level 3 attain degree-level qualifications or above, compared with between 5% and 8% for Level 3 NVQs and 20% and 25% for other vocational qualifications at Level 3. 
  • At Level 2, the percentage of BTEC holders achieving at Level 3 qualification or above is in excess of 50%, with 15% attaining at least degree-level qualifications or equivalent. The corresponding proportions for NVQs at Level 2 are considerably lower, and slightly lower for ‘other’ vocational qualifications.


Earnings differentials

We then estimated earnings differentials for BTECs and other vocational qualifications using a variety of counterfactual groups (including results from the Labour Force Survey restricted to younger people). The main findings indicate that: 
  • For males in possession of Level 3 BTECs as their highest qualification, we failed to observe positive earnings differentials compared with individuals achieving at the qualification level immediately below. However, the earnings differentials turn positive if we restrict our attention to individuals holding Level 2 BTECs or individuals enrolling in Level 3 BTECs but failing to achieve. 
  • For Level 2 BTECs there is little evidence of positive earnings differentials, while the estimates for other Level 2 vocational qualifications are more often positive.
  • For females, earnings differentials for both Level 3 and Level 2 BTECs are positive and strong, and more consistent across the different specifications. They typically range between 10% and 15%, and are slightly larger than the estimated earnings differentials for other vocational qualifications at the same level.

Although the analysis using the administrative data did not show positive earnings differentials for males compared to individuals at the level immediately below of the RQF, it would not be reasonable to interpret this as evidence that the qualification does not add any skills or proficiencies that are valued in the labour market. This is because BTECs often enable learners to progress and attain at higher levels. Indeed they are the main vocational route to higher education and equivalent qualifications.

Furthermore, earnings differentials are positive in some specifications and it might be the case that earnings differentials become higher in later years (as reflected in the LFS estimates), as we are considering estimates for young people within a relatively short time of leaving the education system.

--------------------------------------------------------------


"Further analysis of the earnings differentials associated with BTECs", Pietro Patrignani, Sophie Hedges and Gavan Conlon, CVER Briefing Note 006 (February 2018) is available at http://cver.lse.ac.uk/publications/default.asp

Friday, 2 February 2018

Which skill signals matter truly in getting a job?

Lisa Simon from the ifo Center for the Economics of Education on the impact of skill signals on hiring decisions


Individuals make costly investments (in terms of time, effort or money) to signal skills to potential employers, such as getting good grades in school or college or learning a foreign language. However, there is little understanding on which skill investments will pay off in finding employment upon entering the labour market. Labour market entrants use skill signals to convey productivity to future employers as these cannot directly observe applicants’ skills in the first stage of the hiring process (i.e. written applications including CVs). While there is a well-known relationship between cognitive or non-cognitive skills and individual labour market outcomes, the role of skill signals themselves is less well established. Analysing and isolating the effect of how skill signals such as grade-point-averages (GPA) affect labour market outcomes is difficult for at least two reasons: First, different types of skills are usually correlated, i.e. a smart person with a high GPA usually also has other good skills. Second, we do not know from observational data, to which extend a potential employer really observes skill signals.

In a new paper with Marc Piopiunik, Guido Schwerdt and Ludger Woessmann, we conduct an experiment with a representative sample of nearly 600 human resource managers to analyse the effects on employment of skill signals within three domains: cognitive skills, social skills and maturity. In an online survey, our participants are asked to choose between two CVs, which appear side-by-side on the screen: “Which applicant would you rather invite for a job interview at your firm?” The (fictitious) CVs have randomly assigned skill signals, which allow estimating the causal effect of each skill on being invited for an interview. The choice experiment mimics the first stage of an application process, in which human resource managers review written applications and decide who to invite for an interview. We look at two distinct groups of labour market entrants for whom relevant skills, requirements and expectations vary: secondary-school graduates applying for an apprenticeship and university graduates with a BA in Business. Cognitive skills are signalled through the (school or university) GPA, IT and foreign language skills. Social skills are signalled through social volunteering and playing team sports. Maturity is signalled through age, long internships and the high-school GPA in case of university graduates. The experiment was followed by a short survey on HR manager characteristics and hiring preferences.



Our results

We find that skill signals in all three domains – cognitive skills, social skills, and maturity – affect the probability of being invited for a job interview. The figure below shows point estimates and confidence intervals for all signals for both labour market entrant groups. GPAs prove important for all, with a stronger effect for university graduates than for secondary-school graduates. IT and language skills are particularly relevant for females. Social skills are highly relevant for both genders and particularly important for secondary-school graduates entering the labour market at a young age. Maturity is particularly relevant for males, especially for secondary-school graduates. These heterogeneities by labour market entry age and gender are consistent with varying relevance, expectedness, and credibility of the different skill signals in different contexts. 




We also find heterogeneities with respect to HR managers’ personal characteristics. For secondary-school graduates applying for an apprentice position, managing directors and older HR managers put less weight on school GPAs, but instead more weight on IT skills, social volunteering, and experience through internships. Among college graduates, HR managers in large firms value college GPAs more, possibly due to a more standardized procedure of applicant selection. Furthermore, we find that the self-reported hiring priorities of HR managers are consistent with their decisions in the choice experiments, confirming the intended information value of the skill signals.

Our study reveals important aspects about how signals of skills are processed and utilized in the labour market. Gender differences in the effects of language skills, IT skills, and maturity are generally in line with gender stereotyping. Social skills are most effectively signalled by social volunteering among secondary-school graduates but by engaging in team sports among college graduates, possibly reflecting limited credibility of volunteering activities of older individuals who may behave strategically.

Our results also suggest that skill signals with straightforward verifiability in real hiring situations, such as GPAs, internships, and age, tend to have higher returns than skills that are more costly to verify, such as language or team sports, in particular at large firms. In terms of policy implications, our paper stands in contrast to a vast literature that deals with discrimination on the labour market due to innate characteristics such as race or gender. Our study shows which skill signals that can be changed and acquired through investments of effort, time or money, impact hiring decisions.

--------------------------------------------------------------
"Skills, Signals, and Employability: An Experimental Investigation" by Marc Piopiunik, Guido Schwerdt, Lisa Simon, and Ludger Woessmann, CVER Research Paper 012 (February 2018) is available at http://cver.lse.ac.uk/publications/default.asp

Tuesday, 19 December 2017

How much do college principals matter?

CVER's Jenifer Ruiz-Valenzuela, along with Camille Terrier and Clémentine Van Effenterre have been looking at evidence from Further Education Colleges in England, and ask How much do college principals matter?


The past twenty years have witnessed a large increase in research about the role of Chief Executive Officers (CEOs) in the management of firms; although much less is known about the effectiveness of CEOs in the public sector. A better understanding of what makes good public-sector managers is crucial to better inform decisions on leadership and management in key public services such as health, transport and education.

Tuesday, 5 December 2017

Apprenticeship and automotive skills: the UK, Germany and Spain compared


CVER's Hilary Steedman and colleagues have been looking at training in one area of the automotive sector
 

Car Service is central to the supply chain of the wider automotive sector, identified as a leading performer in the UK government’s 2017 Industrial Strategy. We asked Car Service employers in Germany, UK and Spain about skill shortages and their experience of training apprentices in the workplace.