6227.0.55.002 - Experimental estimates of education and training performance measures based on data pooling, Survey of Education and Work, 2007 to 2010, Sep 2011
ARCHIVED ISSUE Released at 11:30 AM (CANBERRA TIME) 09/09/2011 First Issue
Page tools:
 Print Page
 Print All
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
OVERVIEW
THE INDICATORS There are four COAG measures that are based on the Survey of Education and Work (SEW). In addition to estimates by state and territory, the measures are further disaggregated by socioeconomic status. While Aboriginal and Torres Strait Islander status is also a key priority for data development, related performance measures on education and training for Indigenous people are sourced from Indigenous-specific surveys and therefore the topic is outside the scope of this study.
TRENDS BETWEEN 2001 AND 2010 To set the context for examining change over the short term, this section briefly reviews trends in the indicators between 2001 and 2010. Over the past 10 years, there has been an upward trend in NEA 7 (Year 12/Certificate II, 20-24 years) at the national level, where this measure rose by about 6.5 percentage points from 79.1% in 2001 to 85.6% in 2010. Trends at the state/territory level are less clear from the survey data. There has been a certain element of year to year fluctuation, especially in the jurisdictions with smaller populations, most notably the NT. This could well reflect sampling variability and suggests that caution should be applied when interpreting change in NEA 7 using standard estimates from the SEW, particularly from one year to the next (see the section on significance testing for further explanation of sampling variability and measures of change). The other attainment measure, NASWD 2 (low qualifications, 20-64 years) is based on a much larger population (a 45 year age range) than the NEA indicators (based on 5 to 7 year age ranges only) and as such has a larger sample and more reliable estimates for all jurisdictions. It has also changed more than NEA 7, falling by 12.6 percentage points from 58.0% in 2001 to 45.4% in 2010. The downward trend over the past 10 years in this ‘at risk’ indicator is reasonably clear for all jurisdictions. The trend in engagement: NEA 9 (engagement, 15-19 years) and NEA 10 (engagement, 18-24 years) was less clear and appeared to differ among jurisdictions. For both measures, the overall level of full engagement in employment or education/training was much the same in 2010 as it was in 2001 but was subject to some degree of fluctuation over the intervening period. At the national level, the rate of full engagement among 15-19 year olds measured by NEA 9 was 72.5% in 2001 and 69.8% in 2010. The corresponding rates of full engagement for 18-24 year olds from NEA 10 were 71.4% and 72.6%. These measures incorporate participation in work or study, both of which are influenced by the economic cycle. Like NEA 7, the engagement measures are based on a small sub-population (5 to 7 year age ranges). Therefore, the pattern over the past 10 years was influenced both by variation in the actual level of engagement by young people and to sampling variability. Once again, caution should be exercised when interpreting change in these indicators over the short term. Time-series data for the indicators can be found in datacubes associated with Education and Work, Australia (cat. no. 6227.0) THE POOLED DATASETS In order to examine trends over the short term and the potential benefits of data pooling, data were combined from successive cycles of the annual SEW. Two-year data represent pooling over 2007 to 2008 and 2009 to 2010. Four year data represent pooling over the entire period 2007 to 2010. The pooling methodology used in this study involved benchmarking the initial weights in the combined or pooled dataset to required totals based on geographic, demographic and labour force characteristics of the population at the end point of the pooling period. As such, the reference point for the pooled estimates is that of the latest survey. The samples for earlier years in the combined datasets can be regarded as 'lagging' samples. For instance, when pooling the 2007 and 2008 surveys, the pooled estimates are for the 2008 period and the sample for 2007 is assumed to be a lagging sample for 2008 (see methodology section for further explanation). Accordingly, single and two-year pooled estimates are available for 2008 and single, two-year pooled and four-year pooled estimates for 2010. Since the pooled data originates from time periods prior to and including the reference year, the pooled educational qualification estimates may be affected by structural changes in the population and changes in labour force participation during the full period of the pooled data and so are expected to be 'lagging' behind the individual yearly survey estimates for each time point. In the time horizon of this study from 2007 to 2010, one time comparison: 2008 to 2010 can be made to assess change using two-year pooled estimates. Since only one set of four-year pooled estimates could be calculated, no time comparison is available from four-year pooled data. The gains in the accuracy of estimates from data pooling over two years are effectively equivalent to those from doubling the size of the original survey sample, and from four-year pooling to conducting a survey with a sample four times that of the original. For SEW, the single-year sample in 2010 was about 40,000 persons, the two-year pooled sample 2009-2010 about 70,000 persons and the four-year pooled sample about 154,000 persons. The size of the sample for the Labour Force Survey (LFS) and, as a consequence, associated supplementary surveys such as SEW was reduced in 2009 before being reinstated in 2010. Therefore the pooled dataset over 2009 to 2010 (about 70,000 persons) was smaller than would be expected based on the current sample size (about 80,000 persons). As a result, to some extent, the reliability of the pooled data was less and Relative Standard Errors (RSEs) higher than could be expected from future pooling based on the full LFS sample. As such, potential gains from pooling may be slightly underestimated in this report. HOW DID DATA POOLING AFFECT THE LEVEL OF PERFORMANCE MEASURES AND THEIR RELIABILITY? For the 2010 reference point, the estimated levels of the indicators examined were broadly similar for single-year and pooled estimates, provided allowance is made for the lag in the pooled data (Table A). The largest difference in the indicator, and underlying estimate, occurred for NASWD 2 (low qualifications, 20-64 years) where the indicator based on four-year pooled data was 2.2 percentage points or proportionally 5% higher than the single-year indicator (47.6% compared with 45.4%) (Table A below). The population base for this indicator is all people aged 20-64 years and so structural changes in the population and changes in labour force participation would likely have influenced the pooled data. For the other indicators the three measures, based on five to seven year age ranges only, had very similar values, with a proportional difference of under 2% between single-year and pooled data. When compared with data taken from a single year, RSEs declined moderately for two-year pooled data and approximately halved for four year pooled data. This is in line with the theoretical calculation that it requires a quadrupling of the sample to halve the standard error. Reflecting relative sample sizes, the smallest RSEs were associated with NASWD 2 (low qualifications, 20-64 years) and the highest with NEA 9 (engagement, 15-19 years). TABLE A. LEVEL OF INDICATORS AND RSEs, single-year and pooled data, 2010
DID DATA POOLING ENABLE BETTER DETECTION OF CHANGE OVER TIME? This section investigates the detection of change in the selected indicators over time, comparing the statistical significance of results using standard single-year data with two-year pooled data. In theory, data pooling should lead to better detection of change over time. In practice, however, data pooling does not necessarily result in more frequent detection of change. Change may have occurred, but the change may not be of a sufficient magnitude to be detected, even using pooled data. Alternatively, change may have occurred, but the magnitude of change may be large enough to be detected by both standard and pooled data. Finally, change may not have occurred, and pooled data should correctly detect no change. Based on single-year data, there were virtually no detectable changes at the 5% level of statistical significance at the national or state/territory level in any of the three National Education Agreement (NEA) indicators over the period 2009 to 2010 (Table B below). While there were some statistically significant results for change within sub-categories of the indicators, such as Socio-Economic Index for Area (SEIFA) quintile, these may have been due to random chance rather than actual change. As already noted, even without further disaggregation by highest year of school completed or SEIFA quintile, these indicators refer to sub-populations of young people determined by narrow age ranges and as a consequence are based on small samples (Datacube Table 10). Over the previous year 2008 to 2009, by contrast, the two engagement measures NEA 9 (engagement, 15-19 years) and NEA 10 (engagement, 18-24 years) showed statistically significant change at the national level and in Victoria (both measures) and Queensland (NEA 9 only) (Table B below). At the national level, the fall in full engagement in employment and education/training was associated with a fall in employment (Datacube Table 1c). Even when change in NEA 7 was assessed over the three-year period, 2007 to 2010 (using single-year data), there were statistically significant rises in Year 12/Certificate II or above attainment for 20-24 year olds at the national level and for NSW only. No change was detectable in the other jurisdictions. A small number of statistically significant results were observed for change in NEA indicators in two-year pooled data, but this was over a two-year period 2008 to 2010 (Table C below). At the national level, falls in NEA 9 (engagement, 15-19 years) and NEA 10 (engagement, 18-24 years) were statistically significant, both for the proportion of people fully engaged and for full-time employment (Datacube Table 1c). For NEA 10, full-time education/training showed a statistically significant rise, but this was not sufficient to offset the decline in employment (Table 1c). At the state/territory level, a statistically significant change was observed in Victoria for NEA 9 and NEA 10, and in Queensland for NEA 9. Whereas the pooled data was assessed over the two-year period 2008 to 2010, the single-year data was assessed over the one-year period 2008 to 2009 or 2009 to 2010. The greatest annual change in the engagement indicators NEA 9 and NEA 10 occurred between 2008 and 2009 during the economic downturn of the Global Financial Crisis. There was comparatively little change at the national level in engagement between 2009 and 2010. Thus the comparison between single-year and pooled data is affected by the time period over which the comparison is made. Indeed, had data been pooled over 2008-09, the fall in engagement may have been obscured. If datasets were available, pooling over one year would help overcome these issues. Looking again at single year data over 2009 to 2010, the indicator from the National Agreement on Skills and Workforce Development, NASWD 2 (low qualifications, 20-64 years), based on a much larger age group and subject to greater annual change, showed a significant fall at the national level and for Victoria (Table B below). There was also a significant fall at the national level for two age groups: 35-44 years and 45-54 years (Datacube Table 1c). Similarly, there was a statistically significant fall in some age groups for Victoria (Datacube Table 3c). Over the previous year, 2008 to 2009, there was a detectable fall in this indicator at the national level but not in any of the states/territories. By comparison, based on pooled data, all jurisdictions except Tasmania and the ACT recorded statistically significant falls in the proportion of the 20-64 year old population with qualifications below Certificate III between 2008 and 2010 (Table C below). At first glance, these findings might appear to endorse the benefits of data pooling in the case of NASWD 2. Certainly, since this indicator steadily declines, data pooling allows the trend over a two-year period to be detected as significant in more cases than when using single-year data over one year. However, this begs the question of whether or not the trend would also have been detected as significant if single-year data were compared over the same two-year period. This issue is examined immediately below. TABLE B. STATISTICAL SIGNIFICANCE OF CHANGE (a), single-year data, 2008 to 2009 and 2009 to 2010
a. Based on 2-tail test at 5% level of significance (Z=1.96) When single-year data were compared with two-year pooled data over the same two-year period 2008 to 2010, results were very similar. At the national level, there were no differences in the findings of significant change between single-year and pooled data over 2008 to 2010 for any of the four indicators (Table C below). Change in NEA 7 (Year 12/Certificate II, 20-24 years) was not significant for either single-year or pooled data while change in the other indicators was significant in both. Similarly, at the state and territory level there were minor variations only between the significance of comparisons using single-year or pooled data. In addition to significant change in the indicators NEA 9 and NEA 10 at the national level, change was detected in only a small number of jurisdictions. Only in NASWD 2 was change detected both at the national level and in a majority of jurisdictions. Data pooling led to similar improvements in the reliability of estimates but marginal gains only in the detection of change when indicators were disaggregated by population characteristics, such as age, SEIFA quintile or highest year of schooling (see following sections). TABLE C. STATISTICAL SIGNIFICANCE OF CHANGE (a), single-year and pooled data, 2008 to 2010
a. Based on 2-tail test at 5% level of significance (Z=1.96) FOOTNOTES [1] Originally 'The proportion of young people participating in post-school education or training six months after school.' (back to text) [2] Full engagement in employment, education or training comprises full-time employment, full-time education/training or a mix of part-time employment combined with part-time education/training. For NEA 10 education/training must be at Australian Qualification Framework (AQF) Certificate III level or above (e.g. a trade qualification, vocational Diploma or university degree). (back to text) Document Selection These documents will be presented in a new window.
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
