Enhancing measures of non-market output in economic statistics: Progress paper

Jason Annabel - Economic Research Hub

This paper discusses recent progress made by the Australian Bureau of Statistics in enhancing measurement of non-market economic activity, following on from the ‘roadmap’ paper published in September 2019 that articulated the ABS’ broad strategy.

As indicated in the 'roadmap' paper¹, the ABS defines the industries of Health Care and Social Assistance, Education and Training, and Public Administration and Safety, as being predominantly non-market in nature. Non-market economic activity occurs when goods and services are provided to consumers free of charge, or at highly subsidised prices.

Non-market measurement is important given the sizeable contribution it makes to the Australian economy. These three industries contribute 17% of Australia’s gross value added² and provide 3,875,200 jobs³. This research is also a step along the road to being able to measure productivity for non-market industries, which will help governments assess the ‘value for money’ received through the provision of public services.

ABS is currently enhancing estimates of six non-market service areas:

• Hospital services;
• Ambulance services;
• Disability services;
• Aged care services; 
• School education (pre-school, primary, secondary and special needs education); and
• University education.

The initial focus is on hospital services and school education.

Measuring non-market activity is difficult because of a lack of meaningful prices. Here is the measurement problem, explained with an illustration:

Market prices allow the output of different products to be weighted appropriately in aggregation. For instance, imagine that comparing the market prices of cars and bicycles tells us that a car is worth 100 times more than a bicycle. When we aggregate across all products to estimate GDP, each car produced will carry 100 times the weight of each bicycle made. The prices of individual products are needed to derive indicators of relative value – or, weights – which are essential in aggregating the spectrum of all goods and services into a single measure of total output.

When there are no prices, it is difficult to weight the output of products. For instance, we might intuitively assume that a brain surgery procedure provided by public hospitals is more valuable to society than a hernia repair and should ‘count’ for more in aggregation, but we can’t tell by how much, because both services are provided to the consumer free of charge. It is difficult to aggregate brain surgeries with hernia repairs, because no price information exists to weight them.

There are a range of measurement options outlined in the System of National Accounts that are available in the absence of market prices⁴. One option is to assume that the quantity of output produced is equal to the quantity of inputs consumed in producing them. For example, the inputs to a public school include salaries paid to teachers, administrators and ancillary staff, as well as materials and supplies that are used up in production. Under this scenario, if public school teachers earn a real pay rise, meaning input costs increase, it would imply that there has been an increase in output, even if the service being provided is unchanged (for instance, if there is no change to the number of teachers employed or the number of students enrolled). Since outputs and inputs are highly correlated under this approach, productivity growth cannot meaningfully be estimated.

A second measurement option is to model a set of ‘pseudo’ price indexes for various types of non-market output. These ‘pseudo’ price indexes would then serve as the aggregation weights, similar to the cars and bicycles example outlined earlier. The indexes would theoretically reflect the prices that would be paid for non-market output, if prices existed. An example might be to model the market value of a tonsillectomy provided in a public hospital, and then estimate how this value changes over time. 

A third measurement option, which is conceptually equivalent to the second option described above, is to estimate the growth in the quantity of different types of non-market output, and then explicitly adjust them for perceived changes in quality over time. Examples of explicit quality adjustment might include accounting for improvements in school test scores over time, or adjusting for changes in hospital waiting periods over time.

A final fourth possibility is to use the costs of production to drive the aggregation weights. It might be difficult to determine by how much more society values public hospital brain surgeries compared to hernia repairs in terms of the prices paid by consumers, but expenditure data permits an objective way to determine how much more expensive one procedure is to perform compared to the other.

When data is appropriately stratified, this approach allows quality change over time to be implicitly reflected as the cost weights change over time. For example, if tonsillectomies previously required an overnight hospital stay, but technological progress now means they can be performed as ‘day’ surgery, the procedure now costs less to perform than it did before. Implicitly, this is an improvement in quality. Differentiating the overall service into homogenous pieces via stratification allows the concept of utility to be reflected in the statistics, the only drawback being that it is observed from the perspective of the producer and not the consumer. It also means that observed data can be used to compile the aggregation weights as opposed to estimating or modelling them.

ABS will be pursuing the fourth option. This is considered to be international best practice in terms of compiling national accounts statistics.

The ABS’ non-market economic activity work is focused on enhancing measures of the output of the Health and Education industries. The distinction between outputs and outcomes is important. Some examples of this distinction include:

• The output of a hospital is the value of the service performed by surgeons, nursing staff and other health professionals in detecting and treating disease. Outcomes can be measured in terms of a patient changed life expectancy or their ‘satisfaction’ with their treatment.
• The output of ambulance services is the value of the paramedic and transportation services provided to patients. Outcomes can be thought of as the effect of the service on the health and wellbeing of an individual, and can be measured in terms of response times and clinically meaningful pain reduction.
• The output of schools encompasses the value of the teaching and administrative services provided by education professionals which impart knowledge to students as reflected in the number of students being educated. Outcomes are the impact on student intelligence and welfare, which could conceivably be measured through an analysis of standardised test scores, or expected lifetime future earnings.

The distinction between outputs and outcomes with respect to health and education activity is further explained in Schreyer (2010)⁵.

The objectives are to construct annual chain-linked Laspeyres-based index series that show growth in the volume⁶ of non-market output over time. Similar to the chain volume measures that underpin the Australian national accounts, these indexes will be reweighted annually. The key difference is that the aggregation weights will be constructed from relative costs of production instead of relative prices.

By way of example, the ABS aims to construct a volume index of the output of school education by aggregating lower-level index series. These lower-level series will be compiled from source data stratified as follows:

• Jurisdiction (state, territory);
• School type (pre-school / primary / secondary / special needs); and
• Affiliation (government / non-government).

The variables and stratification choices for each of the topics will differ according to the nature of the topic and the types of available data sources, but the approaches for each will be conceptually aligned to the example below.

The section below describes progress in measuring each of the six non-market service areas listed earlier. It outlines the data landscape, stakeholder engagement, opportunities and challenges presented by the available data, and next steps.

A majority of hospital expenditure is on admitted patient care, where patients undergo a formal admission process. The ABS recently published some research on admitted patient care⁷, where admissions were stratified by principal diagnosis (using the International Classification of Diseases) and patient age. Research is now focused on investigating ways to compile a more holistic view of hospital activity by incorporating the two other major streams of hospital activity - emergency department activity, and non-admitted patient care.

Inclusion of emergency department and non-admitted patient care in the output measure is important in the context of capturing shifts across different types of care over time. For example, a shift from admitted care to non-admitted care can be considered an efficiency gain if the latter costs less but can achieve the same or a better outcome. These shifts can be partially captured through annual updating relative cost weights across different types of care.

Emergency department care and non-admitted patient care are provided mainly in public hospitals. Private hospital activity in these areas is not being pursued as it is estimated to represent less than 4% of non-admitted events in 2016-17⁸.

ABS is working closely with the Australian Institute of Health and Welfare (AIHW) who collect data for emergency department activity and non-admitted care. AIHW publishes the number of emergency department presentations and non-admitted care occasions or events back to 1994-95. The Independent Hospital Pricing Authority (IHPA) publishes cost weights for emergency department and non-admitted care by relevant classifications back to 2009-10. These cost weights can be used for constructing indexes.

There are challenges in relation to tracking patients through the hospital system. For instance, if a patient is treated as an admitted patient and is then subsequently treated for the same condition as a non-admitted patient, this conceptually represents one treatment but it would be 'counted' twice in the source data. Constructing an integrated dataset that links all three types of care from patient level records is not statistically practical at this time.

It is therefore more realistic to construct separate quantity indexes for emergency department care and non-admitted patient care, with these indexes aggregated with the admitted care index to produce a quantity index for total hospital activity.

Additional challenges arise from variations in the scope and coverage of the available cost data for emergency department and non-admitted care over time, as well as from changes in classifications and coding standards in reporting of emergency department and non-admitted care activity.

Two options exist for measuring growth in emergency department activity. One involves stratification using the Urgency Related Group (URG) classification, whereas an alternative is to regroup the data by triage and ICD chapter. These options are currently being worked through. One benefit of mapping from URG to ICD is that it is stratified similarly to admitted care, but a drawback is that the mapping is approximate, and refinements will be required.

Data for non-admitted care provided by hospitals is classified by the IHPA's Tier 2 classification, which reflects the nature of the service provided and the type of clinician providing the service. Again, there are two stratification alternatives - to use the Tier 2 classification as is, or to map to ICD. The pros, cons and limitations of each option are similar to those applying to emergency care.

There is a lack of expenditure data to match the level of details of activity prior to 2009-10, which means that historical quantity indexes for emergency department activity and non-admitted care will likely have to be constructed from aggregates prior to this year. This will make it difficult to capture shifts across different types of emergency and non-admitted treatment in earlier years. 

In addition, ABS aspires to extend the time series on admitted care back to 1994-95, though there are challenges in doing so, not least of which is sourcing relevant datasets.

The benefit of using a finer disaggregation of diagnosis-based data for admitted patient care has been considered. In theory, lower level detail is better for controlling for compositional changes. However, in practice, the benefit is limited by variability in source data quality over time. It is considered that the significant investment in data acquisition, mapping and repair needed for a lower level of disaggregation would not justify the analytical benefit.

The ABS is in the process of compiling a set of experimental indexes covering emergency department activity and non-admitted patient care. Variation in source data scope and coverage are challenges currently being dealt with.

By June 2020, ABS intends to have refined and published the indexes covering emergency department and non-admitted patient services, and aggregated them with indexes for admitted patient care. This aggregated index will reflect a holistic view of hospital services, and is intended to support improvements to the annual national accounts in 2021.

The vast majority of ambulance services in Australia are publicly funded. Privately funded ambulatory services will be excluded from this study for reasons of immateriality.

The ABS already compiles a directly observed quantity indicator of ambulance services. This is primarily compiled from data sourced from the ROGS, where the quantity metric is the number of patients treated (whether subsequently transported to hospital or not).

The primary data source is likely to be the Productivity Commission's Report on Government Services (ROGS)⁹. The ROGS data covers public ambulance services and is available on a State/Territory basis. It covers both urban and rural services, as well as services provided by salaried staff as well as volunteers. Amongst a range of other metrics, it captures the expenditures incurred at the State level to provide ambulance services, as well as the number of incidents, which are further categorised into emergency, urgent and non-emergency.

Measurement alternatives include the number of patients treated, the number of incidents requiring ambulatory services (an incident is defined as an event that results in a demand for ambulance services to respond), or the number of responses (defined as the number of vehicles sent to incidents). Using the number of responses would include output produced when a vehicle is dispatched to an incident, irrespective of whether patients are subsequently treated or not. 

The ROGS notes that "The role of paramedics is expanding to include the assessment and management of patients with minor illnesses and injuries to avoid transport to hospital." While this objective tends to increase the expenditure per person on ambulances, it has the tendency to deliver downstream savings by reducing hospital admissions as well as reducing the burden on hospitals.

The percentage of patients not transported to hospital has increased over the past decade, reflecting an increase in paramedics treating patients at the scene. This reflects increased capability of paramedics (underpinned by improvements in training and equipment) as well as actions to reduce hospital emergency congestion. Therefore, the volume index needs to be interpreted in the context that the role of the paramedic has changed over time in pursuit of a broader goal, rather than becoming less efficient over time.

There is currently 10 years of data available and analysis is indicating that aggregate indicators were not sensitive to cost weighted aggregation across states instead of total unweighted numbers, so the totals will be used directly. Further analysis is expected to be published in June 2020, with implementation into national accounts and productivity statistics to follow in 2021.

While compositional change at the State level of activity can be captured, the ABS will investigate whether cost weights are available by severity of incident (i.e. costs incurred on emergency incidents versus urgent incidents and non-emergency incidents) within each State. We will explore whether further disaggregation can improve the quality of the aggregate output volume index for ambulances.

The ROGS data on ambulances also includes statistics on performance metrics, including service response times (both in terms of time taken to respond to triple-zero calls for assistance and in time taken to physically respond to emergency incidents). It also includes data on the extent to which patients reported a clinically meaningful reduction in pain from paramedic services, as well as survey-based patient experience metrics. These data could potentially be used to underpin a first experimental foray into explicit quality adjustment in the future.

ABS has sourced data from AIHW that covers the activities of the NDA. It contains the number of service users and hours of services received, alongside key variables such as service type, age, and jurisdiction. The dataset also contains staff hours worked. This dataset can be used in combination with ROGS data on expenditure by jurisdiction by service group to estimate output volume growth. Some initial experimental analysis has been undertaken using NDA data, using service type as the key stratification variable.

ABS has also obtained detailed data from the NDIS, but exploration of this data in terms of measuring output volumes is still in its infancy.

The analysis will focus on the specialist disability services provided under the National Disability Agreement (NDA) and the National Disability Insurance Scheme (NDIS). These services encompass disability supported accommodation, respite and community support services such as therapy, early childhood interventions, life skills and case management. 

The NDA, formerly the Commonwealth State and Territory Disability Agreement (CSTDA), is a high-level agreement between the Commonwealth and State governments around the provision of disability services to people with disability. It is a framework that sets out the roles and responsibilities of each level of government in supporting people with disability. NDA data extends back to 2003-04. 

The NDIS began rolling out in 2013-14, with clients of the NDA being progressively transitioned across to the NDIS. This transitioning process is expected to be complete by the end of 2019-20.

Clearly, disability care is an important measurement objective, especially given the expected ongoing increase in demand for assistance following the rollout of the NDIS. However, there are a number of significant measurement challenges.

Disability care is not identified as an industry in the Australian industry classification, nor are disability products identified in the product classifications that underpin Australia's national accounts. It follows that to implement any improvements in measuring disability care in the national accounts and productivity statistics, output will need to be mapped to existing product and industry classifications. This will make it difficult to 'see' disability care in the accounts. However, policymakers, businesses and employees that provide disability services view it as if it was an industry. For research purposes, the ABS will estimate disability care output as if it was an industry in its own right.

As pointed out in the ROGS, there are difficulties in drawing the boundaries between disability care and aged care, especially given they often overlap. Both disability care and aged care have similar goals, but with different programs. The ABS is assuming that the datasets used to measure disability care and aged care accurately reflect the scopes of the two activities, with no additional adjustment required.

Further, there are difficulties in constructing a seamless time series as the rollout of the NDIS is taking place before the NDA closes. Preliminary analysis shows a marked increase in the aggregate provision of disability services early in the life of the NDIS, but care is required to make sure the NDA and NDIS datasets are combined correctly.

The ABS will continue working through the issues raised above with a view to publishing a comprehensive volume index time series representing growth in output of disability services. Owing to the complexity involved, this work is not expected to be completed by June 2020.

Research into enhancing measurement of government-funded aged care services output is still in early stages of development, with the first phase of the work involving understanding the data landscape. The ABS’ investigation into improving this area of measurement will focus on the two most significant streams of aged care services: residential care and home-based care services.

While aged care services are supported by government funding, they are generally delivered by non-government service providers such as religious organisations, charitable organisations, and private-for-profit organisations. In terms of calculating cost weights, it is likely that the weights will reflect government funding only, and not necessarily the full economic cost of service provision.

The National Aged Care Data Clearinghouse (NACDC) holds data on the number of clients, classified by variables such as State, age bracket and type of care, and expenditure data is likely to be sourced from ROGS. Conversations are underway with AIHW and the Department of Health with a view to bringing together an integrated, holistic dataset matching the specifications required.

The ABS aims to construct a volume index of total output provided by the aged care services industry. However, given research for this topic is still in its infancy, the extent to which this index will implicitly capture quality change is still to be determined.

The ABS' investigation into improving its measures of school output volumes covers all streams of school-based education: pre-school, primary, secondary, and special needs education. Government schools, which deliver the majority of these services, are non-market.

The two main building blocks are student enrolment data (on a full-time equivalent basis) and school expenditure data. The enrolment data represents the quantity variable whereas the expenditure data underpins the cost weights.

As outlined earlier in the paper, the output indexes for schools will be built up by school type (pre-school, primary, secondary, and special needs), by jurisdiction (i.e. State/ Territory), and by affiliation (i.e. government / non-government). As the main focus of this work is to enhance measures of non-market activity, no attempt will be made to disaggregate non-government school activity into sub-categorisations such as religious affiliation.

Most of the student enrolments data at the above level of granularity is collected by the ABS¹⁰, except for special needs school education. In terms of compiling comparable indexes for primary and secondary schools, expenditure data presents a challenge, especially for non-government schools. There are a range of data possibilities for expenditure data, including ABS’ Government Finance Statistics data¹¹, statistics from the ROGS, and Australian Curriculum, Assessment and Reporting Authority. The ABS is also examining whether it can repurpose some of its own survey data for this exercise for non-government schools, with a prime candidate being the dataset underpinning the annual Australian Industry publication¹².

Enrolments data by individual school year of enrolment is available for primary and secondary schools, but the ABS is currently unable to stratify by year of enrolment. The lowest level of expenditure data currently available is total primary (which, for most jurisdictions, groups kindergarten to year 6) and total secondary (years 7 to 12). No published information has been found which shows how much each individual year of school education costs relative to other years, and in some cases (especially for schools which provide continuous education from kindergarten to year 12), obtaining a primary/secondary split is not possible.

Without an appropriate data source, the ABS intends to explore possibilities around modelling expenditure by individual school year, as well as enhancing the accuracy of the enrolments data by adjusting it for student absentee rates.

Additional challenges currently being worked through involve accessing expenditure and enrolments data for special needs schools, as well as historical enrolments data for pre-schools.

The ABS plans to extend this analysis to cover all school types by constructing a volume index of total school output. It is expected that indexes for pre-school, primary and secondary education will be completed by June 2020, though refinements around stratification by individual school year and accounting for absenteeism will not be incorporated by this time. Designing an output volume index for special needs schools is likely to extend beyond June 2020.

Research into enhancing ABS' measures of output for the tertiary education industry is in early stages, with the first phase of work involving a sweep of the data landscape. Analysis is focused on the public universities, which the ABS considers to be, on balance, non-market activity. Adult, vocational and community education providers are excluded from this exercise, as are private sector universities.

The volume indicator currently used by ABS is a headcount of the number of students enrolled at university. Opportunities exist for us to refine this metric by:

• stratifying by academic discipline;
• stratifying post-graduate students separately from undergraduates; and 
• valuing the non-tuition output of universities, such as research papers and books.

Universities also engage in R&D partnerships with the private sector. This aspect of university output is difficult to observe and measure, so the ABS will implicitly assume that this strand of R&D activity grows at the same rate as the combined growth of tuition and research output.

At this stage, the most promising option for measuring tuition output is by utilising student load statistics (Equivalent Full-Time Student Load (EFTSL)) from the Department of Education, disaggregated by university.

The most likely path for measuring research output would be the count of published research output, disaggregated by university, also available from the Department of Education.

Both of these outputs will be cost adjusted using university-level expenditure data from the Department of Education. By utilising data on staff hired for teaching versus research purposes, the total costs for each university can be decomposed into a teaching and research component, and these costs will serve as the cost weights for the output index for tuition and research respectively. This will be premised on the assumption that the ratio of teaching to research staff reflects the university’s ratio of teaching to research costs.

This disaggregation of costs will also allow the two volume indexes to be weighted together to derive a volume index covering total university output.

In addition to the above, the ABS is investigating possibilities to obtain expenditure differences across postgraduate and undergraduate programs, which can then serve as an additional dimension for disaggregation and would capture the notion that postgraduate qualifications reflect a higher quality in education output relative to an undergraduate qualification.

The Department of Education has recently commenced publishing expenditure and activity statistics disaggregated by broad academic discipline, but it is limited by the lack of a lengthy time series, and a lack of full coverage. Stratification by discipline could become possible in the future as this dataset matures.

Over the period to June 2020, the ABS will explore stratification options with a view to designing the most appropriate indicator possible. As mentioned in the 'roadmap' paper, the challenges of measuring output volume growth in the university sector are complex, and it is not anticipated that this work will be completed by June 2020.

A further paper will be published by June 2020 which will present some preliminary results, as well as outlining a plan for implementing the new measures in the Australian national accounts and productivity statistics.
For further queries please email <economic.research@abs.gov.au>.

This paper provides an update on ABS research into enhancing measures of non-market output volumes produced by the Health and Education industries. Criteria have been used to select which specific services have been selected for investigation thus far, including:

• the size of a particular health or education service;
• prevalence of market versus non-market activity within a particular health or education service;
• availability of data used to generate volume indices; and
• public policy interest.

Figure 1 shows the majority of Education output (in current prices) comes from Tertiary education (27% of total Education output). Primary education (24%) and then Secondary education (23%) comprise the second and third largest shares. The 'Remaining education services' category includes adult education, vocational education, sports and physical recreation instruction.

For Health services, Hospitals (25%) and Aged care (21%) comprise the largest shares of total output (in current prices). Ambulance services have a relatively small share of total health output at 2%. However this component of the research has been very beneficial in terms of data availability. The 'Remaining health services' category covers activity that is predominantly market in nature, including dental, physiotherapy, general practice and specialist medical and pathology services.