Statistical review of process for providing regional enrolment projections for electoral redistribution purposes

In January 2024, the ABS confirmed an error in the electoral enrolment projections it supplied to the Australian Electoral Commission (AEC) for use in the Victorian and Western Australian electoral redistribution processes. The ABS subsequently located the source of the error and supplied the AEC with corrected enrolment projections data. The AEC has indicated that the ABS supplied the corrected projections in time to enable the redistribution to be completed within the originally specified timeframe.

Following the discovery of the error, the Australian Statistician commissioned an independent review of the ABS processes for producing enrolment projections, which is being published in full today. The ABS will implement all recommendations in the review. 

Mr Dennis Trewin AO, FASSA
Former Australian Statistician

Dear Dennis,

Thank you for providing this report on your Statistical Review of the process for providing regional enrolment projections for electoral redistribution purposes.

I would like to formally accept the report and its findings, and to thank you for undertaking the review. I intend to implement the full set of recommendations.

The ABS has now improved the quality management procedures for producing enrolment projections in line with your report. Your timely and expert advice reassures me that these improvements will ensure similar errors are not repeated in the future.

I welcome the recommendations and insights you have made in Section 5 of the report to strengthen and reinforce the importance of the ABS’ quality management arrangements more generally. I am actively working with ABS management to ensure these insights are embedded across all areas of ABS statistical production to further improve our quality management practices.

Thanks again for undertaking this work which will support the ABS efforts to ensure the quality of its outputs, and the confidence and trust of data users.

Yours sincerely,

Dr David Gruen AO
Australian Statistician
26 March 2024

Dennis Trewin AO, FASSA

Enrolment projections provided to the Australian Electoral Commission (AEC) for redistributions currently underway in Victoria and Western Australia contained errors in the projected estimates of enrolments at the regional level. This was discovered during the AEC public consultation. Following investigation, the ABS resolved the errors and supplied corrected projections to the AEC. 

The error was made in the SAS program written to undertake the task but was not detected until after the AEC notified the ABS of concerns they had received about the accuracy of the enrolment projections. The objective of the review was to look at the processes, not the methodology, for compiling regional population projections for electoral redistribution purposes. In essence, the report is about why the error was not picked up earlier and the steps that should be undertaken to avoid similar errors in the future.

My main findings were as follows. My quantitative comments are based on Victoria as the error had more severe consequences there than in Western Australia.

1. The SAS program prepared to compile the regional enrolment projections was inadequately tested.
2. Consistent with good practice, a variety of output edits were undertaken before delivery to AEC. However, an important output edit, that would have clearly identified the presence of an error, was not undertaken. For Victoria, it showed that improbably large number of SA1s had a projected change in the electoral population of 8 to 10 per cent over the 5 years to 2028. Virtually no SA1s had a lower projected change, and none had a higher projected change. Population Statistics Branch included this additional output edit in their revised electoral projections and will do so for future projections made for electoral purposes. 
3. One of the output edits involved correlative analysis at the SA1 level between the projections of the population aged 18 years and over and the enrolment projections. This was also undertaken at the SA2 and Electoral Division levels. This showed lower than usual correlations, but this was incorrectly attributed to the impact of COVID on the Medicare data which is the major data source used for estimating change in regional populations.
4. There were no face validity checks of outputs by someone familiar with sub-state population trends in the States. For example, I know the population trends in Victoria well enough to have identified that there was a problem. The Regional Population Unit within the ABS believe they have sufficient knowledge of population trends in the States and Territories to play this role in the future. Furthermore, Population Statistics Branch plans to recommence briefings (similar to those conducted up until 2020) of AEC Divisional Managers before they are publicly released.
5. A number of the Suggestions on the AEC website raised concerns with the Victorian enrolment projections, but these were not monitored by the ABS, nor did AEC raise concerns with the ABS until a week or so before the letter from the Australian Electoral Commissioner on 21 December. The AEC first published the enrolment projections on 25 October and Suggestions started flowing shortly afterwards, some of which raised concerns about the projections. (The concerns were mentioned in Antony Green’s blog of 27 November). It is likely that the error could have been identified in mid-November.
6. Once the ABS became aware of the error, it responded reasonably quickly (10 working days for Victoria and 12 working days for Western Australia). 
7. I believe human resource issues are a partial explanation. The resources available to Population Statistics Branch have remained stable although turnover has been high because of the increased demand for demography skills. Furthermore, the workload has increased significantly over this time. It is one of areas of the ABS where errors can cause significant damage to the ABS reputation.

The main quality gate is at the right place in the process for compiling electoral enrolment projections. However, the set of output edits performed at this gate should be extended to ensure they are more likely to identify possible errors. This was done for the revised estimates and confirmed that the error would have been found with the additional output edits.

Furthermore, Population Statistics Branch should introduce arrangements for reviewing the face validity of the estimates. This needs to be someone who is familiar with the sub-State population trends in the State or Territory. The Regional Population Unit believes it has this capability. Also, as proposed by Population Statistics Branch, briefings of AEC Divisional Managers should recommence prior to AEC making the enrolment projections public.

For future re-distributions, the ABS should monitor the Suggestions on the AEC website for criticisms of the enrolment projections. AEC should be encouraged to contact the ABS if they hear or observe criticisms.

Population Statistics Branch resources are stretched. Although resource levels have remained constant, staff turnover has increased, and the demand for statistical population statistics has also increased. It is a very important area of the ABS and there should be a review to ensure it has all the resources it requires.

The ABS and other official statistical agencies rely a lot on user developed systems using user-oriented tools (e.g. SAS, spreadsheets). Systems Development Methodologies (SDM) are used for larger systems using IT professionals. Standards should also be produced for systems developed using user-oriented tools. These would have a lighter touch than a full SDM but would include requirements to test the accuracy of the system and provide sufficient documentation to enable someone else to use the system.

The ABS Quality Management arrangements involve a system of quality gates where intermediate and final outputs must be 'signed off' before they pass the next gate. This is a good arrangement especially if the quality gates are placed where they will be most cost effective. However, their importance needs to be reinforced, especially for non-regular statistical outputs, and there may need to be some updates to the Quality Management Framework given it has been in place for nearly 15 years.

A face validity or ‘sniff test’ should be applied to all ABS outputs but especially those of high importance. I would include electoral projections for redistribution in that category. Where possible, this should be someone not closely involved in the compilation (‘a fresh set of eyes’). Most often it will be a more senior officer involved in the publication sign-off process but there may be others who have sufficient knowledge of the subject matter to provide some insights.

In my experience, negative feedback from sophisticated users on accuracy issues should be taken seriously. They are right more often than not. This is certainly the case for electoral projections where several of the political parties and a number of psephologists study these numbers in great detail. (Even if their concerns are invalid, they should be provided with an explanation to maintain their confidence in the figures.)

Population Statistics Branch has or plans to make several adjustments to the quality management procedures for the processes to develop enrolment projections for electoral redistribution purposes. If they are successfully implemented, similar errors should not be repeated in the projections provided for future redistributions.

The acceptance letter and full report can be downloaded below. An accessible version is available upon request from demography@abs.gov.au