The role of the Australian Bureau of Statistics in data governance and stewardship in Australia - a regulatory perspective

This article will be presented to the United Nations Economic Commission for Europe,
Conference of European Statisticians, Expert Meeting on Modernizing Statistical Legislation                                      
18 – 19 November 2021, Geneva

Authors: Julie Greenall-Ota, Melissa Gare, Fiona Carnes

1.    The Australian Bureau of Statistics (the ABS) is the central statistical authority for the Australian Government and, by legal arrangements, provider of statistical services to Australian State and Territory Governments. The purpose of the ABS is to inform Australia’s important decisions by delivering relevant, trusted and objective data, statistics and insights.

2.    This paper explores the role of the ABS in data stewardship and governance in Australia in the context of current and past regulatory initiatives. The concept of data stewardship in the current data environment was recently considered by the United Nations Statistical Commission High Level Forum. The Commission summarised data stewardship to include the notions of deliberate care and management of data, maximising the value of data and statistics, and building trust and confidence in the data system ensuring government data is available in an open and effective way.¹ In a similar vein, data governance and capability is increasingly recognised as a driver of public trust in government.² As custodian, steward and user of public data, the ABS is committed to building and maintaining public trust in an increasingly complex data environment. The ABS understands that maintaining such public trust in the collection, sharing, and release of public data is a vital part of the public data system.

3.    This paper commences with an overview of the contemporary Australian Government statistical and data landscape focusing on Australian Government initiatives and investments that have set Australia on the path to transformation into a new digital and data economy.

4.    With this background, the paper then examines a current legislative initiative in public sector data sharing and the contribution of the ABS to this initiative. This is followed by a close examination of two formative past legislative reforms that have driven the role of the ABS as a national statistical office and data steward.

5.    The paper then considers how the ABS has built upon those past regulatory initiatives to maximise the value of data – firstly through various actions to expand user access to micro data over time and secondly through developing a data integration framework and significant national integrated data assets. These efforts have validated the role of the ABS in Australia’s statistical landscape and demonstrated the strong commitment of the ABS to being a trusted partner and data steward in the statistical and data landscape today and into the future.

6.    The data holdings of the Australian Government are a strategic national resource that are vital to evidence-based decision-making and increasing productivity. Unlocking data enables program evaluation, informed policy, risk-based regulatory processes and effective services for all Australians and is critical in informing timely government responses during crises including natural disasters such as bushfires and floods as well as the recent global pandemic.³

7.    The Australian Government’s release of the Public Data Policy Statement in 2015 provided a clear mandate for Australian Government entities to optimise the use and reuse of public data, release non-sensitive data, and collaborate with the private and research sectors to extend the value of public data for the benefit of Australia.⁴

8.    This was supplemented by the Australian Government’s launch of the Public Sector Data Management Project in the same year which established a roadmap and implementation projects to make better use of public data under the leadership of a number of cross-agency forums.⁵ Through these forums, the ABS plays an active role in supporting Australian Government entities to overcome barriers to data sharing and integration.

9.    In 2017 the Australian Government invested in a significant three-year project, the Data Integration Partnership for Australia (DIPA), a whole-of-Government collaboration involving over twenty Australian Government agencies that worked together, to improve technical data infrastructure and data integration capabilities across the APS and raise the profile of the value to be drawn from integrated assets to inform complex policy questions. The ABS played an integral leadership role and delivered core technical infrastructure to support the DIPA.⁶

10.    In the same year, a public inquiry by the Productivity Commission (an independent advisory body of the Government) was conducted into ways of improving the availability and use of public and private sector data.⁷ As part of the response to the inquiry, the Australian Government appointed an interim National Data Commissioner and established the Office of the National Data Commissioner in the Department of the Prime Minister and Cabinet, tasked with implementing a data sharing framework to improve access to and the re-use of public sector data, while maintaining data privacy and security.⁸ ⁹ The work of the Office of the National Data Commissioner led to the introduction to the Australian Parliament of the landmark data sharing legislation discussed further below.

11.    A further Australian Government initiative in 2020 was the launch of the Data Professional Stream by the Australian Public Service (APS) Commission to increase the capability of Government employees working in critical disciplines or function areas. The Australian Statistician, Dr David Gruen, was announced as the inaugural Head of the Data Professional Stream to set direction and provide guidance, leadership and stewardship for the stream, and represent and promote the stream to secure recognition and value across the APS.¹⁰

12.    The Data Professional Stream is underpinned by the Data Professional Stream Strategy, developed by the APS Commission in consultation with the ABS and aimed at lifting the capability of the APS workforce to generate deeper insights to inform decision-making in policy development, programme management and service delivery and support innovative ways to collect, share and use data responsibly to meet the needs of the day.¹¹

13.    Building on these Australian Government initiatives, the Australian Government announced the Digital Economy Strategy in 2021 encompassing a number of whole-of-Government initiatives and investments to support digitisation and the new data economy in Australia and a commitment to develop an Australian Data Strategy.¹²

14.    In December 2020, the Data Availability and Transparency Bill 2020 (the Bill) was introduced to the Australian Parliament.¹³ If enacted, the Bill will establish a new data sharing scheme authorising and regulating access to Australian Government data. The proposed scheme will provide an additional legislated pathway to support the sharing of public sector data for certain specific purposes in line with community needs and expectations.

15.    The Bill currently before Parliament:

1. Authorises data custodians of public sector data to voluntarily share data with accredited users in accordance with specific authorisations, purposes, principles and agreements. This authorisation overrides other laws that would otherwise be contravened by the data sharing, with some exceptions.
2. Specifies the permitted purposes for data sharing under the scheme: delivery of government services, informing government policy and programs, and research and development. Precluded purposes include enforcement-related purposes.
3. Specifies responsibilities imposed on data scheme participants. Data custodians are responsible for assessing each data sharing request from an accredited user and deciding whether to share their data if satisfied the risks can be managed.  Data custodians cannot be required to share data under the scheme.
4. Establishes and specifies the functions and powers of the National Data Commissioner as the regulator of the scheme.
5. Establishes and specifies the functions and membership of the National Data Advisory Council.
6. Establishes the regulation and enforcement framework of the scheme.¹⁴

16.    The proposed scheme enshrines into legislation an additional method for public sector data sharing. This would enable the ABS to use the scheme to supplement the current disclosure and access mechanisms available to the ABS in the Census and Statistics Act 1905 (the Census and Statistics Act).

17.    Drawing on the technical and practical expertise of the ABS in data governance and stewardship, the ABS has worked closely and collaborated with the Office of the National Data Commissioner, including through secondment arrangements of ABS officials, to provide technical advice and support on a number of key aspects of the proposed new data sharing scheme.

18.    In particular, the ABS substantially contributed to the development of the Data Sharing Principles which form an integral part of the governance of the proposed data sharing scheme, and were developed based on the Five Safes Framework, an international multi-dimensional standard used by the ABS for managing disclosure risks.¹⁵

19.   The ABS also contributed to the development of the Accredited Data Service Provider framework drawing on lessons learned from the Accredited Integration Authority framework, which forms part of the governance arrangements for entities including the ABS offering advanced data integration services involving Australian Government data, as further described under ‘Data integration – developing enduring assets and a governance framework’ below.

20.    In December 1973, the Australian Prime Minister, the Honourable Gough Whitlam appointed a committee chaired by Professor L.F. Crisp (the ‘Crisp Committee’) to advise on the general arrangements which would enable the Government to integrate the various departmental services providing data for its social and economic policies and ensure Government decisions were based on relevant, reliable and comparable statistics.¹⁶

21.    While the Census and Statistics Act had been passed almost 70 years earlier providing the Australian Statistician with the authority to conduct statistical collections including the Census, the final report of the Crisp Committee found that Australia had a fragmented and uncoordinated statistical system, with no central statistical authority, no effective machinery for setting priorities and severe secrecy provisions that resulted in limited access to data in a form suitable to the needs of potential users.¹⁷

22.    Two key recommendations of the Crisp Committee laid the foundations for legislative reform leading to a greater coordination role for the ABS within the Australian statistical landscape and maximised access, value, and utilisation of statistical data.

1. The Australian Bureau of Statistics Act 1975

23.    The first key recommendation was to establish the ABS as the central statistical authority with full statutory powers, administratively independent of any Government department to ensure policy neutrality. The Australian Statistician would be a statutory appointee vested with the powers of a head of Government department. This led to the passage of the Australian Bureau of Statistics Act 1975 (the ABS Act) to complement the existing Census and Statistics Act.

24.    Under the ABS Act, the ABS is authorised to provide the following functions:¹⁸

1. to provide statistical services to the Australian and State Governments
2. to collect, compile, analyse and disseminate statistics and related information
3. to ensure coordination of the operations of other official bodies in the collection, compilation and dissemination of statistics and related information – with regard to certain matters including the avoidance of duplication in collections by official bodies, the attainment of compatibility and integration of statistics compiled by official bodies and the maximum possible utilisation, for statistical purposes, of information and means of collection available to official bodies
4. to formulate, and ensure compliance with, standards for the carrying out by official bodies of operations for statistical purposes
5. to provide advice and assistance to official bodies in relation to statistics
6. to provide liaison between Australia and other countries and international organisations in relation to statistical matters.

25.    The establishment of the ABS as the central statistical authority enabled the Australian Statistician to play a wider coordination and leadership role within the statistical system, in addition to steering the ABS to better meet the emerging statistical needs of users.

2. Census and Statistics Amendment Act (No2) 1981 and the Statistics Determination 1983

26.    The second key recommendation of the Crisp Committee was amendment of the secrecy provisions of the Census and Statistics Act to provide maximum access to statistics while fully safeguarding the confidentiality of information about individual persons and businesses. The provisions in force at the time were viewed as preventing the release of information which could not, in any meaningful sense, be regarded as confidential.¹⁹

27.    The Census and Statistics Amendment Act (No2) 1981 was passed by Parliament with the main purpose of authorising, subject to precise safeguards, the disclosure of a wider range of information than was previously enabled under the Census and Statistics Act. The amendment provided that the relevant Minister may, by means of a written determination, authorise the Australian Statistician to disclose specific kinds of information under specific conditions. The amendment precluded such determination from authorising the disclosure of information of a personal or domestic nature in a manner that is likely to enable the identification of any person.

28.    Based on the amendment, the first determination instrument was made and came into effect on 1 March 1983 - the Statistics Determination 1983. This enabled the ABS to expand the types of information that could be disclosed either publicly or on a limited basis, subject to conditions of use in certain circumstances. One of the key disclosable types of information under the Statistics Determination 1983 was unidentified information, also referred to as unidentified microdata.²⁰ The disclosure of this information, subject to conditions of use, provided a mechanism for greater access to information for non-ABS users to create statistics from information held by the ABS for decision-making processes of governments, businesses, and individuals.

1. Enabling increased access to microdata to meet user demands

29.    Utilising the Statistics Determination 1983, the ABS worked with prospective users to seek ways to maximise the full potential of the microdata. In 1985, the ABS released the first unidentified microdata under the Statistics Determination 1983 from the Australian Health Survey.²¹ The high costs to produce and access the files and the restrictions on the level of data detail that could be released to comply with the requirements of the Statistics Determination 1983 presented challenges for potential users.

30.    An amendment to the Statistics Determination 1983 in 1994 allowed the Australian Statistician to place legally enforceable conditions on the use of microdata released, rather than solely reducing the level of detail made available to users. This enabled the ABS to relax several restrictions and increase the level of detail that could be disclosed.

31.    Despite this amendment, microdata still needed to be disclosed in a manner not likely to enable the identification of a particular person or organisation to which it related. This proved particularly challenging for the disclosure of microdata about organisations, with many industries dominated by a small number of large recognisable businesses. This resulted in limited disclosure of unidentified microdata about organisations.

32.    Over time the ABS continued to seek opportunities to expand the accessibility of microdata within the constraints of the Statistics Determination 1983. With costs to access presenting a barrier for researchers, in 1998 the ABS entered into an agreement with the Australian Vice Chancellors’ Committee under which unidentified microdata were made available for research and teaching purposes to participating Australian universities at a fixed cost per year. This allowed approved academics from across Australia to access a wide variety of unidentified microdata and increased the use of ABS data significantly.

33.    Harnessing technological advances was another way in which the ABS was able to develop a safe and enabling environment for microdata access. While historically, users were provided with physical copies of released unidentified microdata for use in their own IT environment, significant advances in technology and a higher sophistication of users, steadily increased the theoretical risk of identification from releasing the data in this way. In 2003, to build trust and confidence in the disclosure of unidentified microdata, the ABS launched the Remote Access Data Laboratory. The Remote Access Data Laboratory required users to register and remotely submit requests that were executed against unidentified microdata held by the ABS with only aggregate results provided to researchers. This enabled more detailed microdata to be analysed in a controlled and monitored environment.

34.    In a further step that demonstrated the commitment of the ABS to broaden the availability of data at scale for statistical purposes, while ensuring the necessary safeguards and protections remained in place, in 2009 the ABS released a new analytical tool for users to analyse 2006 Census data. ‘TableBuilder’ enables non-programmers to use a menu-driven interface to produce tables that are automatically confidentialised for release without revealing any underlying data to the user. TableBuilder was subsequently expanded to include the production of graphs and maps and is widely used today with over 90 datasets currently available from a range of collections.²²

35.    With the expectations of the Australian Government and the research community for greater access and utilisation of microdata increasing over time, the limitations of the Statistics Determination 1983 became more evident. The expiry of the Statistics Determination 1983 in 2018 provided the opportunity to remake a modern Determination instrument with a more flexible scheme under which the ABS could release information in sufficient detail to meet the analysis and decision-making needs of governments, businesses and other stakeholders.

36.    A new Determination instrument was made in 2018 (the Census and Statistics (Information Release and Access) Determination 2018). The new microdata provisions provided the ABS with the authority to disclose information to users about businesses and organisations in a way that may identify information about them, under a written legally enforceable confidentiality agreement. By adopting the ‘Five Safes Framework’²³, the ABS was also able to place appropriate controls on the way data was accessed in addition to protections on the data itself, thereby enabling users to analyse more detailed unidentified microdata about organisations.

37.    Capitalising on the legislative changes relating to microdata, and further technological advances, in 2019 the ABS decommissioned the Remote Access Data Laboratory and expanded use of the DataLab – an alternative analytical environment established by the ABS in 2015. The DataLab has continued to evolve and is now a cloud-based secure analytical platform for high-end users seeking to undertake virtual real-time complex analysis of detailed unidentified microdata.

2. Data integration – developing enduring assets and a governance framework

38.    In 2010, with strong leadership from the ABS, the heads of all Australian Government agencies and the APS Commission endorsed a set of principles to govern the integration of Australian Government data for statistical and research purposes, as well as a set of governance and institutional arrangements to support these principles.²⁴ These ‘Commonwealth arrangements’ aimed to maximise the use of existing data resources, protect privacy and confidentiality, provide transparency and deliver public benefit.

39.    An important part of the governance and institutional arrangements is the ability to hold accountable a government or non-government agency, known as an Integrating Authority, for the safe implementation of a statistical data integration project involving Australian Government data.

40.    For projects assessed as posing a high risk based on the’ Risk Assessment Guidelines’ of the Commonwealth arrangements, the Integrating Authority must be accredited.²⁵ The accreditation scheme is an administrative arrangement designed to recognise a relatively small number of agencies that have the requisite expertise, skills, knowledge, infrastructure and secure environment to undertake high risk data integration projects involving Australian Government data for statistical purposes. The accreditation process requires satisfaction of a number of criteria, audit and technical assessments and approvals.²⁶ The ABS was the first Integrating Authority to be accredited in 2012.²⁷

41.    In 2013, the Australian Government issued a ‘legal and policy framework’ which requires a data custodian to assess whether an Integrating Authority:

1. Is authorised, either by legislation or by obtaining the consent from the data provider, to access and use the data custodians’ identifiable data for the proposed purpose, prior to undertaking a project.
2. Has an appropriate policy framework in place to prevent disclosure of identifiable information, other than where allowed by law.²⁸

42.    As an Integrating Authority, the ABS has worked co-operatively with data custodians to provide support through complex approval processes and legislative amendments on a large number of data integration projects and facilitate the collection of data by the ABS under the Census and Statistics Act.²⁹

3. Development of enduring integrated assets – MADIP and BLADE

43.    The ABS has developed two significant integrated data assets for Australia over the last decade, in partnership with a number of Government agencies - the Multi-Agency Data Integration Project (MADIP) and the Business Longitudinal Analysis Data Environment (BLADE).³⁰ Through these major initiatives and partnerships, the ABS has sought to systemise and scale efforts to create additional value in public sector data.

44.    For each of these projects, the ABS has developed core technical infrastructure including the technical tools to integrate and link data. This technical infrastructure, coupled with the past regulatory reforms relating to microdata access, the ABS investment in microdata access mechanisms such as the Remote Access Data Laboratory and the DataLab, and Australian Government projects such as DIPA in more recent times, has enabled the ABS to provide a safe analytical environment from which researchers can safely and securely analyse microdata in MADIP and BLADE for the benefit of government, the economy and society.³¹ Information on all integrated data assets and integration projects is publicly available on the MADIP and BLADE project registers.³²

45.    In relation to MADIP, as an integrated and secure system of personal information across health, education, government payments, income and taxation, employment, and population demographic (including the Census) over time, it is critical that there is appropriate oversight of legal obligations on collection, use and disclosure of data and privacy obligations regulating personal information.

46.    The ABS has developed a robust governance framework under which the MADIP Project Board, composed of senior representatives from partner Government agencies, sets the strategic direction of MADIP and monitors implementation. The ABS, as the Accredited Integrating Authority, is vested with operational responsibility for MADIP. As part of the governance framework, all integration projects are assessed for public benefit and ‘Privacy Impact Assessments’ are conducted, where appropriate, to identify and manage privacy impacts of operating MADIP.³³ Each of these elements of governance and accountability are important in ensuring a safe and enabling environment and demonstrate the commitment of the ABS to building and maintaining public trust, which is at the core of the public data system.

The data policy and regulatory environment in Australia is in a period of significant transformation. The Australian Government has introduced a range of policy initiatives in the statistical and data landscape, and recently proposed a regulatory initiative in the form of a new legislative data sharing scheme. By looking back to past legislative initiatives that formed the ABS as the central statistical authority in Australia and the role of the ABS in maximising the value of data over time through expanding access to microdata and developing significant national integrated datasets, the ABS has validated its role in Australia’s statistical landscape and demonstrated it is a trusted partner and data steward creating efficiencies and benefits for society.