Modernising ABS agricultural statistics

The Australian Bureau of Statistics (ABS) is modernising the way we produce official agricultural statistics.

The ABS has produced agricultural statistics for over a century by collecting data directly from farmers in Agricultural Censuses and large-scale sample surveys.

However, the increasing availability of other quality data, a commitment to reducing reporting burden and declining survey response rates mean it is necessary to change the way in which Australia’s official agricultural statistics are produced.  This need to modernise the production of agricultural statistics is not unique to Australia and around the world other statistical agencies are making similar changes.

The need for this change was first identified in 2015 in the joint ABS and Australian Bureau of Agricultural and Resource Economics and Sciences (ABARES) National Agricultural Statistics Review. 

At that time, Australia’s official agricultural statistics were largely centralised to the ABS and ABARES and produced using surveys.  Today, the growing use of technology and data within the agricultural sector means data generation is now far more widespread across a mix of farmers, digital agriculture companies, industry groups and governments than in 2015.

With more data and organisations involved in the agricultural statistics and data ecosystem, the ABS cannot undertake a modernisation of official agricultural statistics in isolation.  Support from other organisations including the Department of Agriculture Fisheries and Forestry (DAFF) and ABARES will be essential to help access data, develop new data sources and statistical methods and produce complementary statistics that meet evolving information needs.

This paper explores the context for why the ABS is modernising its Agricultural Statistics Program within the broader agricultural statistics and data ecosystem, the decisions already taken and the principles guiding the modernisation program.

The ABS’ Agricultural Statistics Program has existed in many different forms over the years with the statistics and data produced evolving to inform decisions by governments, industry, and farmers to support Australian agriculture.

In 2023, the Australian agricultural sector requires:

• More timely and frequent agricultural statistics with greater regional detail to support response and recovery from localised events like natural disasters and biosecurity outbreaks.
• Improved accuracy and greater coherence between agricultural data from different sources to improve trust in statistics and effectiveness in informing decisions.
• New insights into the performance of the agricultural sector to inform progress on issues such as sustainability measures, emissions reduction, and labour availability.
• Reductions in regulatory and reporting burden (including survey reporting) for farmers and agricultural businesses.

To meet these requirements, significant changes to the way we collect and produce statistics are needed.

To produce detailed regional statistics more frequently than through the 5 yearly Agricultural Census, use of existing public and private sector data sources, including satellite crop mapping, is essential due to the greater coverage they provide.

New insights into agriculture can be created by incorporating digital agriculture technology, which is now generating automated data about agricultural practices and production.

To improve coherence between official statistics and other sources, modelling approaches that integrate multiple data sources are required. Collaborative approaches to developing these models increase trust in statistics and reduce the likelihood organisations will undertake their own surveys that add reporting burden on farmers. 

To reduce reporting burden, using existing data sources is beneficial because they reduce reliance on surveys.  It is becoming increasingly difficult to collect survey information from agricultural businesses. For the 2021-22 financial year, the ABS was only able to achieve a response rate of 68% for its annual Rural Environment and Agricultural Commodities Survey (REACS) compared to a target of 80%.  Of this 68% many survey forms were only partially completed. These lower response rates impact significantly on the quality of data as is evident from the 2021-22 REACS where water use statistics, detailed livestock breakdowns, statistics for smaller commodity types and regional statistics were not able to be produced.

To ensure the ABS can meet the statistical and data needs of the agricultural sector, we are transitioning away from traditional surveys, harnessing the broad range of information available from existing data sources and modernising our production methods.

The scope of the current ABS Agricultural Statistics Program is outlined below with a summary of the major changes being implemented as part of the modernisation program.

These statistics have been available at the national, state/territory and Statistical Area 4 (SA4) levels annually and at the Statistical Area 2 (SA2) level every 5 years. The main data sources used to produce these statistics were the annual agricultural commodities survey, most recently referred to as the REACS, the Value of Agricultural Commodities Produced (VACP) surveys and every 5 years, the Agricultural Census.

Changes as part of the modernisation:

• The ABS has ceased the REACS and the Agricultural Census.

• VACP surveys will cease where there are other data sources to calculate the value of commodities.

• A reduced range of area, production, value, land, and water use statistics will be published for 2022-23 using existing data from public and private data sources.

• These existing public and private data sources will improve the regional detail and timeliness of many statistics.

These statistics provide quarterly information about the production and value of livestock products at the national and state/territory levels.  These statistics are produced using the Livestock Slaughtered Survey, Poultry and Game Birds Slaughtered Survey and until June 2022 quarter, the Wool Receivals, Purchases and Sales Survey.

Changes as part of the modernisation:

• The Wool Receivals, Purchases and Sales Survey was ceased following the June 2022 quarter as other more timely, and detailed administrative data was available to the industry.
• No changes to the Livestock Slaughtered Survey or Poultry and Game Birds Slaughtered Survey are currently planned. These are important inputs into new livestock herd modelling approaches and estimates of the value of livestock commodities which form part of calculating the Australia's Gross Domestic Product (GDP).

The ABS legislation enables the release of lists of agricultural businesses and their characteristics to other organisations for statistical and research purposes.  This supports the ABARES Farm Surveys Program.

Changes as part of the modernisation:

• ABS is investigating ways other data sources can be used together with the Australian Business Register (ABR) to maintain these lists for use by other organisations and to help understand how the number of agricultural business, their sizes, and farm types are changing over time within different regions.

The ABS services customised data requests for agricultural data on a full cost recovery basis which is possible using statistical output from annual agricultural surveys.

To facilitate greater access to microdata for research purposes, an Agricultural Frame and Agricultural Commodities module have been added to the Business Longitudinal Analysis Data Environment (BLADE) accessible in the ABS DataLab. 

A timeseries of agricultural data (The Farm-level Longitudinal Analysis Dataset (FLAD)), which was created with the assistance of ABARES, has also been made available to Australian Public Service staff for specific research into changes in agriculture over time.

Changes as part of the modernisation:

• ABS is investigating ways that the FLAD can be used for a wider range of research purposes and made accessible through the DataLab.

The ABS’ International Statistics Program is responsible for the compilation and release of monthly and quarterly international trade export statistics for the agricultural sector.  These statistics will continue to be released on their current schedule and there are no changes planned for this program.

Agricultural production contributed approximately 2.4% to GDP in the September 2022 quarter.  The ABS will continue to estimate the contribution of agriculture to GDP, using all available information for this estimation, regardless of its data source.

The Census of Population and Housing collects information about the number of people employed in agricultural occupations. This includes detailed geographic information and other demographic and economic statistics.

There are no changes planned to the collection of this information as part of the agricultural statistics modernisation.

Over the past four years the ABS has been working with industry, academic and government organisations to identify and test data sources that will support the production of agricultural statistics. A summary of the key data source types including specific examples of ways they can be used to produce statistics, is presented below.

Levies are collected by the Australian Government Department of Agriculture, Fisheries and Forestry (DAFF) on behalf of industries to support research and development.  Levy payer data is typically captured at the point of sale and includes information about the total leviable quantity and / or the farm gate or sale value of many commodities. In cases where levy payer registers have been established (see below) information about production and business address of producers is also included.

The ABS is authorised to access this data under the Primary Industries Levies and Charges Collection Act 1991. There are legislative requirements including the Census and Statistics Act 1905 to ensure privacy and secrecy of this data when it is used for producing statistics. More information about the approaches that ABS takes to protect the privacy of all agricultural data is described in Protecting the privacy of agricultural data.

Levy Payer data about horticulture is an important component of the Hort Innovation Horticulture Statistics Handbook which provides production and value information for over 70 horticultural commodities.

Levy Payer Registers exist for most broadacre crops. They provide information on the quantity and value of crops sold and the number of businesses selling each commodity including location. This is a key data source to produce regional broadacre statistics. Levy Payer Register data has been successfully used in the production of statistics for both Sugarcane and Canola commodities.

Levy Payer data also exists for Livestock commodities. The Wool levy payer register data has already been used to produce statistics on wool production and the local value of wool sales. Levy payer data also exists for Meat chickens, Dairy, Livestock processing and Livestock transactions.

Mapping crops using a combination of field data and satellite imagery data has enormous potential to provide up to date regional statistics about land area used for different cropping types. The quality of this data has improved significantly in recent years and there is now a range of data available through digital agriculture companies, government agencies and universities.

ABS has established partnerships to use several different crop mapping datasets to produce statistics including:

DAS (Digital Agriculture Services) create crop maps for the 10 biggest winter crops across Australia. This information creates estimates of crop area for each commodity and yield information for wheat, barley, and canola. DAS create this data at a farm paddock level and aggregate it to small areas for the ABS.

Queensland Department of Environment and Science crop map identifies sugarcane cropping areas alongside other winter and summer crops grown within Queensland.

The Applied Agricultural Remote Sensing Centre’s Australian Tree Crop Map Dashboard and Australian Protected Cropping Map Dashboard integrates industry and satellite data, citizen science and extensive field validation to produce detailed crop maps of all commercial Protected Cropping Structures (glasshouses, polyhouses, polytunnels, net and shade houses), Avocados, Bananas, Citrus, Macadamias, Mangoes, Olives, Truffieres and Soybeans.

The National Livestock Identification System (NLIS) records movements of livestock from properties to other properties, feedlots, or processors to enable traceability in the event of an outbreak. This movement information, even at an aggregate level to preserve privacy, would be an important input into a livestock herd model as it would inform where animals were coming from prior to slaughter to allow regional estimates to be created. The ABS considers the utilisation of NLIS data would be highly beneficial to the modernisation of official agricultural statistics.

Biosecurity information also utilises Property Identification Codes, which are unique codes that link livestock movements to specific land parcels.  This information could help to provide an estimate of the land area being used for livestock grazing. The ABS is discussing access to this information with state and territory governments.

Many industries have detailed information about commodity production, location and value collected through administrative systems or surveys of farmers. The quality of this information varies considerably from comprehensive data on the entire industry to small surveys that can provide information to support a modelling process where other data is not available. Examples of data ABS are already using include:

Australian Sugar Milling Council (ASMC) industry data is sourced from sugar mills. This includes statistics on sugarcane area harvested, tonnes of sugarcane crushed, estimated value of sugarcane and the number of businesses within 5 sugarcane producing regions.

SunRice provide aggregate industry data at the SA2 level about the area, production, and value of rice as well as business numbers. This covers more than 95% of Australia’s rice production and is complemented by Levy payer information.

ABS is currently exploring ways that data from farm machinery and farm software could be used to improve agricultural statistics without adding reporting burden for farmers.

Modern farm machinery such as harvesters produce data about area of crops sown and harvested, including yield. With explicit permission of individual farm businesses this data could be used alongside other crop mapping and levy payer data to improve the accuracy of regional statistics.

Farm software integrates with machinery data and could be another way for farmers to help provide information that supports their industry without filling in a survey. Farm software is increasingly being used within the livestock industry to inform farm practices, collecting information such as number, weight, age, and location of livestock.

The following summary provides guidance on the agricultural statistics the ABS believes are achievable to release for the 2022-23 financial year. The summary also identifies where increased detail may be produced if certain barriers, such as access to specific data sources, are overcome.  The statistics for the 2022-23 financial year will be released before July 2024.

The methodologies for producing broadacre crop statistics and access to appropriate input data sources have been in development for some time.  Table 1 demonstrates that the production of regional statistics for the highest value broadacre crops is achievable for the 2022-23 financial year.

Horticulture Innovation Australia (HORT Innovation) compile production and value statistics for over seventy horticulture commodities at the national, state and territory levels in the Australian Horticulture Statistics Handbook.

The ABS is working with Hort Innovation, ABARES, industry groups and state and territory governments to further develop regional horticulture statistics.

For the 2022-23 financial year, rather than duplicating statistics already available, the ABS will produce regional statistics that align with Hort Innovation estimates for a small number of horticulture commodities with the aim of further developing this approach through collaborative partnerships.

Livestock herd numbers for cattle, sheep, goats, pigs, and poultry are planned to be produced for the 2022-23 financial year. The extent to which this is possible will depend on access to new data sources and the successful development of herd modelling approaches currently being refined in partnership with industry experts.

More detailed breakdowns of the herd numbers, including regional detail, are future aims and unlikely to be produced for 2022-23.

Business count data for livestock businesses may be able to be produced using combinations of Levy, ABR and biosecurity data but these will not be available for 2022-23.  

Livestock value information will continue to be produced through the existing quarterly ABS surveys.

Land use data for the 2022-23 financial year is likely to be limited to broadacre cropping area identified through satellite crop mapping data sources.

Land area used for livestock may be able to be derived from administrative land data linked to biosecurity information, however, this has not been tested and is unlikely to be available for 2022-23.

Methods are being investigated to understand water use for different commodity groups, however, it is unlikely that these will be ready for producing statistics for 2022-23.

Following the 2015 National Agricultural Statistics Review (NASR), the ABS began investigating new data sources to support agricultural statistics with less burden on farmers.

In 2019, the ABS worked with Deakin University to identify alternative data sources that would enable surveys to be stopped or reduced. This work identified a range of existing data sources, but it also noted that most of these would not directly replace individual survey questions. Instead, it would be necessary to use an approach that combined elements of multiple data sources together to produce agricultural statistics. This was an important finding as it opened the door to considering how new methods could combine data sources to make the most of existing data.

The ABS and ABARES worked with industry experts through both the Red Meat and Grains Statistics working groups on ways to use multiple data sources to produce statistics.  This collaborative approach built an understanding of the different data sources that could input into statistics. It also built trust in the resulting statistics because they aligned with other data sources used within the industry.

In partnership with these industry working groups the ABS published experimental agricultural statistics for sugarcane and canola. This demonstrated that using existing data sources could produce accurate, timely statistics with greater regional detail and less burden on farmers and agricultural businesses. These experimental estimates have been important for starting to develop methods and systems that can be applied across a range of cropping commodities to produce statistics. This means that ABS is well positioned to quickly expand this approach where data is available.

The ABS Data Quality Framework uses seven dimensions of quality: institutional environment, relevance, timeliness, accuracy, coherence, interpretability, and accessibility.  The ABS aims to optimise the different dimensions of quality for the statistics it produces.

Experimental releases developed in partnership with industry have demonstrated that using a combination of data sources can strengthen coherence and improve timeliness and regional detail.

When producing experimental statistics, the ABS has deliberately created these for time periods that overlap with existing survey-based data to better understand and assess accuracy. These comparisons show the new data sources are mostly within the 95% confidence interval range of the survey-based statistics. 

In some cases, there will be conceptual changes with the use of new data sources. An example of this is the scope of survey outputs which included businesses with an estimated value of agricultural output (EVAO) of $40,000 or more. Non-survey data sources do not have this restriction which means they measure the full extent of Australian agriculture. Comparisons with 2020-21 Agricultural Census data will help to understand the impact of this conceptual change on accuracy and coherence.

The ABS takes Privacy very seriously. The ABS Privacy Policy for Statistical Information sets out how we handle personal information collected for the purpose of producing official statistics.

As part of the move to using new data sources for agricultural statistics the ABS has also consulted the National Farmers Federation (NFF) to ensure that our practices align with the Australian Farm Data Code.

Industry collaborations enable the ABS to talk directly to industry representatives and farmers.  In speaking directly with industry, we can discuss how we are protecting the privacy of data that inputs into official agricultural statistics and how additional protections are applied to regional data based on this feedback.

Collaborative partnerships between data users and custodians are required to find solutions to a range of challenges to the successful transition to a modernised agricultural statistical and data ecosystem. The ABS is just one part of this ecosystem and without support it will not be possible to produce the statistics needed to inform Australian agriculture.

Statistical working groups have been a fundamental part of early successes in producing high quality agricultural statistics from new data sources. Continued involvement by industry experts from a range of organisations will be vital to further develop fit for purpose statistical data using the best available data sources.

Addressing current barriers to access key data sources is vital to modernising how agricultural statistics are produced and better informing biosecurity risk management and responses to natural disasters.  Data access barriers may be overcome through constructive discussion with data custodians to better understand the barriers in place and what kind of privacy protections, for example, can be established to safely use data sources.

Partnerships are also required to efficiently develop national data assets that can be used by a range of organisations to support an efficient and sustainable agricultural sector. These assets could be delivered by other organisations as part of a broader agricultural statistics and data ecosystem. One example where this could be done is through the development of a unified national crop map to bring together and further develop the range of existing satellite crop mapping into a single source of truth. A unified national crop map would integrate into a national land use map and support a range of functions including emergency response and resilience, biosecurity planning and act as an input into other agricultural statistics.

With the development of new data sources, evolving information needs and new ways to produce statistics, a collaborative approach between organisations within the agricultural statistics and data ecosystem will be needed to deliver the different components of agricultural information in the most effective way. 

The ABS wants to hear from you. If you have ideas about potential new data sources to support agricultural statistics, you want to be part of the collaborative development of new methods or you have questions about elements of the modernisation program, then please get in touch with us using agriculture@abs.gov.au.