1520.0 - ABS Data Quality Framework, May 2009  
ARCHIVED ISSUE Released at 11:30 AM (CANBERRA TIME) 04/05/2009  First Issue
   Page tools: Print Print Page Print all pages in this productPrint All

APPLYING THE ABS DATA QUALITY FRAMEWORK

The ABS DQF is a general framework to enable a comprehensive and multi-dimensional assessment of the quality of a statistical dataset, product or release. It is intended that the framework enable data users and providers to:

  • assess the quality of a data item or a collection of data items, with reference to the user's specific purpose and requirements; and
  • design a statistical collection or product which is fit for purpose.

While ABS advises consideration of all seven quality dimensions, it is a matter of judgment as to the relative importance of each. We encourage users and producers to consider which quality dimensions are most relevant and important for their particular purpose. Quality relates to the fitness for purpose of the data or statistical product, and as purpose will vary among users, different users may make different assessments of the same product's quality. For example, if the credibility and trustworthiness of the data source are particularly important, then a careful examination of the Institutional Environment dimension will be especially important and this may have more weight in making an overall quality assessment. Alternatively, if a key purpose is to compare and contrast data, then the Coherence dimension will be particularly relevant.


Application of the ABS DQF by users of statistics

ABS recommends that when assessing the quality of a data item, dataset or other statistical product, a quality statement is developed. A quality statement is a presentation of information about the quality of a data item or a collection of data items, using the ABS DQF. The purpose of quality statements is to clearly communicate key characteristics of the data which impact on quality, so that potential users can make informed decisions about fitness for use. Quality statements should report both the strengths and limitations of the data.

Quality statements vary in length and detail, depending on the audience and medium for release. For example, the ABS has produced specific quality statements based on statistical releases called "quality declarations". Quality declarations are succinct summaries which quickly communicate key statistical quality messages, as well as providing links to more detailed information about statistical output. ABS quality declarations are designed primarily for electronic dissemination, hence their short length, and they enable layering of information in a web environment whereby each successive layer contains more detailed information. Quality declarations complement, but do not replace, the more comprehensive and complete ABS quality statements that currently exist (e.g., explanatory notes, and concepts, sources and methods documents).


Application of the ABS DQF by producers of statistics

The focus on the fitness of statistical information has emphasised the need to build quality into the production and delivery processes of collection agencies. The ABS recommends that producers of statistics consider the seven quality dimensions before designing collections, collecting statistics and producing outputs. This approach can enable informed decisions about factors including appropriate methodology, desired outputs and their accessibility, the coherence of the collection in relation to other collections or products and the relevance of the collection given its purposes.

Some suggested principles for managing each quality dimension are provided below.

Institutional environment

Collection agencies should build a culture that focuses on quality, and an emphasise on objectivity and professionalism. Adequate resources and skills should be made available for the purpose intended. Cooperation of respondents can be encouraged by providing appropriate legal mandate and guarantees.

Relevance

To be relevant, the collection agency must stay abreast of the information needs of its users. Mechanisms for doing this include various consultative and intelligence-gathering processes, and regular stakeholder reviews.

Timeliness

The desired timeliness of the information derives from considerations of its main purposes: the period for which the information remain useful depends of the rate of change of the phenomenon being measured, the frequency of measure and the immediacy of the response that users may want to make based on the latest information. In addition to considering these aspects when planning target data release dates, consideration needs to be given to the capability of the organisation to produce the statistics within the given time frame. This capability includes staffing resources, system requirements, and the level of accuracy required of the data. The release of preliminary data followed by revised and final figures is often used a strategy for allowing less accurate data to be available sooner for decision making, with the subsequent release of more complete data occurring at a later stage.

Accuracy

Explicit consideration of the trade-offs between accuracy, cost and timeliness is important during the design stage. The coverage of the target population that can be achieved by the data collection strategy should be assessed. Proper testing of the instruments for data collection will ensure the reduction of response errors. Adequate measures have to be in place for encouraging response, following up non-response, and dealing with missing data (e.g., through imputation or adjustment made to the estimates). All stages of collection and processing should be subject to proper consideration of the need for quality assurance processes, including appropriate internal and external consistency checking of data with corresponding correction strategies.

Coherence

For managing coherence, collection agencies should use standard frameworks, concepts, variables and classifications, where such are available, to ensure the target of measurement is consistent over time and across different collections. As well, the use of common methodologies and systems for data collection and processing will contribute to coherence. Where data are available from different sources, consideration should be given to their confrontation and possible integration.

Interpretability

Managing interpretability is primarily concerned with the provision of sufficient information about the statistical measures and processes of data collection. Users need to know what has been measured, how it was measured and how well it was measured. The description of the methodology allows the user to assess whether the methods used were scientific or objective, and the degree of confidence they could have in the results. For meeting specific objectives, using analytical, descriptive or graphical techniques can often add value to help draw out the patterns in the data.

Accessibility

Management of accessibility needs to address how to help users know about the existence of the data or statistical product, locate it, and import it into their own working environment. Output catalogues, delivery systems, distribution channels and media, and strategies for engagement with users are all important considerations relating to this quality dimension.


MORE INFORMATION

For more information on any of the issues discussed above please contact the Methodology and Data Management Division, ABS (Canberra) by email at methodology@abs.gov.au, or by telephone via the ABS National Information and Referral Service on 1300 135 070.