TARGETED INTENSIVE FOLLOW-UP
An important stage of the survey cycle is the process of Intensive Follow Up (IFU) in which concerted effort is made to increase response rates to ensure high quality estimates. While all ABS surveys strive to reach high response rates, the processes chosen to achieve this may vary considerably from survey to survey.
The ABS has developed a general framework for targeting IFU which was presented at the International Conference on Establishment Surveys II in Buffalo, New York, USA in 2000. Since then we have worked on refining the framework and testing the methodology.
The general method for targeting IFU is to rank non-respondents in term of importance. The technique for ordering may take several forms. As a group, these methods are referred to in the ABS as 'targeted IFU'. The recent work focused on three levels of targeted IFU, each with increasing requirements and corresponding opportunities for gains in quality of survey estimates. The three targeted IFU levels reflect both the changing types of auxiliary information available and increasing quality requirements of the survey.
Level I: Aims to examine key elements to the survey data quality to ensure that the providers responding at final close-off provide a balanced, representative sample that matches the sample design as closely as possible in order to minimise the non-response bias of the survey. Level I is the most general type of targeted IFU and can be applied to any business survey (one-off, irregular or regular) and is likely to be used for the majority of irregular and newly implemented surveys. Non-responding providers are prioritised based on stratum level information, with all providers within a stratum receiving the same score, (based on such things as newon status, weight and proportion of defunct/nils in the strata).
Level II: Is a more accurate IFU process, which augments Level I IFU by using available auxiliary information. In contrast to Level I, Level II may give a different score to non-responding providers within the same stratum, as the auxiliary information may be at the individual provider level. Level II is particularly suited for ongoing surveys, where previous data for providers is available, or where other related data can be sourced. Examples of auxiliary data include difficulty to impute (based on previous provider response); related data reported in ABS survey with common providers; long-term non-response; and historical contribution to estimates.
Level III: Level III IFU is a different process to the other types. It centres on calculating the error of imputation for non-responding providers for all key data items. The error of imputation is the absolute difference between the providers actual data and imputed data, relative to the level estimate for the cycle they last responded. Level III produces a score based entirely on historical data for the provider. (Where historical provider-level data is unavailable, providers are given a maximum score). The error of imputation is calculated at the class of interest (for example, one-digit industry) and is standardised against a set of benchmarks to produce the unit's score. When the criteria for implementation are met, Level III IFU can make provider-level inferences about the influence and reliability of any given provider's response in the context of the overall estimates. High cost is a limiting factor for Level III, both to cover the initial investigation prior to implementation and maintenance. Complex imputation methodologies can complicate its evaluation and implementation.
Each level has increasing requirements for implementation, from general applicability through to more rigid requirements of survey data stability and large sample. It is expected that each survey would use the level best suited to that survey. For ongoing surveys, any initial technique implemented could be evaluated later as more information and/or additional data sources were made available. An evaluation should also measure whether the IFU method is achieving its objectives (ie improve quality, reduce costs, etc).
Based on the evaluations conducted in Western Australia on the Labour collections and the development of a three tier framework we have proposed that:
- the targeted IFU framework be used for describing IFU processes and practices undertaken within business surveys in the ABS;
- we work through how the technique could be used in practice to meet the requirements of different surveys; and
- we test how the scoring methods reflecting the frameworks principles can be applied in practice to other collections.
The results so far have been positive and it is expected that the introduction of targeted IFU techniques to survey areas which currently have none will allow an improved allocation of resources for IFU, and provide a framework for decision making regarding costs associated with IFU processes. It is also expected that the use of a targeted IFU technique will result in gains in estimate quality as the responding sample more closely aligns with the sample design.
For further information, please contact Paul Sutcliffe on (02) 6252 6759.
E-mail: p.sutcliffe@abs.gov.au
Print Page
Print All