Feature Article - Technical report: Implementation of computer assisted interviewing in the Labour Force Survey
This article was published in the January 2005 issue of Australian Labour Market Statistics (cat. no. 6105.0).
SUMMARY
The ABS progressively implemented computer assisted interviewing (CAI) into the Labour Force Survey (LFS) over the period October 2003 to August 2004. This method of interviewing replaced the traditional 'pen and paper' method used previously. Under CAI, interviewers record responses directly onto an electronic questionnaire in a notebook computer. Users of LFS data were informed of progress with the implementation of CAI in issues of the monthly publication Labour Force, Australia (cat. no. 6202.0) over the period August 2003 to August 2004.
The ABS expected that the implementation of the CAI interviewing method would not have a significant impact on published labour force estimates. During the implementation period the ABS conducted a range of analyses on each month's data. These analyses confirmed that the change in interviewing method did not materially affect the aggregate estimates for any month during the implementation period.
BACKGROUND
The LFS obtains information from a sample of private dwellings (currently about 30,000 houses, flats, etc.) and a sample of non-private dwellings (hotels, hospitals, nursing homes, etc.). About 0.45% of the population of Australia are surveyed each month. Selected dwellings remain in the survey for eight consecutive months, with one-eighth of the sample replaced each month. Information is obtained from the occupants of selected dwellings by trained interviewers, using face-to-face and telephone interviewing.
PREPARING FOR CAI
During 2002 and 2003 the ABS developed and tested a CAI version of the paper questionnaire and associated processing systems and procedures. The CAI version was designed to replicate the paper questionnaire and procedures as closely as possible. This was done to minimise the chances of any impact on labour force series from the change to CAI.
During these two years the ABS also trained interviewers and office staff in the use of CAI. A number of other ABS household surveys have been enumerated using CAI over the past five years.
A phase-in strategy was developed to meet a number of statistical and operational objectives. These included minimising the effect on published estimates should data collected in the initial phase-in of CAI not be available for use in compiling survey estimates, and ensuring that a reasonable measure could be made of any potential statistical impact of implementing CAI.
IMPLEMENTING CAI IN THE LFS
The CAI method was implemented in four phases, starting with 10% of survey interviews in October 2003, increasing to 40% between February and April 2004, to 70% in June and 100% in August 2004. The phases were:
Phase 1: For the October 2003 survey, a random sample of 10% of interviewers in all states and territories were selected to use the CAI method. The selected interviewers conducted all their interviews using CAI. This means approximately 10% of LFS interviews were conducted using CAI in October 2003. The remaining 90% of interviewers used the 'pen and paper' method. The interviewers selected for CAI in October 2003 continued to enumerate all of their interviews with CAI in subsequent months. As a result, the proportion of interviews allocated to Phase 1 remained constant at about 10% for all months after October 2003.
Phase 2: For the February 2004 survey, an additional random sample of 30% of LFS interviewers in New South Wales, Victoria, South Australia and the Australian Capital Territory were selected to use CAI. ABS had planned to increase the sub-sample of interviewers using CAI to 30% in all states and territories in February, but a number of technical problems prevented this. For the April 2004 survey, an additional random sample of 30% of LFS interviewers in Tasmania, Queensland, Northern Territory and Western Austrlalia were selected to use CAI for the first time. Interviewers in the Phase 2 sample continued to use CAI in subsequent months.
Phase 3: For the June 2004 survey, an additional random sample of 30% of interviewers in all states and territories was selected to use CAI for the first time.
Phase 4: The remaining 30% of interviewers conducted their interviews using CAI for the first time in August 2004. That is, all persons in the LFS in August 2004 were interviewed using CAI.
The interviewers allocated to each phase were selected to have a representative distribution of: capital city and balance of state; private and non-private dwellings; and the survey's eight rotation groups.
MEASURING THE IMPACT
The impact on LFS estimates of changing the interview technique is called a mode effect. The CAI mode effect associated with any LFS estimate (e.g. number of unemployed persons in Australia) is the difference between the estimate that would result if all data were collected by CAI, and the estimate that would result if all data were collected by the 'pen and paper' method.
The approach used to phase in CAI provided a relatively powerful means for the identification and measurement of a possible mode effect. The mode effect was estimated by a method that drew strength from comparing that part of the sample which had been converted to CAI with the remainder of the sample, as well as comparing the data before and after CAI was introduced. In particular, the comparison took account of any variation in the age and sex structure of the sample, and in the historical levels of the data prior to the introduction of CAI.
A range of analyses was conducted using data from August 2003 to August 2004 to estimate the CAI mode effect for each of the phase-in groups, as well as for the whole sample. The analyses were intended to test if the CAI mode effect was larger than 0.3 percentage points in the unemployment rate and 0.4 percentage points in the employment to population ratio, at the Australia level. (The statistical tests performed had a chance of identifying these sizes of impact with 80% probability, and a 10% chance of false identification.) These thresholds correspond to approximately 3 times the standard errors of the month to month movement of the corresponding estimates.
Results from these analyses showed that the estimated mode effects were very small when assessed against the sampling variability expected for these estimates. That is, there was no statistical evidence of a mode effect on the published key labour force estimates (employed persons, unemployed persons and the unemployment rate).
As reported by the ABS in the September 2004 issue of Labour Force, Australia (cat. no. 6202.0), operational difficulties were experienced during enumeration of the August 2004 LFS, resulting in a lower than usual response rate. Software problems with the CAI notebook computers contributed to the drop in response rate in August 2004 (the software problems have since been fixed). Analysis conducted by the ABS subsequent to the publication of August 2004 LFS results indicated that the reduced response rate caused a small understatement of the August employment estimate. Notwithstanding this impact, there was no evidence of a sustained CAI mode effect during the period August 2003 to August 2004.
FURTHER INFORMATION
For further information about the Labour Force Survey please contact the Assistant Director, Labour Force on Canberra (02) 6252 6565.
For email enquiries please contact Client Services on client.services@abs.gov.au.
Print Page
Print All