Predicting Travel Costs in Household Surveys
In addition to modelling the cost of intensive follow-up processes for ABS economic surveys (see related article above), the Operations Research Unit (ORU) is also undertaking a number of projects into modelling the cost of interview processes for ABS household surveys. These projects are also part of a conceptual framework which links input parameters and processes, such as survey sample size, interviewer characteristics, and follow-up effort to cost and statistical outcomes, such as bias of output estimates and response rates. Currently we are developing statistical models to estimate these inputs more robustly, and to link these inputs to outcomes, to enable predictions to be made, and to inform decision making on what operational practice efficiencies can be made.
As part of this work, the ORU has been developing a model to predict the cost of interview travel time and motor vehicle allowance (MVA) for household surveys. Initially this is being done for the Monthly Population Survey (MPS). Currently these costs account for about 40% of the total enumeration cost in standard workloads.
This analysis makes use of MPS paradata. Interviewers record individual visits to each household, with the outcome of that visit (e.g. interview, noncontact, etc) recorded on the CAIWMS (Computer Assisted Interviewer Workload Management System). These call-level records can be combined to identify interviewer call patterns at a broader level, such as the number of times an interviewer visits each household, and the number of times they visit a particular block. Matching these with records of the actual time and distance claimed for a trip provides a dataset that can be used to develop models of time and distance as a function of interviewer call patterns. Factors found to be good predictors of time and distance were calls made to households, trips made to blocks, and number of fully-responding households.
Various factors could cause bias in the fitted model. In particular, because of quality limitations of the paradata, not all calls can be matched with time/distance records, causing missingness that may lead to bias. Bias can be reduced by calibrating cost predictions for the fitting months against the actual survey costs recorded by our Population Survey Operations area for those months to produce calibration factors that can then be used to correct predictions for future months.
To validate this time/distance model, it was fitted and calibrated on MPS data from July 2007 to April 2008, and then tested on call records from June-December 2008, a period that included months where the MPS sample size was reduced due to funding cuts. Predictions were actually better in the validation months than for the fitting months. This is due to a large number of missing/unmatched records for August and September 2007 that caused errors in prediction for these months. The actual MVA costs between June and December 2008 decline with the sample cut, and the predicted model costs reliably track this pattern, staying within about 7% of the cost at each time point and matching movements from month to month.
Besides use for budget forecasting, this model can also be used to dissect costs and determine how different activities affect MPS costs and therefore influence the choice of efficient survey operations.
For example, some MPS households require visits after the first month, either because the interviewer wasn't able to establish telephone contact or because the occupant has requested face-to-face interviews. Applying this model showed that although these non-first-month-in-sample (NFMIS) households produce only 35% of face-to-face interviews, these households account for approximately 54% of travel/MVA costs. This is because first-month households are clustered, allowing an interviewer to work efficiently by visiting several neighbouring households in one trip. A NFMIS household is likely to be the only one in its block that needs to be visited face-to-face, requiring the interviewer to make a special trip just for that one household.
One practical example of informed decision making on efficient processes would be the cost of increasing the response rates by allowing further interview contacts. By comparing cost and response at each call, it's possible to estimate the marginal cost per interview at different parts of the survey, giving the time and expenditure required to secure one more respondent. As the survey goes on, this cost increases because the chances of response become poorer, and because it's less efficient to visit a small number of remaining non-respondents. For example, in the first month in sample, making the first visit to each MPS household costs about $28 for each response achieved but by the fifth call, the ABS is spending approximately $60 for each additional response. Under a limited budget, it is never possible to secure 100% response rate, but comparing marginal costs in different parts of the sample helps show ways to achieve the largest possible response within that budget.
More recent work includes an investigation of the effects of geography on interviewer travel time and distance, in combination with factors discussed above. ORU determined that the distance between the workload and the interviewer's home is an important predictor of travel time and distance in most cases. However, when the workload is more than 100km from the interviewer's home, the exact distance becomes unimportant, because the interviewer is generally working from a temporary base rather than travelling from home.
MPS costs and statistical quality are both closely linked to response rates. Therefore, ORU is currently working on a standardised way to link these phenomena to allow for "big picture" decision-making in household surveys. In the future, ORU aims to extend the cost model and linked model to other household surveys in the SSS program.
For further information, please contact Geoffrey Brent on (03) 9615 7685 or geoffrey.brent@abs.gov.au.
Print Page
Print All