1504.0 - Methodological News, Sep 2007  
ARCHIVED ISSUE Released at 11:30 AM (CANBERRA TIME) 21/09/2007   
   Page tools: Print Print Page Print all pages in this productPrint All

ABS Using Operations Research to Improve Efficiency and Effectiveness of Business Surveys

The ABS and the world around us is becoming more complex. Huge numbers of choices and relentless time and cost pressures make the decisions we face more daunting and more difficult. Meanwhile, our systems are generating massive amounts of data about how we do our business providing us with opportunities for independent and objective evaluation of our processes. For example, the introduction of a centralised Provider Management Information System (PIMS) for recording all contacts made to providers in Business surveys has given us the opportunity to objectively analyse and improve the effectiveness of intensive follow-up (IFU) processes. The Operations Research Unit (ORU) was established within the Statistical Support Branch in July last year and is tasked with taking advantage of this and other similar opportunities to answer questions such as:

  • Which strategies result in the highest rate of response for the lowest amount of cost?
  • What should we do differently and how should we do it?
  • Where are we expending a lot of effort without receiving a lot in return?

Investigations to date have centred around the annual Economic Activity Survey (EAS) and two sub-annual surveys - the Average Weekly Earnings Survey (AWE) and the Quarterly Economy Wide Survey (QEWS). Generally the IFU strategies for these and other collections involve the use of reminder letters and follow-up phone calls. However, the number and timing of these letters and phone calls relative to the due date varies considerably. For example, some collections use pre-approach letters, some quarterly collections have up to four reminder letters, while some annual collections may only have two. The length of the IFU cycle varies from collection to collection and the overall response rate or imputation rate required by each collection also differ. Within each collection, contact of providers is usually prioritised by significance. This significance is assigned based on different factors for each collection. Providers are called randomly throughout the day without regard as to whether some times are better to call particular providers or not.

Investigations into EAS, QEWS and AWE contact data have identified a number of patterns that can be used by the Provider Contact Unit (PCU) to improve effectiveness of the IFU processes. For example, it was found that:

  • towards the end of the cycle, a large amount of effort is expended sometimes without obvious gain, some providers are contacted up to 30 times without securing a response. In particular, in the AWE survey, calling providers more than 5 times takes 10% of the overall effort in terms of calls and results in an increase to the non-imputed contribution of estimates of 3% and an increase in form receival rate of 3%. Being able to identify a point in the IFU process after which further gains are expected to be very small will enable to develop more effective strategies for following up outstanding providers that do not respond to simple intensification of effort;
  • best time of day to contact a provider varies between industries. This information can be used to schedule provider follow-up in a way that maximises the chances of making a contact with the provider. In particular it appears that a higher rate of contact for businesses in Education could be achieved by avoiding the 1-2pm time slot;
  • there is a lag between a follow-up attempt and a response triggered by that attempt. In the annual survey of EAS, the lag was around 14 days whereas for the quarterly survey of QEWS, the lag was around 6 days. By refraining from further contact with the providers during this lag, the provider load and follow-up effort can be reduced without negative impact on the response or imputation rate;
  • providers are less likely to respond if a long period elapses between follow-up attempts. This is not usually a problem in QEWS, but not contacting someone for over 30 days can be a problem in EAS. By concentrating the follow-up effort in a way that ensures that all providers are followed-up within a reasonable time from the last attempt better response and imputation rates can be achieved earlier in the IFU period and number of providers that need a large number of contacts at the end of the IFU period can be reduced;
  • around 93% of providers who responded without phone follow-up in one quarter respond without phone follow-up in the subsequent quarter. The PCU has used this information to minimise unnecessary contact with "good" providers.

Analysis of existing contact data is not sufficient to determine whether proposed modifications to follow-up strategies actually result in expected improvements and to ensure that these changes do not compromise the quality of statistics collected. In order to do that, carefully controlled experiments that allow alternative strategies to be tested and if necessary modified are required. Currently, two trials are planned to test alternative dates for phone follow-up in QEWS and EAS collections. These trials will allow the level of improvement to be quantified and if successful will enable evidence-based changes to IFU strategies for these collections.

Future work will concentrate on following up the results of the trials and expanding the research to more surveys. Further work is also planned on details relating to reminder letter timing, the use of pre-approach letters, extensions, significance flags and more control of call patterns.

For further information, please contact Louise Gates on (02) 6252 6540.