Methodological News, September Quarter 2024

This issue contains one article:

• An investigation into the Computer Assisted Telephone Interview experience

A research project to better understand the Computer Assisted Telephone Interview (CATI) experience for respondents and interviewers, and the subsequent impact on data quality was completed in July 2024.

This investigation was designed to support the move away from Computer Assisted Personal Interviews (CAPI) as the main follow-up mode for collections with an online first approach. 

This project involved Australian Bureau of Statistics (ABS) research officers listening in to data collection calls between respondents and ABS interviewers for a household survey which includes questions about financial data, and where record checking is strongly encouraged. 

Calls were only monitored when respondents consented to participation in the research project, and the project was conducted with the aim of observing as many respondent CATI pathways as possible during selected periods in the survey cycle where these collection calls were most likely to occur. 

The CATI data collection calls involved:

• respondent inbound calls, where respondents called the ABS in response to receiving a letter about the survey
• scheduled outbound calls, which were typically made following a home visit by an interviewer or scheduled as a result of the respondent’s inbound call, and 
• single session interviews typically lasting between 40 and 90 minutes. 

Key findings from this project include that:

• many CATI respondents observed had initially attempted to complete the survey online and/or expressed a preference to complete the survey via that mode
• some CATI respondents were refusal conversions rather than having CATI as their preferred mode of response 
• most inbound calls from respondents, who went on to complete the survey via that phone call, were from persons aged 65 years and older
• respondent burden increased as the time completing the survey progressed, and was associated with noticeable drops in the willingness of respondents to refer to records and provide precise data, and 
• all respondents, when asked, were willing to have their data collection calls with the ABS monitored for research and development purposes. 

This research project also enabled a better understanding of the barriers to survey completion encountered by respondents who ended up completing via the CATI pathway. These barriers included: 

• the ABS survey online account creation process was too difficult
• confusion about the end point of the online survey
• lack of a computer and/or low digital literacy skills
• trust and legitimacy concerns related to the collection and materials received, and
• unexpected loss of access to the online survey e.g. due to missing the deadline for online completion.

These findings will now be used to inform further research to identify improvements that can be made to ABS systems and processes to facilitate a better and easier online experience for respondents to complete via that mode. 

Another key focus of this project was to better understand where improvements could be made to the content design of surveys to better facilitate the collection of high-quality data by CATI. Findings included that: 

• the survey length exceeded respondent expectations
• manual calculations required to provide some responses were burdensome for both interviewers and respondents
• there were challenges with record checking including respondents not knowing what to look for
• the repetitive nature of some question sets without visual aids caused confusion, and 
• the role of interviewers was important in assisting some respondents to provide quality data. 

Overall, this research method has provided valuable insights into the experiences of completing ABS surveys via CATI.

We thank the respondents involved in this research project for their willingness to have their calls with the ABS monitored for research and development purposes. Findings will be used to improve the completion pathways available to respondents and enhance survey form design.   

For further information, please contact Kirsten Gerlach or Philip Carmo.

Please email methodology@abs.gov.au to:

• contact authors for further information
• provide comments or feedback
• be added to or removed from our electronic mailing list.

Alternatively, you can post to:

Methodological News Editor
Methodology Division
Australian Bureau of Statistics
Locked Bag No. 10
Belconnen ACT 2617

The ABS Privacy Policy outlines how the ABS will handle any personal information.

Releases from June 2021 onwards can be accessed under research.

Releases up to March 2021 can be accessed under past releases.