• Summary
• Downloads
• Explanatory Notes
• Related Information
• Past Releases

Increasingly Distant: New and Improved Questionnaire Development Methods Without Face to Face Testing

The high quality of ABS official statistics is partly maintained through significant development of new questions and new surveys prior to enumeration. Cognitive testing, a type of in-depth interviewing, is used to explore respondent’s interpretation of what questions mean and what emotional reaction they cause. Usability testing is also conducted for self-administered questionnaires, to ensure these are easy to use and engaging while still capturing fit-for-purpose data.

Both methods ideally gather non-verbal communication from test participants as well as verbal, since movements and facial expressions convey a great deal of information about their experience of our questionnaires. A positive experience is important because this influences a respondent’s motivation while completing our surveys, and therefore how much effort they put into reading instructions and response options, searching their memories and records, explaining their responses and so on. Traditionally, tests exploring these issues are conducted face to face and mostly in ABS office research labs.

The COVID-19 pandemic abruptly removed our ability to conduct tests this way. However, survey development needs remained, with the importance of data quality continuing and the need for effective web form design in particular greatly increased. Emerging methods under investigation for other reasons became immediately critical, and discussions with similar user-researchers in other organisations found that they too were facing- and embracing- the challenges and opportunities of remote testing.

Ensuring the security of our statistical data, and by extension, of our questionnaires and internal IT network, meant that collecting data by video conference and granting outside access to our test environment required careful consideration. Recording both video and audio are still important to support rigorous analysis across interviews. Ways around the reliance on technology, and respondent capability to manage it from their end, are progressing slowly. Our journey so far on socially distant testing includes:

1. Skirmishing with ABS staff. The ABS is a large organisation and at any time, there are hundreds of people sufficiently unaware of the particular content and design issues of a single survey. Lack of diversity in areas like educational level and job characteristics are irrelevant for many test objectives, and this method provided invaluable insights on web form navigation design.
2. Testing with other household members. The majority of ABS staff are currently working from home, and their family and housemates are equally locked-down. Appropriately secured access to developing questionnaires via staff computers allowed usability testing with a more diverse population, as well as some insight on relevant household dynamics.
3. True remote testing. Changes to standard question wording affecting all ABS household surveys required cognitive testing with certain demographic groups. We are adapting our rigorous standard techniques to conduct interviews remotely, using screensharing of draft questions and asking participants to think aloud as they consider how they would respond. An added value of this method is more closely recreating the ‘natural’ setting in which respondents will complete the real surveys.
4. Probing using web panel surveys. A range of existing commercial web panels allowed access to a large number of diverse respondents who provided feedback on their understanding of a potentially-problematic survey question. The speed and geographic spread of responses were especially useful, and metrics such as time to respond were an unexpected bonus over our normal methods.