MODE SUITABILITY MODEL
While aspects of these standards will be of interest to those outside the ABS, they were developed for internal use. As such, some information contained in these standards will not be applicable to an external audience. ABS staff should refer to the Corporate Manuals database for the most recent version of these documents, as some details (names, phone numbers etc.) have been removed from the online version.
Contents
Overview
This document provides a list of factors that should be considered when determining a suitable mode to use as the
primary mode of data collection for a survey. Several modes of collection are assessed against each factor, in terms of the mode's suitability for surveys with certain characteristics.
The document will eventually contain the main modes of data collection used at the ABS. As of June 2008, the modes covered in the document are:
- Computer Assisted Telephone Interviewing (CATI)
- Mail forms
- Fax forms
- Offline electronic forms (e.g. Excel and PDF forms)
It is strongly recommended that people involved in determining an appropriate data collection method for a survey read the Mode Suitability Framework document, for a more detailed discussion of the advantages and disadvantages of common modes, and the types of surveys they are suitable for. The present document is intended to be read in conjunction with the Mode Suitability Framework, providing a succinct summary of the factors to consider, and allowing the suitability of different modes for a survey to be easily compared.
How to use this document
Thirteen factors that are relevant when determining a mode's suitability as the primary data collection method for a survey are listed in the table below. Each mode is assessed against the factors, and is given a suitability rating in terms of how suitable it is for surveys with that characteristic. The rating system uses a "traffic light" approach with each colour representing the following:
= Against the factor, the mode is suitable
= Against the factor, the mode may be suitable, but there are modes that are more suitable for surveys with that characteristic.
= Against the factor, the mode is unsuitable.
- NA = The factor is not relevant for the mode.
For each factor listed in the left hand column of the table, find the relevant characteristic that matches the survey that is being assessed against the particular mode/s listed at the top of the table. For example, if considering an offline electronic form (such as an Excel form) for a survey that is run quarterly, the survey would get a "green light" against factor 1 ("Survey frequency"), indicating that against this factor, an offline form is likely to be suitable. Each survey should obtain only one rating per factor.
To get an overall indication of a mode's suitability for a survey, assess the survey against each factor for the mode being investigated, and then add up the number of green, amber, and red lights obtained for that mode.
Scoring
- The more green lights
, the greater the suitability of the mode for the survey.
- If a mode has more than 4 amber lights
, it is likely that another mode will be more suitable, and it is therefore recommended that a different mode is considered.
- As a general guideline, if a mode yields more than 2 red lights
, it is highly likely to be unsuitable for the survey, and an alternative mode should be considered.
Important notes on using this document
This document is intended to provide a range of issues to consider when determining an appropriate mode to use as the
main mode of data collection for a survey. The ratings are not intended to apply when considering the mode as part of a mixed mode design. This is because mixing modes introduces greater complexity, and the suitability of each mode would depend on which other modes are being offered. This would make it difficult to provide general suitability guidelines for the mode using the simple rating scale that has been used below. For example, when assessing the suitability of fax forms below, it is assumed this is being considered as the main mode of data collection (e.g. not as an alternative method to a mail survey in a mixed mode design).
The suitability scores below also assume that (a) the target respondents have access to the technology/equipment required to be able to complete the forms (e.g. a fax machine if fax forms are being considered); and (b) the ABS has access to the required frame in order to make contact with respondents.
As surveys may have their own complex and unique issues, the model is not intended to provide an absolute measure of suitability, and is to be used as a guide only. The
Mode Suitability Framework document contains further explanation about the rationale behind the ratings given in this document.
Suitability ratings
Factor | Mode of primary data collection
(See relevant sections in Mode Suitability Framework for more information on the advantages, disadvantages, and suitability of each mode) |
|---|
| | | |
|---|
| Size of the survey |
1. Survey frequency
The frequency with which a survey is run is an important factor to consider when determining an appropriate mode. The higher the frequency, the less suitable a mail survey is, due to the delays between dispatch and receipt of the form. When converting to a mode where the initial set-up costs are high (e.g. CATI, IVR), surveys that are run more frequently may be more suitable from a cost-benefit perspective. Fax forms may be more suitable for higher-frequency surveys, because respondents are more likely to be anticipating the form's arrival, compared to a one-off or infrequent survey. |
 | CATI | Mail forms | Fax forms | Offline forms |
| Monthly | | | | |
| Quarterly | | | | |
| Half yearly | | | | |
| Annual | | | | |
| Less frequent | | | | |
2. Sample size
When considering converting a survey to a mode with a high set-up cost (e.g. IVR, CATI), surveys with bigger sample sizes are generally more appropriate from a cost-benefit perspective. The suitability of other modes, such as mail surveys, is less affected by sample size differences. |
| CATI | Mail forms | Fax forms | Offline forms |
|---|
| 10 thousand or more | | | | |
| Under 10 to 5 thousand | | | | |
| Under 5 to 2 thousand | | | | |
| Under 2 to 1 thousand | | | | |
| Under 1 thousand | | | | |
3. Survey generalisability
The amount of content that a survey has in common with other surveys should be considered. For example, if most of the questions in the survey are from Standard Question Wording (SQW), then CATIs, IVR surveys, web forms etc. will be more easily adapted to the other surveys which use SQW, therefore potentially saving development time. Note that this factor does not include coverage and reference period questions which are common to almost all surveys. Also note that the length of the survey should be considered when assessing the survey's generalisability - it may still be cost effective to develop CATI, electronic forms etc. for short forms that do not have much (or any) content common with other surveys. |
| CATI | Mail forms | Fax forms | Offline forms |
|---|
| All questions common to many other surveys | | | | |
| Most questions common to many other surveys | | | | |
| Most questions common to some other surveys | | | | |
| Some questions common to some other surveys | | | | |
| No questions common to any other survey | | | | |
| Provider load |
4. Time taken/interview length
This factor refers to the estimated time taken to complete the survey form, either for respondents (for self-administered forms) or for interviewers. Interview length is particularly important to consider for telephone surveys, as this method is unsuitable for long surveys due to provider load concerns. Fax is also unsuitable for long forms, due to the effort/cost required for the respondent to send the form back to the ABS, and possible problems with pages becoming separated during transmission. |
| CATI | Mail forms | Fax forms | Offline forms |
| 5 minutes or less | | | | |
| 6-10 minutes | | | | |
| 11-20 minutes | | | | |
| 21-30 minutes | | | | |
| 31 minutes or more | | | | |
5. Total number of data items
This factor is included in addition to time taken as a measure of provider load, because self-reported time taken can be imprecise. A data item in this context is roughly interpreted as equalling the potential number of answers (including yes/no questions), or the number of decisions the respondent may have to make. This includes counting every item in a "mark all that apply" question as one, because when conducted in a verbal interview, each item will need to be considered separately. As described in Factor 4, telephone surveys and fax surveys are less suitable for forms with a large number of data items due to the increased burden placed on respondents. |
| CATI | Mail forms | Fax forms | Offline forms |
| 5 or less | | | | |
| 6-10 | | | | |
| 11-20 | | | | |
| 21-40 | | | | |
| 41 or more | | | | |
6. Question complexity
The complexity of the survey questions is very important to consider, but is a difficult factor to measure. The intention of this factor is to capture the amount of explanatory notes required to understand the questions, as well as the space taken up by complex matrix structures etc. This factor uses the proportion of the form and other material (not including front page) taken up by instructions, which allows for a consideration of text regardless of how tightly cramped the spacing is. Note that forms with complexity at the higher end (above around 60%) may be problematic in terms of provider load, and consideration should be given to simplifying the form. |
| CATI | Mail forms | Fax forms | Offline forms |
| Less than 20% | | | | |
| 20% to less than 40% | | | | |
| 40% to less than 60% | | | | |
| 60% to less than 80% | | | | |
| 80% or more | | | | |
7. Amount of sequencing
This factor assesses the amount of the sequencing in the form. Being required to follow sequencing instructions adds to provider load (or interviewer load where interviews are used), and can lead to errors in terms of which questions are/are not completed. Forms with large amounts of sequencing may be better administered electronically, where it is assumed here that sequencing will be automated (e.g. CATI, CAPI, computerised self-administered questionnaires). |
| CATI | Mail forms | Fax forms | Offline forms |
| None | | | | |
| Low | | | | |
| Moderate | | | | |
| High | | | | |
8. Proportion of items requiring record checking
This factor distinguishes questions which may be answered off-hand (such as attitudinal and behavioural questions), from questions which require checking of records, or require input from a range of other people. Most questions requiring numerical data will fall into the latter category, and are less suitable for telephone surveys, where the telephone call should be kept as short as possible. |
| CATI | Mail forms | Fax forms | Offline forms |
| None | | | | |
| Low | | | | |
| About half | | | | |
| High | | | | |
| All | | | | |
| Risk |
9. Sensitivity/confidentiality concerns
Survey respondents may be reluctant to provide honest answers to questions asking about sensitive or confidential information. Sensitive data includes anything which may be personally embarrassing for the respondent, anything that may be professionally embarrassing for the respondent, their area within the business or for the whole business, anything that may be commercial-in-confidence and so on.
Telephone surveys are the worst method for collecting sensitive data, while self-administered mail surveys are best. While fax forms and offline forms are also self-administered and therefore better than telephone surveys, the method of returning these forms may cause respondents to be more concerned about confidentiality compared with returning the paper form in a sealed envelope. For example, concerns about online security, or concerns that a fax may be received by the wrong party, mean that these methods are likely to be less suitable than mail forms for collecting sensitive information. |
| CATI | Mail forms | Fax forms | Offline forms |
| Very low | | | | |
| Low | | | | |
| Medium | | | | |
| High | | | | |
| Very high | | | | |
10. Importance of estimates
For some surveys, it is more important that the data is of high quality than for other surveys. We should develop our expertise in new technologies on the less important surveys before we convert the more important ones. |
| CATI | Mail forms | Fax forms | Offline forms |
| User funded | | | | |
| Other ABS surveys | | | | |
| Non-MEI contribute to National Accounts | | | | |
| Contribute to MEI | | | | |
| MEI | | | | |
| Respondent characteristics |
11. Literacy of respondents
This factor considers the literacy of respondents, and is relevant for self-administered forms, where respondents must read and understand the instructions and questions without assistance from an interviewer. According to Biemer and Lyberg (2003), a general rule of thumb is that forms aimed at the general population should be at a fifth-grade reading level. The percentages below represent the proportion of respondents expected to have a reasonable level of literacy. |
| CATI | Mail forms | Fax forms | Offline forms |
| 95%-100% | | | | |
| 85%-94% | | | | |
| 75%-84% | | | | |
| 74% or below | | | | |
12. Geographic location of respondents
This factor takes into account the location of the majority of respondents, and how difficult it would be for an interviewer to reach them. Mail surveys are usually more suitable than face-to-face interviews for reaching respondents in diverse geographic areas, where actually visiting the respondents would be too difficult/costly. Telephone surveys, electronic forms, and fax forms can also be suitable for reaching difficult to contact respondents, provided that respondents have telephone/internet/fax access. |
| CATI | Mail forms | Fax forms | Offline forms |
| Metropolitan (inner and outer), with telephone/internet/fax coverage | | | | |
| Metropolitan (inner and outer), without telephone/internet/fax coverage | | | | |
| Rural, with telephone/internet/fax coverage | | | | |
| Rural, without telephone/internet/fax coverage | | | | |
| Other factors |
13. Budget
Interviewer-administered surveys are more expensive to conduct than self-administered surveys (due to training costs, travel costs, etc). Face-to-face interviews are the most expensive, followed by telephone interviews. Mail-out surveys are generally the least expensive. Applying a dollar amount to the budget available is not practical, as what is considered a large budget for one survey may be considered small for another. Therefore, the ratings used for this factor are necessarily subjective. |
| CATI | Mail forms | Fax forms | Offline forms |
| Large | | | | |
| Medium | | | | |
| Small | | | | |
References
Print Page
Print All