1530.0 - ABS Forms Design Standards Manual, 2010
ARCHIVED ISSUE Released at 11:30 AM (CANBERRA TIME) 25/01/2010 First Issue
Page tools:
 Print Page
 Print All
| ||||
QUESTION STRUCTURE
Introduction The way in which questions are asked will influence how well the form works. When you design questions you should take into account;
For most ABS business surveys many of the questions used can be drawn directly from the Standard Question Wording developed by Economic Standards and Classifications, and from other business forms. Similarly, there are standard modules for ABS household survey questionnaires which should be used where appropriate. However, when developing new questions or updating old ones the following should be taken into account. What is a question? In these Standards the definition of a question is: Any set of words which ask the respondent to give information. An example is shown in Diagram 6.1. Diagram 6.1 In the last example 'Males' and 'Females' are subsidiary questions (also called sub-questions). In some cases a question may consist of a caption or item only. For ABS forms this style is generally restricted to financial data items as these are used on many forms, are based on accounting standards, and are well understood by respondents (see Diagram 6.2). A question consisting of a caption or item only does not use a question mark or any other punctuation such as a full stop. Diagram 6.2 Designing questions You are trying to create a dialogue, or conversation, between the form and the respondent. The respondent must be able to easily understand and take part in this dialogue. To do this you need to know how respondents behave.Research into form-filling behaviour (see Frohlich 1986) has suggested some of the principles which guide respondents: Principle of linear progression: work through the questions in the order they appear on the form. A respondent will keep going through questions as long as there is an apparent order. The respondent will continue to answer questions sequentially unless he or she is clearly instructed to do otherwise or until an apparently irrelevant question is encountered. Routing instructions have to be very obvious if they are to be followed - inaccurate routing decisions are largely made up of failures to route away from the current question series. This is why "go to" instructions are not necessary to direct respondents to the following question. Respondents will often read questions that do not apply to them, even when they comprehend the routing instructions and other typographical distinctions, due to curiosity or a distrust of the skip convention. Respondents will then fill in the question if it appears relevant to them.The respondent preference for sequential order is also why matrices should be used sparingly. The lack of an obvious single path of response increases the respondent's cognitive burden. Respondents are likely to overlook information not presented in a normal reading format and the separate presentation of conceptually related information such as the row and column headings of a matrix leads to slower processing. Respondents can't focus on all the information at once and have to store the different pieces in short term memory long enough to integrate them (Jenkins & Dillman, 1997).It should be noted that although respondents prefer a linear progression through a form, for business collections this is often not possible. Some information may not be available, or several different people may need to enter data into the one form. In this context respondents will go through and answer all the questions they can, skipping the ones they can't. This is one reason why it is important to group related questions together and have the beginnings of new parts and sections easy to find so that respondents can go back later and fill in the questions that have been skipped. Principle of least reading effort: only read what seems to be necessary to maintain form filling progress. Research on Social Security forms in the UK by Frohlich (1986) found that on average, subjects failed to read over half the relevant explanations and instructions printed on the form. Subjects consistently read those explanations most necessary to understanding the questions and they paid particular attention to instructions next to tick boxes to know where to go next. They paid the least attention to instructions following answer spaces.To give instructions the best chance of being read, they need to be presented with their respective questions so that respondents see the information as relevant. If the information can be applied as soon as it is read, the respondent will remember the information and use it (see Jenkins & Dillman, 1997). In this context, instructions include mentions of the reference period as well as examples and other notes. Principle of question routing: jump directly to a new question if the form tells you to. It is too late to give instructions or explanations after the respondent has jumped to the new question. All notes, includings and excludings and other instructions should be between the question and the answer space, never below the answer space. Any sequencing should be placed in the appropriate position, after any information relevant to the respondent and before any irrelevant answer options. The combination of following skip instructions immediately and not giving all instructions full attention leads to another reason why respondents should not be explicitly directed to the next question in a form. When forms use "go to next" instructions respondents can learn to assume, incorrectly, that all "go to" instructions mean go to the next question. This reduces the effectiveness of "go to" instructions asking them to skip the following question/s (Barnett, 1991). Principle of question omission: miss out questions that don't seem to apply to you. Remember you do not have the advantage of actual dialogue or conversation, where if you ask respondents something they don't understand, or don't think is applicable to them, they can say:
or 'That doesn't apply to me' The respondent giving information on a self-administered form cannot tell you this and will either leave the question unanswered, give an answer to suit the question the respondent thinks you should have asked or even change the question. Sometimes it is appropriate to include a "not applicable" option in questions so that respondents are able to record that they do not think it applies to them. Form designers should avoid questions that allow respondents to correctly leave the answer space blank when they have been routed to that question, as it is then unclear whether the respondent should have recorded an answer or not.There are three other principles which cover respondent behaviour at points of confusion: Principle of question preview: If in doubt about the meaning of the current question, read the next question. Principle of question review: If in doubt about your interpretation of the previous question, review that question and the answer provided. Principle of topic scan: If in doubt about the relevance of the current question topic, scan the local topic context. It is important, therefore, that questions are presented in a meaningful order and grouped by topic. This enhances both the comprehension and the recall of the respondent. By careful question design and testing you should prevent any confusion arising in the first place. Be aware of any context effects the surrounding questions may have. Layout of questions A question starts with a question number in the left margin. The main question text is presented on the same line as the question number with the answer box to the right or below it and any instructions in between the text and the box. A question may contain sub-questions if there is a strong association between the parts or if the main question is a filter to the next main question. When a question carries on to the following page, the main question text should be repeated with "- continued" after it to help keep the respondent oriented within the form. Labelling the sub-parts of a question should be used whenever the sub-parts are long or have their own instructions etc. so that the labelling is useful to make it clearer which bits belong together. If there are sub-sub parts they would need to be quite short and simple otherwise the whole question gets too complicated and the sub-question should be made a proper question by itself. The labels should follow the pattern 1 (a), (b), (c), 2 (a) and so on, so that each labelled question has its own answer box, (see Diagram 6.3). Decimals to label questions and sub-questions e.g. 1, 1.1, 1.2, should not be used on ABS forms. Diagram 6.3 Use of sections or parts When a form contains groups of questions on different topics, it can aid the respondent to give these groups distinct part or section headings, such as "Income" and "Expenses". This is especially useful for surveys where the respondents do not need to complete all of the groups of questions. Sections or parts should be based on fairly broad topics and the questions within the section or part should always be more related to each other than they are to the questions in the surrounding groups. To help the respondent navigate through the form and realise that the topic has changed, it is preferable for a new section or part to start on a new page. In a long form, section or part headings should be labelled sequentially using whole numbers or letters. When using sequencing to sections or parts it is recommended that you label the section or part with letters (for example, Part A - General information, Part B - Employment) rather than numbers to reduce the risk of the respondent getting confused with sequencing to question numbers. Overuse of sections or parts, like any other element on the page, makes the form too visually complicated. Forms designers should always try to avoid having a section or part that contains only one question and in general the headings should not replicate question text as this increases clutter without aiding comprehension. Sections or parts should not be used instead of questions- they are headings only. When used, sections or parts must be used consistently throughout the whole form and every question must belong to one. Similarly, when using sub-parts or sub-sections, these must be used consistently within the part or section, so that every question belongs to a sub-part or sub-section. If some questions within a part belong to a sub-part and other questions do not, it is difficult for the respondent to tell which questions go together, and therefore what the questions mean. Open-ended questions There are two basic types of questions, open-ended and closed. These give respondents different degrees of freedom when choosing an answer. Open-ended questions allow respondents to give the answer in their own words, rather than having to select from options (for example see Diagram 6.4). Diagram 6.4 The answer box or area should allow sufficient space for a high percentage of the likely answers. If you use open-ended questions you may have to give some examples or directions on how to answer, for example (see Diagram 6.5). Diagram 6.5 You may also have to make it clear what the respondent should do if a non-response, not applicable, or zero answer applies, for example see Diagram 6.6. Diagram 6.6 The question needs to be understood by all respondents in the same way. Even an apparently simple question can be understood in different ways, for example, the following question (see Diagram 6.7) may be answered in several ways. Diagram 6.7 A more precise way of asking for this information, indicating the unit of measurement the respondent should use, is shown in Diagram 6.8. Diagram 6.8 Advantages of open-ended questions They allow a large number of possible answers, and they collect exact values from a wide range of possible values. Open-ended questions are often used in initial pilot testing to determine the range of possible answers and the availability of the data being sought. Disadvantages of open-ended questions This type of question is more demanding than, say, tick boxes because respondents have to write out an answer. Open-ended questions are also more demanding on office processing. You will need to create a coding frame to interpret a variety of responses, which will differ from each other in detail and accuracy, for example (see Diagram 6.9). Diagram 6.9 These responses could result in two different codes, depending on your coding frame. A better approach is to ask for exactly what is wanted including an example of the detail required, for example (see Diagram 6.10). Diagram 6.10 Open-ended questions can also cause problems because of poor handwriting, particularly in an OCR form as the machines can scan text but the recognition is not as accurate as for numbers. Closed questions Closed questions provide respondents with definite choices. The respondent need only indicate which choice is the most appropriate, for example (see Diagram 6.11). Diagram 6.11 At some point in the form, the respondent should be instructed how to complete closed questions, i.e. to tick or cross the box for the right option. This instruction usually goes in the "Please read this first" box and is repeated in the caption headings as in Diagram 6.12. "Tick" is used in most ABS business forms, in part because it is seen as an affirmation whereas "cross" can be seen as a negation, making the response more ambiguous. However "cross" is also accepted because respondents tend to keep their crosses more within the boundaries of the given box than their ticks. Question designs that require other methods of completion such as the circling of the appropriate option, or the crossing out of incorrect options, are not acceptable on ABS forms. Closed questions have the following types of answers. Yes/No answers Respondents choose either 'Yes' or 'No', for example (see Diagram 6.12). Diagram 6.12 Limited choice answers A number of choices are offered, only one of which should be chosen, for example (see Diagram 6.13). Diagram 6.13 Multiple choice answers More than one answer can be chosen, for example (see Diagram 6.14). Diagram 6.14 You must provide all possible alternatives in a closed question, including choices to cater for a non-response, zero, or not applicable answer. No alternatives should be left out or implied. The alternatives should be self-explanatory and mutually exclusive (see Diagrams 6.15 and 6.16). Bad: Diagram 6.15 Good: Diagram 6.16 You should take care to provide answer categories that reflect the respondents' characteristics or experience, for example the following question (see Diagram 6.17) would be difficult to answer unless records of the quantity of fuel bought are kept by the respondent. Diagram 6.17 As far as possible, answers should encompass the full range of variation across the survey sample or population, while keeping the list of options to a manageable length. Use words and terms in the answer choices that the respondent is familiar with. You must be sure through testing that respondents understand the words. Bias Answer choices can be of two types: ordered (see Diagram 6.18) or unordered (see Diagram 6.19). Diagram 6.18 Whenever there is an inherent order in the list of options, this should be used. For example, a forms designer should never present a list like that in the above example with the "1 year" option after the "Over 1 year" option. Diagram 6.19 Unordered answers are those where there is no graduation of answers. When using unordered answers you should try to control bias in the pattern of answers (see Schwarz, Hippler & Noelle-Neumann, 1992). Possible solutions are to list the most common alternative first (which also reduces the reading that respondents have to do), or to list choices randomly. The longer the list of options, the more pronounced the effects of order become. To avoid respondents choosing from only the first few options of a long list so they don't have to read the others, options should be categorised into groups or sub-questions. Unordered lists should be no longer than about ten items long, fewer if each item is longer than two or three words. Forms designers should also phrase the alternatives similarly. The alternatives should be of similar length and be similar in other respects. Forms designers should also be aware that presented with a list of options, the respondent may interpret the options as having an order that was not intended. This perceived order could lead the respondent to misinterpret the question or give inconsistent answers. Rating scales Questions that ask respondents to rate things, the favourability of an item, the frequency of a behaviour, and so on, are a type of ordered closed question. Traditionally an integral part of household surveys, rating scales are increasingly being used in ABS business surveys due to a shift away from purely accounting-type questions towards more motivational and behavioural items. Rating questions should be presented in a similar format to other closed questions, using a list of options with tick boxes (see Diagram 6.20). The different options should have verbal labels, not numbers, because the labels help the respondent understand the question and reinforce that a rating scale is an ordinal measure not an interval or ratio one. The list should be presented vertically down the page the same way as other closed question options unless the scale is being used as a heading in a matrix. Diagram 6.20 One of the most important considerations when designing rating scales is that the categories are balanced. This means that there should be an equal number of positive and negative options to choose from. Using more options for one side (usually positive) biases the responses towards that side for several reasons. These include: social desirability, where the lack of sufficient negative options leads the respondent to believe a negative response is discouraged (see e.g. Tourangeou & Smith, 1996); using the categories as information so that the respondent might incorrectly assume the range represents the true distribution of the population (see e.g. Schwarz & Hippler 1991) and so agree with the majority; and central tendency where the respondent picks the middle category regardless of the label. For similar reasons, it is generally preferable that rating scales include a neutral option such as "Neither satisfactory or unsatisfactory", or "Don't know" in the middle of the scale or "Not applicable" at either end. These are important because again, respondents should have a formal way to indicate when (they think) a question does not apply to them. If forced, respondents will produce an opinion on the spot that is neither accurate nor stable (see e.g. Flynn, 1996). The appropriate neutral option can be difficult to determine and form designers in doubt should see DCM for advice. So that the question itself does not bias the respondent, it is important to word the rating question in a balanced way, e.g. "Do you favour or oppose...." to imply that responses in either direction are acceptable (see Schuman & Presser, 1981). It is also generally better if negative options are presented first in the list, especially for sensitive questions. As part of balancing the options it is important to test whether the wording chosen for negative and positive options, e.g. "Difficult" and "Easy" are actually considered to be opposites by respondents. The negative and positive options should also be equivalent in intensity, e.g. "Very difficult" versus "Very easy" rather than "Extremely difficult" versus "Fairly easy". In general, three or five categories are the appropriate number of options for rating scales on ABS forms. For five categories, these would consist of one low and one high intensity option for each direction and one neutral. If required seven categories may be used but any more than that generally makes too fine a distinction between ratings that are of necessity vague. Ranking questions Ranking questions are those which ask the respondent to number the different options in a question in order of importance (see Diagram 6.21). Diagram 6.21 In general ranking questions should be avoided for two reasons:
The recommended way of measuring this sort of item is ask respondents to rate each item individually, using a verbal scale rather than numeric similar to other rating questions. This would allow the respondents to imply a ranking order where there was one, rate items the same where there was no difference, and indicate where items were not applicable. The respondent would rate the importance of each item by ticking a box under the scale of e.g. "Very important, Important, Somewhat important, Not at all important, Not applicable". This would be presented in a simple matrix with items down the left side and the scale along the top so the respondent could refer to the scale easily but without it being repeated for each item (see for example, Diagram 6.22) Diagram 6.22 Advantages of closed questions Closed questions are cheaper and easier to process and easier to respond to (assuming that all of the answer choices are applicable to the respondents). Closed questions are an advantage when:
Disadvantages of closed questions You will need to put more effort into developing closed questions than into open-ended questions. The respondent is not given the opportunity to compensate for a poorly framed closed question. Partially closed questions If you find that it is difficult or impractical to present all possible choices, the list of alternatives should end with: 'Other (please specify)' followed by an appropriately sized answer box for respondents to specify the nature of their "other". This is then called a partially closed question which is a compromise between open and closed questions. Choices are provided but respondents have the option of creating their own choice, for example (see Diagram 6.23). Use this question type where most responses can be anticipated but the remainder cannot. Diagram 6.23 Choosing between question types Your choice between open-ended, closed and partially closed questions will depend on:
A closed question generally desensitises a question. For example, asking for personal income by using income groups is likely to be less sensitive than asking for a dollar value. You will have to test any question to find out whether you have made the correct choice. Use of classifications Formal classification systems (such as Australian and New Zealand Standard Industrial Classifications (ANZSIC)) are an attempt to describe in structured categories something which is almost infinitely varied. You will come across two main problems in adopting formal classification systems:
When you develop questions you will sometimes have to make compromises with the formal classification used by the ABS in order to get useful answers from respondents. This should always involve consultation with the appropriate standards area. The classification has to be understood by the respondent before it can work. Therefore, all classifications you intend to use on the form should be thoroughly tested with potential respondents to find out:
For example, to collect occupational data from a population of health employees, consult with Population Statistics standards section on the most suitable Australian Standard Classification of Occupation (ASCO) categories to use as a basis for the response categories of the question. Where you know, following testing, that respondents are going to experience difficulties in using the classification, investigate alternatives in the following order:
Diagram 6.24 All of these alternatives should be tested before deciding on the best method for your particular collection. Sequence of questions The questions on a form should follow a sequence that is logical to the respondents. You should design the sequence of questions to:
Group questions into areas of related topics and ask all questions relating to that topic before going on to the next one, for example: If users want to measure the income generated by a business in a particular industry they need data by specific relevant categories, as shown in the next example (see Diagram 6.25). Diagram 6.25 If the data are to be collected from accountants, useful groupings would be to follow the headings of Income and Expenditure, Statement of Financial Performance or Statement of Financial Position (see Diagrams 6.26 and 6.27). Diagram 6.26 Diagram 6.27 However, such a grouping may only seem logical to an accountant. Choose a logic or grouping that reflects the understanding of particular respondents. Location of sensitive questions You should give thought to the location of questions that may be sensitive to the respondent. Generally, sensitive questions should not be placed at the beginning of the form. Place them in a section of the form where they are most meaningful in the context of other questions. In the following example (see Diagram 6.28) which shows an Income question from an unincorporated business, the item 'owner's drawings' might be sensitive. Putting it in the context of this question may have the effect of making it less sensitive. Diagram 6.28 When a sensitive question has several options for the respondent to select from, the less desirable options should be presented first in the list. This indicates to respondents that it is acceptable for them to choose those options (see Diagram 6.29). Similarly, yes/no questions without sequencing should generally present the "no" option first because it is seen as socially desirable to the respondent to answer yes to any neutral question (called acquiescence). Diagram 6.29 Filter questions Some questions, usually referred to as filter questions, ask respondents to make a choice, where one of the choices leads them to the next question, while the other choice leads them to a different question or place on the form. A filter question is used to exclude respondents from subsequent questions if they do not apply, for example (see Diagram 6.30). Filter questions help the respondent to understand the sequence of questions. Diagram 6.30 It is important that filter questions are used carefully. Respondents should not be made choose between options without enough information- an instruction box with definitions or includings will often be necessary. Sometimes respondents cannot make an informed judgement without seeing the contents of the following questions, including in some cases the available response options. Sometimes respondents may not remember pertinent activities until they read further. In these cases a "none of the above" at the end is more appropriate (see Diagram 6.31). Sensitive topics should not use filter questions because they discourage reporting. Diagram 6.31 In general, place the sequenced response first in filter questions because that response takes the respondent to a subsequent question and then they do not have to read the other option. The order of the 'No'/'Yes' answers should be consistent throughout the form. Filter questions are preferable to conditional questions (see Diagram 6.32). With a conditional question it is unclear whether a blank answer is:
Diagram 6.32 Make questions simple A complicated or double-barrelled question is a potential source of non-response and increases the likelihood of errors, for example (see Diagram 6.33). Diagram 6.33 This question can be made simpler as follows (see Diagram 6.34). Diagram 6.34 This places the broadest or most common category first and uses the specialised or less common categories as examples. An indicator of a complex question is the use of conjunctions 'and', and 'or'. Qualifiers included in the body of a question or questions with a lot of information carrying words may cause problems. Qualifiers may impose or add an unfamiliar concept to what was a familiar one. Too many information carrying words may interfere with the respondents' grasp of the main element of a question, for example (see Diagram 6.35). Diagram 6.35 This question could be improved as follows (see Diagram 6.36). Diagram 6.36 Short qualifiers can be presented at the end of a question in brackets and plain type, see Diagram 6.37 Diagram 6.37 Often two questions are far better than one complex question. For example Diagram 6.38 below shows a much simpler set of questions compared to the complex single question , in Diagram 6.39. Diagram 6.38 Diagram 6.39 Memory bias You should be aware of bias due to respondents' poor memory. This has varying effects but common ones are:
Diagram 6.40 The answer to this question will be boosted by those who bought a car in the last year to year and a quarter or more. Try to ask questions that relate to some form of record keeping to enhance accurate reporting (see Diagram 6.41). Diagram 6.41 Remember How the question is presented (open, closed, partially closed) and the layout used are as important as the words used to ask it. You must always test questions with actual respondents to find out whether the questions can be understood and answered accurately. References
|
||||