General forms design principles: Question type

The type of question you are asking has important implications for many aspects of the survey cycle beyond the data collection phase (e.g. data processing and data analysis).

Consider the following guidelines to inform decisions made when developing new questions or updating old ones:

• determine the type of data to be collected
• determine the answer format required
• determine the type of measurement to be used.

Refer to the additional resources for further information about the types of questions used in surveys.

Use this chapter in conjunction with the 'Question structure' chapter which provides guidelines on asking, arranging, and presenting questions.  

Use 'factual' questions to gather facts or concrete, quantifiable information, and observable phenomena.

Responses to 'factual' questions can usually be verified through:

• asking another similar question
• conducting a post-enumeration study (an activity completed following a survey to evaluate the accuracy of the data collected) 
• other independent sources of information (e.g. records and documents, another observer of the event).

Respondents can answer 'factual' questions through checking their records (Diagram 1), recall (Diagram 2) or the use of classifications (Diagram 3). 

Survey questions can also be used to test factual knowledge (e.g. Who is the Prime Minister?).

Use 'non-factual' questions to gather information on subjective phenomena including attitudes, beliefs, awareness, knowledge, and preferences.

Responses to 'non-factual' questions cannot usually be verified (e.g. what motivates a business and the impact of caring on a relationship shown in Diagram 4 and Diagram 5 respectively).

Note that responses to 'non-factual' questions can be influenced by:

• attitudes that are not fully developed
• limited in-depth thinking around the issue
• changes to question wording
• context effects, including the presence of other people.

It should be noted that the distinction between non-factual and factual questions is more of a continuum than a dichotomy. Some questions are not factual or non-factual, but rather 'somewhere in between.' For example, the following question does not fall clearly in the factual or non-factual category: 'Do you expect to be working for this (employer/business) in 12 months' time?'

Avoid the use of proxy reporting for non-factual questions (e.g. attitudinal questions) because the respondent answering questions on behalf of other household members generally could not know what another person would report.

Use behavioural questions to gather information about respondents' activities based on factual circumstances (Diagram 6).

Make it easier for respondents to answer behavioural questions by:  

• setting a reasonable and specific time frame (e.g. Last week...)
• covering topics that respondents can remember easily
• encouraging record checking, where possible. 

Also see 'Be aware of memory bias' in the 'Question structure' chapter.

Questions about behaviour can also be hypothetical (e.g. 'What would you do if...?') but these types of questions are best avoided because responses tend to be unreliable. 

Hypothetical questions are best used when respondents are familiar with the situation (e.g. asking a respondent responsible for the payroll, 'If an employee went on holidays at the end of January and was paid in advance for all of February, would you include them in the number of employees reported for the pay period ending on or before 21 February?').

Include demographic questions so that the main groups of respondents can be identified (e.g. by age, industry etc.). This can help explain the survey findings.

Place demographic questions at the end of the survey where possible because:

• they are not 'simple' starter questions (topics such as sex and age can be sensitive and complicated for some respondents)
• the added context of the survey content encourages respondents to answer honestly
• it helps ensure details for the right respondent are captured if more than one person is completing the questionnaire.

They can be placed at the start of the survey in some circumstances, but this section should be kept as brief as possible.

Place demographic questions throughout the survey as filter or sequencing questions when necessary.

There are three types of questions: open-ended, closed and partially-closed.

The answer format determines the degree of freedom respondents have when answering a question.

Consider the following factors when determining which question type to use

• The level of detail that data users require (e.g. for 'total income' respondents can either select from an income range or write in an exact dollar amount)
• What information is potentially available from the respondent (e.g. do respondents have enough detailed information to report an exact dollar amount?)
• The position of the questions in the form
• Whether the survey is a one-off or a continuing request
• Whether it is a sensitive question (also see 'Carefully consider where sensitive questions are placed' in the 'Question structure' chapter)
• How the data will be processed (e.g. captured electronically or manually coded)
• Availability of your organisation's resources (e.g. time, money, staff)

Test the question to determine whether the correct type has been selected. 

Open-ended questions allow respondents to freely enter their answer, rather than having to select from options (Diagram 7).

Ensure the size of the answer box is appropriate for the expected responses, with a larger box for longer responses (Diagram 8), and a smaller one for shorter responses (Diagram 9).

Provide examples or directions on how to answer an open-ended question (Diagram 10).

Advantages of open-ended questions are that they: 

• allow many possible answers
• obtain the exact value from a wide range of possible values
• add richness to responses that is difficult, or impossible, to achieve through closed questions
• determine the range of possible answers and the availability of the data being sought during initial testing
• can speed up an interview because there are no response options for the interviewer to read aloud.

Disadvantages of open-ended questions are that they:  

• are time consuming to answer. Respondents must write out or verbally formulate an answer compared to selecting a response in a close-ended question
• are more resource intensive to process manually where a coding frame is used to interpret responses. This is because responses will often differ in detail and accuracy making them difficult to categorise (Diagrams 11a and 11b) 

• can be difficult when electronic scanners can not accurately recognise poor handwriting, especially numbers, when using paper forms that rely on Optical Character Recognition (OCR)

• can be difficult for interviewers to code or write verbatim responses depending on the level of detail respondents provide.

Closed questions require respondents to select an answer from a list of response options (Diagram 12).

Include instructions that tell respondents how they should complete closed questions. These instructions can be placed in the 'Please read this first' box (Diagram 13), included in caption headings (Diagram 14) and mentioned in instructions specific to the question (Diagram 15).

Use terminology that makes sense for the survey mode when providing instructions for completing closed questions such as 'tick' in paper forms (Diagram 14) and 'select' in web forms (Diagram 16).

Do not instruct respondents to circle the appropriate option or cross out incorrect options.

Response options for closed questions must adhere to the following principles:

• Cover all possible response options, ensuring the list is exhaustive with no possible answers left out or implied.  
• Include options that cater for a zero, 'not applicable', 'don't know' or 'prefer not to answer' response if it makes sense for a particular question.
• Ensure that the response options provided reflect the respondents' characteristics or experience.
• Keep the list of response options to a manageable length.
• Ensure response options are self-explanatory.
• Use words and terms in the response options that are familiar to respondents. Test the question to ensure respondents understand the words as intended.
• Ensure that all response options for a particular question are mutually exclusive. Do not overlap your response categories.

Consider the advantages of closed questions which include:

• comparatively less effort for respondents to answer as an exact value is not needed
• less time taken to complete the survey because respondents can generally select their answer from a list of response options
• the ease and cost effectiveness of processing data because nearly all the responses can be anticipated
• making a question less sensitive (e.g. asking respondents to select from a personal income range is less sensitive than asking for an exact dollar value).

Consider the disadvantages of closed questions which include:

• the effort required to develop well-crafted questions that avoids the need for respondents to qualify their answers
• difficulty in developing comprehensive lists of response options
• slower administration during interviews if a running prompt needs to be used, where each response option is read out and respondents must indicate whether it applies to them or not 
• repetition when rating scales are used (e.g. satisfied/dissatisfied) in a personal interview because the scale must be verbally presented to the respondent each time it is used.

A partially-closed question is a mixture of an open-ended and closed question. Respondents can select an answer from a list of response options but also have the option of writing-in a response that is not on the list (Diagram 17).

Present partially closed questions by placing the list of pre-determined response options first and ending the list with an 'Other (please specify)' option followed by an appropriately sized answer box.

Consider the advantages of partially-closed questions which include:

• the ease and cost effectiveness of processing data compared to open questions 
• being able to still use a closed question format even when there is uncertainty around whether the list of response options is comprehensive
• having a format that is ideal for dress rehearsals or a pilot test instrument to help develop a more comprehensive list of response options for the final instrument.

Consider the disadvantages of partially-closed questions which include:

• instances where respondents restrict their responses to the choices that are explicitly offered and not use the 'Other (please specify)' option
• greater demand on office processing if the response options are not comprehensive and the 'Other (please specify)' option is used extensively.

Types of measurement include nominal categories, ranking questions, rating scales and numeric quantities.

Select the type of measurement to be used for data collection carefully because it impacts subsequent statistical procedures that can be used for data analysis.

Bryman, A. (2012). Social research methods. Oxford University Press.

Walliman, N. (2011). Research methods: The basics. Routledge Taylor & Francis Group.