SURVEY TESTING

Introduction
Survey testing is an important part of the survey development process and provides guidance on the :

• adequacy of the sampling frame
• variability of the target population with regard to the survey subject
• expected non-response rate and the effectiveness of measures aimed at reducing non-response
• suitability of the data collection method, including testing of various methods to determine the most suitable
• adequacy of the questionnaire, including testing of alternative versions to determine the most effective
• effectiveness of interviewer training and the adequacy of survey instructions
• answer categories to be used for pre-coded questions
• likely cost and duration of parts of the survey and the survey as a whole
• organisation of the survey.

In some cases, data collected in the tests may be useful preliminary indicators of the survey results; estimating sampling error, sample sizes and population variability; and estimating likely response rates. These preliminary results will involve a smaller sample and thus produce higher standard errors, which need to be taken into consideration.


TYPES OF TESTING
There are two main types of testing that can be used to evaluate a survey: Qualitative testing and Quantitative testing. Within qualitative testing are techniques such as skirmishing, focus groups, pretesting and observational studies. Pilot testing and dress rehearsals are quantitative tests. Each type of testing is used at a different stage of the survey's development and aims to test different aspects of the survey.

Skirmishing
Skirmishing is the process of informally testing questionnaire design with groups of respondents. The questionnaire is basically unstructured and is tested with a group of people who can provide feedback on issues such as the questions' frame of reference, the level of knowledge needed to answer the questions, the range of likely answers to questions and how answers are formulated by respondents.

Focus Groups
Focus groups are small groups of people who represent or who share similar characteristics with the target population. Focus groups are used to test concepts or questions relating to a small segment of the population, who may not be adequately represented in a larger pilot test.

Focus groups are usually small as large groups can be difficult to control. They need to be led by an experienced facilitator whose job is to control the group discussion and ensure that all participants have the opportunity to put their views forward. In conducting focus groups you should address the following issues:

• The relevant subject matter experiences of the group,
• The characteristics of the target population,
• How well do the participants understand the concepts and definitions used,
• The abilities of the group (e.g. can they add numbers if asked), and
• What is the general group reaction to the form wording and layout.

Focus groups can help us to refine our image of respondents, both in general terms and in relation to significant sub-groups of our survey population.

Informal discussions are held with the focus groups to gain an insight into the attitudes, concerns and responses of the group, related to the questionnaire. When introducing a new questionnaire, it is quite useful to gather a number of likely respondents together and discuss the issues involved. Out of these discussions can come such things as the availability of the data and details of the people best suited to supplying it.

The volume of quite specific information obtained from focus groups means that, depending on the nature of the concepts or questions being investigated, only one or two focus groups are required.

Due to the way focus groups examine very specific issues, they should not be used as the sole method of testing a new or revised questionnaire. Another danger with focus groups results from a phenomena called 'group speak', where problems can be built up to exaggerated proportions as members of the group 'feed off' each other during discussions.

Pretesting (also known as Skirmishing)
Pretesting is the process of informally testing questionnaire design with potential respondents. The questionnaire is basically unstructured and is tested with a group of people who can provide feedback on issues such as the questions' frame of reference, the concepts, the level of knowledge needed to answer the questions, the range of likely answers to questions and how answers are formulated by respondents. Pretesting is also used to detect obvious flaws or awkward wording of questionnaires as well as testing alternative designs. At this stage we may use open-ended response categories to work-out likely responses. The questionnaire should be redrafted after pretesting.

Observational Studies
Observational studies involve getting respondents to complete the draft questionnaire in the presence of an observer. Whilst completing the form, respondents explain their understanding of the questions and the methods required in providing the information. Much can be gained from such studies including identifying problem questions through observations, questions asked by the respondents, or the time taken to complete particular questions. Data availability and the most appropriate person to supply the information can also be obtained through observational studies. Respondents should be made aware that it is the form that is being tested and not the respondent. The respondent should not be given assistance in completing the form.

Pilot Testing
Pilot testing involves formally testing a questionnaire or survey with a small representative sample of respondents. Semi-closed questions can be used to gather a range of likely responses which are later used to develop a more highly structured questionnaire with closed questions. Pilot testing is used to identify any problems associated with the form, such as questionnaire format, length, or question wording, and allows comparison of alternative versions of a questionnaire. Pilot testing can also be used to assess the adequacy of instructions to interviewers and ascertaining interview times. For a new survey, several pilot tests may be necessary before the survey methodology is considered satisfactory.

Dress Rehearsals
A dress rehearsal is a final trial run of the survey where the chosen sampling methodology is used to select a small sample from the target population. Dress rehearsals are used to detect any problems that may arise in the survey design and/or processing system. Dress rehearsals also provide an opportunity to obtain data on survey costs and estimate population variances. Administrative costs, including average interview times as well as cash outlays, are obviously useful in deciding staffing and budgetary arrangements. Estimates of population variances appropriate to the sample design (or designs) under consideration may be needed to predict the sampling errors of any proposed sample plans.

If appropriate information is available from previous surveys, it may be possible to estimate population variances and costs from this information rather than from the results of a dress rehearsal.


CONDUCTING SURVEY TESTS
Composition of Survey Tests
Once testing procedures have been decided upon, the next step is to determine how these tests will be run. The samples chosen for pilot testing and dress rehearsals need to be as representative of the target population as possible. This maximises the validity of the test results and ensures that consequent modifications to the survey are appropriate. The sample may be chosen using judgement sampling to obtain a fairly representative sampling method or using probability sampling methods. Observation studies and focus group testing do not need representative samples for them to be carried out.

Comparison of Techniques
Comparisons between pilot tests and dress rehearsals can be used as a basis for choosing between alternative procedures for part of a survey. One approach to such testing could be to allocate two equal size samples to interviewers and each interviewer uses both of the alternative procedures. The two techniques can then be compared without the interfering bias of differing interviewer performance.

Frequency and Size of Tests
The number of times survey testing should be conducted and the sample sizes used are determined by the complexity of the survey and the availability of funds. For example, a simple, small survey may need only 50 respondents for a dress rehearsal, whereas a larger, more complex survey may need 200 or more respondents.


ANALYSIS OF TEST RESULTS
The results of survey testing can bring to light a number of problems with the questionnaire and survey design. Some of these problems may be identifiable during pre-testing; others may be identifiable only when the questionnaire is administered to a sample of the target population during a pilot test or dress rehearsal.

Problems with Questionnaires
Lengthy questionnaires will often intimidate respondents into rushing through questions or refusing to continue. This can lead to a high percentage of non-response and a general misunderstanding of questions. Poor layout and explanation of questions can also lead to non-response for questions or multiple answers being given when only one should be chosen. Other problems that may be identified include a large number of 'Other' responses to a particular question, lack of variation in the responses to a question and general misunderstanding of a question.

Solutions to Questionnaire Problems
The solution to these problems may lie in the sequencing of questions, the instructions accompanying questions, the wording of questions and the answer categories provided. Changing the answer categories can produce more variation in the responses given to a question. We can do this by increasing the list of answer categories, changing the order of the list or even changing the emphasis of the question and/or the answer categories.

Survey Design Problems
In addition to problems with questionnaire design, survey testing may also uncover problems with the sampling methodology and administrative calculations. These may include deficiencies in the sampling frame and/or sample size chosen, sources of sampling and non-sampling error, problems with the processing system and both budgetary and time constraints. Once identified, these problems can be addressed and either eliminated or minimised before the full survey is conducted. When used in this way, pilot testing is an invaluable tool for maximising, within resource constraints, the quality of results obtained from the final survey.

Preliminary Survey Results
The results of pilot testing can also be used to provide a 'preview' of the results of the full survey. The data gained from the test can be analysed in the same way as the final survey, incorporating tables, statistical analyses (e.g. correlations, scales, etc.) and discussion of the findings. The data will be subject to higher standard errors although it may still be useful.