4363.0.55.001 - Australian Health Survey: Users' Guide, 2011-13
ARCHIVED ISSUE Released at 11:30 AM (CANBERRA TIME) 05/08/2013
Page tools:
 Print Page
 Print All
| ||||
DATA QUALITY
Measure of sampling variability One measure of the likely difference is given by the standard error (SE), which indicates the extent to which an estimate might have varied because only a sample of dwellings was included. There are about two chances in three that the sample estimate will differ by less than one SE from the figure that would have been obtained if all dwellings had been included, and about 19 chances in 20 that the difference will be less than two SEs. This is known as the margin of error (MoE) at the 95% confidence level. The margin of error at the 95% confidence level is expressed as 1.96 times the SE. The 95% confidence interval is the estimate +/- MoE i.e. the range from minus 1.96 times the SE to the estimate plus 1.96 times the SE. Another measure of the likely difference is the relative standard error (RSE), which is obtained by expressing the SE as a percentage of the estimate to which it relates. The RSE is a useful measure in that it provides an immediate indication of the percentage errors likely to have occurred due to sampling, and thus avoids the need to refer also to the size of the estimate. More detail on the calculation of SEs, MOEs and RSEs can be found in the Technical Note. Proportion estimates annotated with a hash (#) have a margin of error greater than 10%. Users should give the margin of error particular consideration when using this estimate. Note, for the AHS the first annotation of MoEs for proportion estimates commenced with the release of the Nutrition First Release - Foods and Nutrients publication (cat. no. 4364.0.55.007) in May 2014. Publications with MoEs prior to this release do not contain annotations based on MoEs. Estimates with relative standard errors less than 25% are considered sufficiently reliable for most purposes. However, estimates with relative standard errors of 25% or more are included in ABS publications of results from this survey. Estimates with RSEs greater than 25% but less than or equal to 50% are annotated by an asterisk to indicate they are subject to high SEs relative to the size of the estimate and should be used with caution. Estimates with RSEs of greater than 50%, annotated by a double asterisk, are considered too unreliable for most purposes. These estimates can be used to aggregate with other estimates to reduce the overall sampling error. Relative standard errors for estimates are published in 'direct' form. In NHSs prior to 2007-08, a statistical model was produced that related the size of estimates to their corresponding RSEs, and this information was displayed via a standard error table. For 2011-12 NHS and NNPAS, RSEs for estimates were calculated for each separate estimate and published individually using a replicate weights technique (Jackknife method). Unlike the previous method, direct calculation of RSEs can result in larger estimates having larger RSEs than smaller ones, since these larger estimates may have more inherent variability. More information about the replicate weights technique can be found in the Technical Note. Standard errors of proportions, differences and sums Proportions formed from the ratio of two estimates are also subject to sampling error. The size of the error depends on the accuracy of both the estimates. The difference between, or sum of, two survey estimates (of numbers or percentages) is itself an estimate and is therefore also subject to sampling error. The SE of the difference between, or sum of, two survey estimates depends on their SEs and the relationship between them. The formulas to approximate the RSE for proportions and the SE of the difference between, or sum of, two estimates can be found in the Technical Note. Testing for statistically significant differences For comparing estimates between surveys or between populations within a survey it is useful to determine whether apparent differences are 'real' differences between the corresponding population characteristics, or simply the product of differences between the survey samples. One way to examine this is to determine whether the difference between the estimates is statistically significant. This is done by calculating the standard error of the difference between two estimates (x and y) and using that to calculate the test statistic using the formula below: (x-y) ________________ SE(x-y) If the value of the test statistic is greater than 1.96, then we may say that we are 95% certain that there is a statistically significant difference between the two populations with respect to that characteristic. Otherwise, it cannot be stated with confidence that there is a real difference between the populations. Non-sampling error Lack of precision due to sampling variability should not be confused with inaccuracies that may occur for other reasons, such as errors in response and recording. Inaccuracies of this type are referred to as non-sampling error. This type of error is not specific to sample surveys and can occur in a census enumeration. The major sources of non-sampling error are:
These sources of error are discussed below. Errors related to scope and coverage Some dwellings may have been inadvertently included or excluded because, for example, the distinctions between whether they were private or non-private dwellings may have been unclear. All efforts were made to overcome such situations by constant updating of lists both before and during the survey. Also, some persons may have been inadvertently included or excluded because of difficulties in applying the scope rules concerning the identification of usual residents, and the treatment of some overseas visitors. Response errors Response errors may have arisen from three main sources:
Errors may be caused by misleading or ambiguous questions, inadequate or inconsistent definitions of terminology used, or poor overall survey design (for example, context effects where responses to a question are directly influenced by the preceding questions). In order to overcome problems of this kind, individual questions and, the questionnaire overall, were thoroughly tested before being finalised for use in the survey. Testing took two forms:
As a result of both forms of testing, modifications were made to question design, wording, ordering and associated prompt cards, and some changes were made to survey procedures. In considering modifications, it was sometimes necessary to balance better response to a particular item/topic against increased interview time or effects on other parts of the survey. For example, questions for collecting data on usual intake of fruit and vegetables referred to consumption in the form of 'serves', which required the use of a prompt card to define a serve, and a fair amount of recall and calculation on the part of the respondent. Although every effort was made to minimise response errors due to questionnaire design and content issues, some errors will inevitably have occurred in the final survey enumeration. As the survey is quite lengthy, reporting errors may also have resulted from interviewer and/or respondent fatigue (i.e. loss of concentration), particularly for those respondents reporting for both themselves and a child. Inaccurate reporting may also occur if respondents provide deliberately incorrect responses. While efforts were made to minimise errors arising from fatigue, or from deliberate misreporting or non-reporting by respondents, through emphasising the importance of the data and checks on consistency within the survey instrument, some instances will have inevitably occurred. Reference periods used in relation to each topic were selected to suit the nature of the information being sought. In particular to strike the right balance between minimising recall errors and ensuring the period was meaningful and representative (from both respondent and data use perspectives), and would yield sufficient observations in the survey to support reliable estimates. It is possible that the reference periods did not suit every person for every topic and that difficulty with recall may have led to inaccurate reporting in some instances. Lack of uniformity in interviewing standards may also result in non-sampling errors. Training and retraining programs, and checking of interviewers’ work were methods employed to achieve and maintain uniform interviewing practices and a high level of accuracy in recording answers on the survey questionnaire (see the Interviews section of the Data collection page). The operation of the Computer Assisted Instrument (CAI) itself, and the built in checks within it, ensure that data recording standards are maintained. Respondent perception of the personal characteristics of the interviewer can also be a source of error, as the age, sex, appearance or manner of the interviewer may influence the answers obtained. Non-response bias Non-response may occur when people cannot or will not cooperate in the survey, or cannot be contacted by interviewers. Non-response can introduce a bias to the results obtained insofar as non-respondents may have different characteristics and behaviour patterns in relation to their health to those persons who did respond. The magnitude of the bias depends on the extent of the differences and the level of non-response. The 2011-12 NHS and NNPAS achieved an overall response rate of 85% and 77% respectively (fully/adequate responding households, after sample loss). Data to accurately quantify the nature and extent of the differences in health characteristics between respondents in the survey and non-respondents are not available. Under or over-representation of particular demographic groups in the sample are compensated for at the State, section of State, sex and age group levels in the weighting process. For NNPAS, a seasonal adjustment was also undertaken for person weights. Other disparities are not adjusted for. Households with incomplete interviews were treated as fully responding for estimation purposes where the only questions that were not answered were legitimate 'don't know' or refusal options, or any or all questions on income, or where weight and height were not obtained. These non-response items were coded to 'not stated'. To improve the sample representativeness for the biomedical component, as well as the second day dietary recall and pedometer component in NNPAS, households that contained at least one fully responding person but also contained a selected adult or child who did not respond to the survey, were also retained and considered adequately responding. This process added an additional 77 and 162 households respectively to the NHS and NNPAS samples. Note that this is different to previous Health Surveys, which have only retained households with complete surveys for all selected respondents. Processing errors
Other factors affecting estimates In addition to data quality issues, there are a number of both general and topic-specific factors which should be considered in interpreting the results of this survey. The general factors affect all estimates obtained, but may affect topics to a greater or lesser degree depending on the nature of the topic and the uses to which the estimates are put. This section outlines these general factors. Additional issues relating to the interpretation of individual topics are discussed in the topic descriptions provided in other sections of this Users' Guide. Scope The scope of the survey defines the boundaries of the population to which the estimates relate. The most important aspect of the survey scope affecting the interpretation of estimates from this survey is that institutionalised persons (including inpatients of hospitals, nursing homes and other health institutions) and other persons resident in non-private dwellings (e.g. hotels, motels, boarding houses) were excluded from the survey. Coverage The NHS and NNPAS surveys include all geographic areas except very remote and migratory. Due to the close timing of the National Aboriginal and Torres Strait Islander Health Survey (NATSIHS) and the National Aboriginal and Torres Strait Islander Nutrition and Physical Activity Survey (NATSINPAS), all persons living in discrete Aboriginal and Torres Strait Islander communities in any geographic location were also excluded, as well as a small number of persons living within Collection Districts that include Aboriginal and Torres Strait Islander communities. Personal interview and self-assessment nature of the survey The NHS and NNPAS were designed using personal or proxy (e.g. parent or guardian answering for a child, or a carer answering for a disabled person) interviews to obtain data on respondents’ own perceptions of their state of health, their use of health services and aspects of their lifestyle. The information obtained is therefore not necessarily based on any professional opinion (e.g. from a doctor, nurse, dentist, etc.) or on information available from records kept by respondents. Concepts and definitions The scope of each topic and the concepts and definitions associated with individual pieces of information should be considered when interpreting survey results. This information is available for individual topics of this Users’ Guide. Wording of questions To enable accurate interpretation of survey results it is essential to bear in mind the precise wording of questions used to collect individual items of data, particularly in those cases where the question involved ‘running prompts’ (where the interviewer reads from a list until the respondent makes a choice), or where a prompt card was used. The 24-hour dietary recall in NNPAS makes extensive use of prompt cards in a ‘Food Model Booklet’ (available from the Downloads page of this product) to assist respondents to more accurately report measurement or portion sizes. Further information on the 24-hour dietary recall can be found in the Nutrition chapter of this Users' Guide. Testing has shown that reporting of medical conditions is improved where direct questions are asked about a specific condition or where conditions are specifically identified in a prompt card, and that data is less robust where it is up to the respondent to identify conditions in response to a general question. It is not possible or practical to mention all conditions in questions or prompts, therefore the approach taken in the NHS and NNPAS was to identify main conditions which include (where applicable):
In the NHS an additional list of conditions was also provided as a prompt for other long-term conditions. As some conditions are specifically identified in the questionnaire and others are not, response levels and accuracy of condition reporting may be affected. Where the level and nature of condition identification has changed between surveys, comparability over time may be affected. Further information on the collection methodology for conditions can be found in the Health conditions chapter of this Users’ Guide. For further information on question wording please refer to the survey questionnaires available from the Downloads page of this product. Reference periods All results should be considered within the context of the time references that apply to the various topics. Different reference periods were used for specific topics (e.g. 'in the last week' for alcohol consumption and adult physical activity, 'ever' and 'in the last 12 months' for actions taken, midnight to midnight the day prior to interview for 24-hour dietary recall). Although it can be expected that a larger section of the population would have reported taking a certain action if a longer reference period had been used, the increase is not proportionate to the increase in time. This should be taken into consideration when comparing results from this survey to data from other sources where the data relates to different reference periods. Classifications and categories The classifications and categories used in the survey provide an indication of the level of detail available in survey output. However, the ability of respondents to provide the data may limit the amount of detail that can be output. Where respondents may have used non-medical terminology, symptoms rather than conditions, or generic rather than specific terminology, conditions may only be able to be output in general terms (e.g. 'heart condition nfd' rather than 'Angina' or 'Atrial fibrillation'). Classifications used in the survey can be found in Appendix 3: ABS Standard Classifications. Survey specific classifications can be found in Appendix 2: Classification of Health Conditions, Appendix 6: Classification of Adult Physical Activity, and Appendix 7: Classification of Child Physical Activity. Collection period The 2011-12 NHS was enumerated from 6 March 2011 to 17 March 2012, the NNPAS was enumerated from 29 May 2011 to 9 June 2012, and the NHMS collection occurred during the period of March 2011 to September 2012. When considering survey results over time or comparing them with data from another source, care must be taken to ensure that any differences between the collection periods take into consideration the possible effect of those differences on the data, for example, seasonal differences and effects of holidays. Biomedical quality control and quality assurance
Analysis of the results of the 1995 National Nutrition Survey (NNS) and the 2011-12 NNPAS suggests that, like other nutrition surveys, there has been some under-reporting of food intake by participants in these surveys and that the patterns of under-reporting have changed over time. It is difficult, from the available data, to accurately estimate the amount of under-reporting that has occurred and therefore how much energy and nutrients might be missing from the intakes reported by respondents. One method is to estimate the mean amount of energy for the population to achieve an EI:BMR ratio of 1.55 (i.e. the conservative minimum energy requirement for a normally active but sedentary population). Using this method, it is estimated that the average energy intakes may be understated by as much as 17% in males and 21% in females. The factor most closely associated with under-reporting was BMI, where people who were overweight or obese were most likely to have lower than expected energy intakes. For more information, see the Under-reporting in Nutrition Surveys page of the Nutrition chapter in this Users' Guide. Another factor affecting the accuracy of the 24-hour dietary recall data is that most young children are unable to recall their intakes. Similarly, parents/carers of school-aged children may not be aware of a child’s total food intake, which can lead to systematic under-reporting. Young children were encouraged to assist in answering the dietary recall questions. See the Interviews section of Data collection for more information on proxy use in the 24-hour dietary recall module. Another non-sampling error specific to Nutrition surveys is the accuracy of the nutrient and measures database containing thousands of foods used to derive the nutrient estimates. The databases used for the 2011-12 NNPAS were developed by Food Standards Australia New Zealand specifically for the survey. A complete nutrient profile of 44 nutrients was created based on FSANZ’s latest available data, however, not all data was based on directly analysed foods. Some data was borrowed from overseas food composition tables, food label information, imputed data from similar foods or data calculated using a recipe approach. See AUSNUT 2011-13 for more information. There are two sections within this Users' Guide that refer to aspects of data quality of the 24-hour dietary recall data. Food and Measure Coding had specific quality assurance processes including dual-coding, adjudication and quality review by FSANZ. Major aspects of the Nutrition data editing and validation strategy are outlined in the Data Quality page of the Nutrition chapter of this Users' Guide. Usual nutrient intakes model-based error and quality assurance Usual intakes are estimated using a statistical model applied to the two days of nutrient intake data from 2011-12 NNPAS. In addition to the types of sampling and non-sampling error described in this section, these model-based estimates are also subject to prediction error and simulation variance. Model bias cannot be explicitly captured, however every effort is made to ensure an appropriate model specification is used through external literature research and statistical testing. Information on data quality of the usual nutrient intakes is available in the Data Quality page of the Usual Nutrient Intakes chapter of the Users' Guide.
|
||||