2952.0 - Census Paper 02/02 - Report on Testing of Disability Questions for Inclusion in the 2001 Census, 2001  
ARCHIVED ISSUE Released at 11:30 AM (CANBERRA TIME) 22/11/2002   
   Page tools: Print Print Page Print all pages in this productPrint All

Introduction and background

1. The World Health Organisation (1980) has defined disability in the context of health experience as: 'any restriction or lack (resulting from an impairment) of ability to perform an action in the manner or within the range considered normal for a human being'. Because it is dependent on respondent perceptions, disability is a difficult concept to measure even through special in-depth surveys involving personal interviews and a large number of questions.

2. For planning, funding and delivery of community services, small geographic area estimates of people with a disability are required. There is also a demand for disability data for small population groups, such as Indigenous people.

3. Since 1981, the Australian Bureau of Statistics (ABS) has conducted quinquennial surveys on disability. These provide widely-used and detailed information on the extent of disability in Australia, its effect on daily living and participation in the usual activities of the community, and the need for, and receipt of, assistance. However, survey information cannot be produced for small geographic areas or population groups. (Small area predictors have been modelled using survey, census and other data. However, these do not respond to local factors that affect the number of people with a disability in an area.).

4. Consequently, there is a high level of demand for the 2001 Census to collect information on the prevalence of disability in the Australian population, to be used in conjunction with the more detailed survey data. The topic of disability attracted the largest number of submissions from users (31) in the last round of consultations about the content of the 2001 Census, with strong support for disability status to be collected in subsequent censuses.

5. The criteria for including a topic in the census are:

  • the topic is of major national importance
  • there is a need for data on the topic for small groups of the population or for small geographic areas
  • the topic is suitable for inclusion in the census.

6. It is broadly agreed within the ABS that the first two criteria are met. However, there have been concerns about the third criterion because of the complex nature of defining and identifying disability, the limited space available on the census form and the method of form completion (self-enumeration by any household member). This report discusses the approach and results of the census testing.

Testing

7. Testing of a range of disability questions was undertaken in the lead up to the 1996 Census. None was found to provide results sufficiently comparable to those obtained using surveys.

8. A different approach to disability questions has been undertaken for the 2001 Census. Following discussions with the Census Consultative Group on Disability, two versions of questions were suggested, and tested in two rounds of focus group discussions in Sydney. Focus group participants included younger and older age groups, people with intellectual and psychiatric disability and their carers, and Indigenous people. Field testing of census questions (based on those recommended by the focus groups) was undertaken in Victoria during November 1997 and in South Australia during September 1998.

9. The approach taken in both tests was to compare responses to the disability questions in the test census with the responses to a follow-up survey based closely on the questions used to establish disability in the ABS Survey of Disability, Ageing and Carers. In the census test, respondents were asked whether they experienced difficulties with a range of functions (e.g. hearing, living independently) and what caused the difficulties (e.g. age, disability, difficulty with English language). The disability questions used in each of the tests are shown in Appendix 1.

10. Three approaches were trialled in the two census tests:
  • the November 1997 test used one form with a yes/no response for the difficulties question
  • the September 1998 test was split between two different forms, a yes/no response similar to the November 1997 test (form 4) and a scaled response (none/a little/a lot) to the difficulties question (form 5). Each form was administered to approximately 50% of the sample.

11. The test areas had different profiles. The November 1997 test area had a higher concentration of disability pension recipients and hence a younger disability population. The September 1998 test area had a higher proportion of older people compared with the November 1997 test area and the general population. These factors contribute to the different rates of disability observed in the follow-up surveys.

Results

12. In assessing test outcomes a number of comparisons and indicators are relevant:
  • The closeness of the census and survey disability rates - it is rarely possible to achieve the same precision in a census as a survey. This is the case for all items in the census, not just disability. The census is a blunt instrument because of the need to use simpler questions appropriate to self-enumeration. To the extent to which differences are expected, the stability of the relationship between census and survey results is then an issue.
  • The proportion of false positives, i.e. those with disability in the census but not the survey. These should ideally be as low as possible.
  • The proportion of false negatives, i.e. those with disability in the survey but not the census. Experience in the development of disability surveys indicates that the more detailed and probing the questions, the greater the proportion of people with disabilities identified. Census questions would therefore be expected to produce a lower rate of disability than a survey. A greater proportion of false negatives than false positives would then be acceptable.

13. The November 1997 test produced a census disability rate of 14.5 % compared to the survey rate of 23.1%. The disparity between these figures and the associated high false negative rate (48%) indicated that the census had only captured a part of the disability population. A second test was conducted in September 1998 to see if the questions could be improved. More specifically, its purpose was:
  • to test whether a different type of response to the census questions on the difficulties experienced with a range of functions improved the match with the total survey disability
  • to evaluate whether disability identified by the census questions had an acceptably close and stable relationship to an identifiable (more severely affected) subset of the survey disability population.

14. The results of the three tests are summarised in the table below:


Table 1: November 1997 and September 1998 Test Results
November 1997 Test

September 1998 Test


(yes/no response)


Form type 4
(yes/no response)

Form type 5
(scaled response)

Census disability rate %
14.5
15.0
22.4
Survey disability rate %
23.1
19.9
21.7
False positives -
census disability, survey no disability
(% of census disability population)


17.4


30.9


36.0
False negatives -
survey disability, census no disability
(% of survey disability population)


48.0


47.9


34.1
Ratio of census to survey disability (%)
63.0
75.4
102.9


15. The September 1998 test has narrowed the gap between the census and survey rates of disability at the expense of higher false positive rates. The fact that the 1998 test (Form type 4) questions, which are only marginally different from the November 1997 test questions, lead to such different outcomes, underscores the difficulties inherent in measuring disability.

16. One of the aims of the September 1998 test was to establish whether disability identified by the census questions had a relationship with an identifiable subset of the follow-up survey disability population. The scaled response questions in Form type 5 were included with this intent. The definitions of disability for the census and the survey were tightened (i.e. made more restrictive) to derive more severely affected subsets of the population with disabilities, and these populations were matched against each other (see Appendix 2).

17. Results of this analysis for Form type 5 (scaled response) are shown in Table 2. No improvements were achieved in matching the survey and census disability populations by restricting the disability definitions. In particular:
  • the differential between census and survey disability rates is greater for the more restricted definition than the broad one
  • the false positive rates remain high (30% and over) and the false negative rates increase with restrictions in the census definition, (to as high as 60%).

Table 2: Results of Restricting Census and Survey Definitions - Form type 5 (scaled response)(a)

Census disability definition -

1 (broadest)
2
3
4
5
Survey definitions -

Any restriction

Moderate restriction

Severe restriction

Census disability rate %
24.8
22.4
18.8
7.9
7.9
Survey disability rate %
21.7
21.7
21.7
13.8
11.5
False positives -
census disability, survey no disability
(% of census disability population)
39.2
36.0
31.4
30.4
38.3
False negatives -
survey disability, census no disability
(% of survey disability population)
30.8
34.1
40.7
60.1
57.4
Ratio of census to survey disability (%)
113.9
102.9
86.4
57.3
69.0

(a) This table presents results of selected combinations of four different census disability definitions and three different survey disability definitions. See Appendix 2 for details.


International experience

18. Other countries also have a demand for disability information from their national censuses. They have experienced problems in achieving high quality data in a census, except for questions which were extremely limited in scope (e.g. missing limbs). The experience in other industrialised countries includes:
  • Canada (since 1986) and New Zealand (1996) have included disability questions to increase the efficiency of their disability sample surveys. Disability rates from their censuses have not been published. Canada will include a limited set of questions in their 2001 Census to serve as a screen for sample selection, but also to provide general information on the population reporting some form of disability.
  • The United Kingdom tested a disability question for their 1991 census but did not include the topic because of a high degree of non-response and poor data quality. They elected to include a question on long-term illness in place of disability, and published results from this question.
  • The most recent United States Census (1990) included some disability questions. Output identifies the proportion of people with work, mobility or self-care limitations.

19. The experience of the above countries was that there was a high rate of people who appeared to have a disability (based on detailed testing), but were not classified as having a disability from the census questions. The reverse problem was also identified to a lesser extent (people who did not appear to have a disability but were classified as having a disability from the census questions).

Table 3: Disability Tests in Other Countries

Australia (Nov 97)

Australia (Sept 98)

Canada 1986

NZ 1995

UK 1991

Form
type 4

Form
type 5

Census disability rate %
14.5
15.0
22.4
9.2
19.7
6.8
Survey disability rate %
23.1
19.9
21.7
12.0
35.6
12.6
False positives -
census disability, survey no disability
(% of census disability population)


17.4


30.9


36.0


21.6


9.4


22.6
False negatives -
survey disability, census no disability
(% of survey disability population)


48.0


47.9

34.1


40.2


49.9


58.2
Ratio of census to survey disability (%)
63.0
75.4
102.9
76.3
55.3
54.1



Conclusion

20. While the results from the November 1997 census test raised some concerns about data quality of the disability item, they were broadly consistent with similar tests overseas.


APPENDIX 1: Census Test Questions

November 1997 Test

For developing health policies and community-based programs, there is a need to measure the extent of disability existing in Australia
14. For each of the following, tick YES or NO
Does the person...

  • have difficulty in doing everyday activities such as eating, showering or dressing?
o Yeso No
  • have difficulty talking to or hearing other people?
o Yeso No
  • have difficulty learning or remembering things?
o Yeso No
  • have difficulty reading or understanding things?
o Yeso No
  • have difficulty walking, kneeling or climbing stairs?
o Yeso No
  • Have difficulty living in independent housing without help from other people?
o Yeso No
  • Have difficulty doing any other things people of the same age usually do?
o Yeso No

15. What causes the difficulty shown in Q14 for the person?o Short-term health condition (lasting less than six months)
o Long-term health condition
o Disability
o Age
o Difficulty with English language
o Other cause - please specify
oooooooo
oooooooo
oooooooo
o No difficulty


September 1998 Test

Form type 4: Yes/no response
For developing health policies and community-based programs, there is a need to measure the extent of disability existing in Australia
18. Mark YES or NO for each of the following:
Does the person have difficulty...

  • doing everyday activities such as eating, showering or dressing?
o Yeso No
  • hearing?
o Yeso No
  • learning, understanding or remembering things?
o Yeso No
  • reading or seeing even with glasses?
o Yeso No
  • walking, kneeling or climbing stairs?
o Yeso No
  • living independently?
o Yeso No
  • doing any other things people of the same age usually do (for example working, studying, etc.)?
o Yeso No

19. What causes the difficulty shown in Q18 for the person?o Short-term health condition (lasting less than six months)
o Long-term health condition
o Disability
o Age
o Difficulty with English language
o Other cause - please specify
oooooooo
oooooooo
oooooooo
o No difficulty


September 1998 Test

Form type 5: Scaled response


For developing health policies and community-based programs, there is a need to measure the extent of disability existing in Australia
18. How much difficulty does the person have in:

None

A little
A lot

  • doing everyday activities such as eating, showering or dressing?
o o o
  • hearing?
o o o
  • learning, understanding or remembering things?
o oo
  • reading or seeing even with glasses?
o o o
  • walking, kneeling or climbing stairs?
o oo
  • living independently?
o o o
  • doing any other things people of the same age usually do (for example working, studying, etc.)?
o oo

19. What causes the difficulty shown in Q18 for the person?o Short-term health condition (lasting less than six months)
o Long-term health condition
o Disability
o Age
o Difficulty with English language
o Other cause - please specify
oooooooo
oooooooo
oooooooo
o No difficulty



APPENDIX 2: Census and Survey Disability Definitions

Census definition 1 (broadest)

Disability is identified if any of the following apply:
  • response to Q19 is 'disability' or 'long-term condition'
  • positive (yes/a little/a lot) response to Q18, Q19 is 'age', and age >60
  • positive response to Q18 'hearing', conditional on age (>40) and no difficulty with English language
  • positive response to Q18 'everyday activities', ' learning or understanding', or ' doing things people of the same age do', and age >9
  • positive response to Q18 'reading or seeing', or 'walking, kneeling, etc.', Q19 response is 'age', and age >40
  • 3 or more positive responses to Q18, and Q19 response is 'short-term condition'.


Census definition 2

Disability is identified if any of the following apply:
  • response to Q19 is 'disability' or 'long-term condition'
  • positive (yes/a little/a lot) response to Q18, Q19 is 'age', and age >60
  • positive response to Q18 'hearing', and no difficulty with English language
  • positive response to Q18 ' learning or understanding', and age >9.

Census definition 3

Disability is identified if any of the following apply:
  • response to Q19 is 'disability' or 'long-term condition'
  • positive (yes/a little/a lot) response to Q18, Q19 is 'age', and age >60.

Census definition 4 (most restricted)

As for 2, but only for Form type 5, and using only 'a lot' responses to Q18.

Survey definitions

Any restriction

A person has a disability if he/she has one of the following:
  • loss of sight (not corrected by glasses)
  • loss of hearing, with difficulty communicating or use of aids
  • loss of speech
  • chronic or recurring pain that restricts everyday activities
  • breathing difficulties that restrict everyday activities
  • blackouts, fits or loss of consciousness
  • difficulty learning or understanding
  • incomplete use of arms or fingers
  • difficulty gripping
  • incomplete use of feet or legs
  • a nervous or emotional condition that restricts everyday activities
  • restriction in physical activities or physical work
  • disfigurement or deformity
  • needing help or supervision because of a mental illness or condition
  • head injury, stroke or other brain damage, with long-term effects that restrict everyday activities
  • treatment for any other long-term condition, and still restricted in everyday activities
  • any other long-term condition that restricts everyday activities.

Moderate activity restriction

A person with a disability (as defined under any restriction above) who needed assistance, or had difficulty with:
  • self care - bathing or showering, dressing, eating, using the toilet and managing incontinence
  • mobility - moving around at home and away from home, getting into or out of a bed or chair, and using public transport
  • communication - understanding and being understood by others: strangers, family and friends; and/or
  • daily activities, such as health care, housework, home/garden maintenance, meal preparation, managing money/correspondence, and transport; and/or
  • were restricted in schooling or employment.

Severe activity restriction

A person with a disability (as defined under any restriction above) who needed assistance with:
  • self-care mobility, communication; and/or
  • a range of daily activities; and/or
  • were restricted in schooling or employment (as specified in moderate activity restriction above).


APPENDIX 3: Census Papers


2001 Census Papers

02/012001 Census: Digital Geography Technical Information Paper
02/022001 Census: Report on Testing of Disability Questions for Inclusion in the 2001 Census
02/032001 Form Design Testing

1996 Census Working Papers

96/2 1996 Census Form Design Testing Program
96/31996 Census of Population and Housing: Digital Geography Technical Information Paper
97/1 1996 Census: Homeless Enumeration Strategy
99/11996 Census: Industry Data Comparison
99/21996 Census: Labour Force Status
99/3 1996 Census Data Quality: Housing
99/41996 Census: Review of Enumeration of Indigenous Peoples in the 1996 Census
99/61996 Census Data Quality: Occupation
00/11996 Census Data Quality: Journey to Work
00/21996 Census Data Quality: Qualification Level and Field of Study
00/31996 Census Data Quality: Industry
00/41996 Census Data Quality: Income


For further information about these papers please contact Rosa Gibbs by telephone: (02) 6252 5942 or email: rosa.gibbs@abs.gov.au.