3. Limitations of this and the Gravitas report
- 3.1 Timeliness
- 3.2 Sample Size and Associated Limitations
- 3.3 Subdividing the population
- 3.4 Correlation
Page 13
It must be stated at the outset that the work of Gravitas commissioned by the RNZFB is not in question. We have trusted the work of Gravitas in finding the survey weightings used for the Gravitas' analyses and carried them over into our work making comparisons between the two reports valid.
Survey weightings are used to reconcile differences between the group of members sampled and the RNZFB membership. Each person sampled by Gravitas represents a different number of individuals on the RNZFB database at a certain point in time and depends on the number of members that fall into certain demographic classifications. For example, the subgroup of Rural NZ European females over 65 years old who are vision impaired and not in employment has a total of 6 respondents in the survey (3% of all survey respondents and 8.2% of the RNZFB membership), each of whom carry a weighting of 153.887 (that is each of these respondents represent about 154 members in the total RNZFB membership). It should be noted that the membership may have changed somewhat since this time, but again, the findings of this report are kept on the same footing as the Gravitas report for comparability.
The findings of both reports should be used is rough estimates, rather than definitive numbers. We advice use of such terms as "the costs of blindness research indicates..." as by their own admission, the Gravitas report is not entirely comprehensive to all possible costs of blindness. The principal problem is that the majority of estimated costs in the Gravitas report and some in this report are based on such small numbers of respondents that they cannot be relied upon for their accuracy. Gravitas have provided the authors with the data that they collected on the
Page 14
RNZFB's behalf, and minor differences will exist in the way which data were handled and cleaned.
3.1 Timeliness
It has been some time since the data was collected for the costs of blindness research project. Estimated costs could be adjusted for inflation where appropriate, but gauging the appropriateness of the adjustment is difficult. Under normal circumstances, an increase in the cost of a product or service compared to income causes a reduction in the consumption of that service. The method used to make any adjustment may, therefore, introduce greater doubt about the validity and usefulness of the estimates gained.
3.2 Sample Size and Associated Limitations
The small sample size of approximately 200 RNZFB members makes it difficult to distinguish any differences that different demographic groups of people incur to meet their costs of blindness. If the model in Section 2 is to be used, each component on the right hand side of the equation is limited by the small sample size.
There are many different subgroups of RNZFB members, and any two people would not necessarily subdivide the population in the same way. Six distinct ways of categorizing the population were used, but there were in fact fewer people in the sample than there are subgroup combinations when all six of the categories are considered together. Of these 216 possible combinations of demographic groups, only 78 of these combinations were represented in the sample. Obviously, some judicious merging of the individuals sampled is required if we want to see if different
Page 15
subgroups of blind and vision impaired New Zealanders have different costs of blindness. Additionally, subgroups used in this analysis are much broader than some would like. It is clearly impossible to determine the existence of any differences between different subgroups of blind and vision impaired New Zealanders using this data set. When estimates of the blind and vision impaired population are made using this data set, certain small categories are given a zero estimate for the number in the population because they were not directly represented in the sample. These people clearly exist and need to be grouped in with another demographic group that is considered similar enough to warrant the merger. The groupings presented in Exhibit A8 are extremely subjective so various options are provided in Appendix A which may prove useful. These options for subgrouping have been used in other analyses, but as will be seen, little difference in the costs incurred were found.
Appendix A provides tables that show the make-up of the sample. Notable differences between the sample and the population include:
1. Members over the age of 65 were under-represented in the sample.
2. Members under the age of 18 were under-represented in the sample. In fact, only 14 individuals were sampled and as a consequence, would normally be grouped together which loses the opportunity to consider the different costs of blindness for younger members based on other factors such as ethnicity or location. In hindsight, it may have proved more useful to ignore sampling this group if such a small number was going to be surveyed, and instead state the research was for the costs of blindness relating to adults.
3. Approximately 19% of all survey respondents were in employment. This equates to approximately 11% of the RNZFB membership. These respondents are predominantly between 18-65 years old, live in metropolitan locations or are of European ethnicity. Note that these age, location, and ethnicity generalizations are not able to be
Page 16
grouped together so it is not correct to say that the majority of the population that are employed are 18 to 65, of European descent and live in metropolitan areas. This joint outcome cannot be determined.
4. With respect to those people in the age range 18 to 65 that are in employment: Although there are as many blind people as vision impaired people in the sample, the number of vision impaired in the population is actually over twice the number of blind in the population.
5. For the over 65 year old group, (approximately 36% of survey respondents and an estimated 68% of RNZFB membership), there are almost twice as many female respondents than male. Nearly all these respondents are not in employment and of European ethnicity.
6. We estimate that at least 60% of RNZFB members have an annual household income of less than $30,000.
7. We estimate that approximately 13% of the RNZFB membership have children living with them (with or without a partner) and presumably are responsible in some way for financially supporting them.
8. We estimate that at least 47% of the RNZFB membership who live with children have an annual household income between $30,000 to $59,999. (Recall that this is a larger annual household income than most of the RNZFB membership, when not considering living situations).
9. We estimate that at least 64% of the RNZFB membership who live alone have an annual income between $10,000 and $29,999.
10. We estimate that approximately 41% of the RNZFB membership live without a partner (alone & children only), while about 44% of the RNZFB membership live with a partner (with and without children).
Page 17
11. We estimate that at least 50% of the RNZFB members who live with a partner (with or without children) have an annual household income between $10,000 and $29,999.
The differences between sample and population need to be considered when making assertions about blind and vision impaired New Zealanders on the whole. When percentages were considered, they have been adjusted for the over-representation of some groups within the sample, and therefore the under-representation of other groups. Greater detail of estimated numbers of RNZFB members are provided in Appendix A and have also been found by considering the representativeness of the sample where these estimates have been given.
Whenever these estimates differ from popular belief, there are two possible reasons. Either popular belief is in error, or the sampling has resulted in misleading conclusions. Each phenomenon needs to be considered independently and if necessary investigated further by interrogation of the current RNZFB member database.
These simple examples of the demographic make-up of the RNZFB membership based on the current data need to be compared to the rest of the New Zealand population. For example, total household income for the blind and vision impaired population need to be compared to that of their sighted peers.
A second concern of much greater importance is the fact that the small sample size has resulted in a very small number of respondents for many questions in the survey. Many costs are experienced by only a small portion of those people surveyed, but there are a very wide range of costs that these people incur. The authors determined that no analysis would be possible on the distribution of the various costs of blindness unless at
Page 18
least twenty responses were available. This also means that many costs of blindness cannot be separated for subgroups of the blind and vision impaired population as some or all of these subgroups had fewer than twenty respondents. The costs that can be described using quartiles are given in Appendix B.
The model we propose is based on the average amount spent and on the proportion of people experiencing particular costs. When estimating a proportion, the total number of people in each group needs to be sufficiently large to ensure the desired level of accuracy is met. It would be highly impractible to achieve the same level of accuracy as political polls for example. To illustrate this point, many demographic groups within the RNZFB membership do not have as many people within them as there would need to be to get the accuracy desired. On the other hand, if the total number of people in any such group is small enough, we would be better off asking them all if they incur a cost.
We have determined that it is not possible to identify the proportion of people within a subgroup that incur a particular cost of blindness unless there are a sufficient number of people from that subgroup within the sample. The problem is that this number should be determined before the data is collected and that a desire to have something like 100 people per group is not an option that remains available. The total number of people in any subgroup must be greater than the minimum number of people that responded to questions that identified a particular cost. In this presentation we determined that this number should ideally be above fifty respondents. In general, the number of sample respondents does not meet this desired level, but somewhat fortuitously, there was sufficient data to be confident that we have found some differences in the propensity to incur taxi and daily living costs.
Page 19
3.3 Subdividing the population
Various ways of grouping the RMZFB membership were considered, and as shown in Appendix A, it proved difficult to find groupings that both made sense and had sufficient survey respondents to allow greater analysis. Using the survey weightings provided by Gravitas, the estimated number of RNZFB members for these groupings were estimated, and after comparison with the database will show any flaws with the sample. There is of course a chance that the make-up of the RNZFB membership has changed slightly, and weightings determined using the current membership database would differ from those used in this and the Gravitas report. Comments in the Gravitas report about the quality of the database are well made, and unless significant change has occurred since Gravitas collated the data, the indicative estimates given by Gravitas for the total value of costs of blindness remain useful. The Gravitas report makes no attempt to find the actual cost of blindness for individuals however, and it is for this reason that the authors prefer the model proposed in the previous section which would find different costs of blindness for different demographic groups.
3.4 Correlation
The Lansink-led validation group that met in January 2005 asked what analyses of any trade-offs between different costs of blindness was possible. The simplest way to look at this phenomenon is to use the correlation between pairs of costs. The threshold of twenty was used in this instance also, but this means that there needed to be twenty respondents that gave a cost for both costs of blindness. Thus, all costs of blindness that had fewer than twenty respondents were immediately discarded from consideration.
Page 20
The reliability of any correlation presented is somewhat low, even if there were twenty or more respondents. Comments need to be restricted to very general trends and further data collection is required to draw sound conclusions. For example, if only those people who actually incur an expense for personal taxi use and for daily living are considered, there is a positive relationship between the amounts they spend on the two cost categories. In other words, those respondents who have higher taxi costs also tend to have higher costs of daily living. See Figure 1 which plots daily living costs (the sum of costs for seven tasks involved in daily life)and personal taxi costs.
The problem with this analysis is that there are many people who experience personal taxi costs that do not incur daily living costs and vice versa. If we take these people into consideration, there is clear evidence of a trade-off. These extra observations cause the straight lines of points along the bottom and the left of the scatter plot in Figure 1. If these values are incorporated into the correlation analysis, the correlation becomes negative, showing that there is in fact a trade-off between these two cost categories.
Page 21
Figure 1. Scatter plot of daily living costs vs personal taxi costs.
Unfortunately, the notion that the smaller the level of an individual's disposable income, the more pronounced the trade-off between costs of blindness cannot be investigated with the current data set. It is also difficult to investigate the notion that the costs of blindness increase or decrease as a consequence of increased total income given the small sample size. Many respondents did not provide information about the level of the household income. As an aside, this is a notoriously difficult problem to solve, as this lack of response cannot be ignored easily. Attempts to show how the different levels of household income affect various costs were made difficult by the 16% of survey respondents that did not or could not state their household income.