Simple Interactive Statistical Analysis
Input.
Program presumes that the input consist of positive (integer) numbers without decimals. Categorize the sampled cases in the boxes provided, one can choose to use the CASRO boxes or the AAPOR/ISER boxes, or to use them in combination, as required. The program will sort the cases between AAPOR/ISER and CASRO. Optionally a fraction eligible respondents can be given. If you do not give this fraction the fraction will be estimated for you.
Explanation.
Often response rates in survey research are calculated simply by dividing the number of completed interviews by the number of individuals who were selected to participate in the research, i.e. the sample. However, this method is too simplistic and does not do justice to the complexity of research design, sampling process and the practical difficulties of contacting and assessing potential survey participants. Therefore the Council of American Research Organization (CASRO) proposed a method to better consider the various situations encountered in survey research. This method formed the basis for the development of a standard for the calculation of response rates by the American Association for Public Opinion Research (AAPOR). This standard was further refined by the Institute for Social and Economic Research (ISER). Both the publication by the AAPOR and the publication by ISER are available for free on the internet. The user is advised to download these publications and study them before using this SISA procedure.
Both the AAPOR and the ISER system consist of a large number of detailed categories. Each sampled address, number or case should be coded according to one of these categories. The number of coded cases must sum to the total sample size (N). The detailed categories are summarized into 8 sub-categories, numbered in the SISA procedure on the basis of the ISER system as -1,2,3,4,5,6a,6b,7-. These 8 sub-categories can be summed in turn into the four main CASRO categories, numbered -A,B,C,D- in the SISA procedure. Table one can be used to convert the SISA, AAPOR and ISER coding, whereby it should be considered that the overlap between 6a into 3.2 and 6b into 3.1 is not perfect.
|
Name |
SISA |
AAPOR |
ISER |
|
Completed (>80%) |
1 |
1.1 |
1 |
|
Partial Completed (50-80%) |
2 |
1.2 |
2 |
|
Completed (>50%) |
A |
1.0 |
|
|
No-Contact |
3 |
2.1 |
3 |
|
Refusal/Break-off (<50%) |
4 |
2.2 |
4 |
|
Other (language problem etc) |
5 |
2.3 |
5 |
|
Refused eligible |
B |
2.0 |
|
|
Contacted/ Unknown Other |
6a |
3.2 |
641, 651, 661, part of 67 |
|
Not-Contacted/ Unknown if household/occupied HU |
6b |
3.1 |
61-63, 642, 652, 662, remainder of 67 |
|
Unknown |
C |
3.0 |
6 |
|
Ineligible |
7D |
4.0 |
7 |
The eight AAPOR/ISER categories will here be discussed. The user should note that in a research situation sampled cases are not coded according to these higher level codes, they will be coded according to the detailed category codes used in the AAPOR/ISER systems and the result will be summarized to the higher level code.
- 1) Completed. Rarely is an interview fully completed. According to the AAPOR system an interview is completed if the respondent was cooperative and at least 80% of the questions have been reliably and validly answered, although SISA would probably prefer 90%.
- 2) Partial. As above, at least between 50 and 80% of questions validly answered, SISA would prefer between 70 and 90%.
- A) Completed interviews according to CASRO, sum of the above. Category 1 in AAPOR.
- 3) No-contact. The eligibility and existence of the respondent could be confirmed but the respondent could not be contacted.
- 4) Refusal. The respondent could be contacted and his or her eligibility could be assessed. The respondent refused to cooperate in the survey. Break-off, the respondent initially collaborated but refused later in the interview or the respondent could not be recontacted after an interview was suspended. Less than 50% of the questions have been answered, SISA would prefer 70%.
- 5) Other. The respondent could be contacted and his or her eligibility could be assessed. However, the respondent was unable to cooperate in the interview because of an illness or language problems. This category also includes interviews that were lost or invalidated.
- B) Refusals according to CASRO, sum of the above. Category 2 in AAPOR.
- 6a) The selected person, address or number could be contacted but his or her eligibility could not be determined. Mostly a person is spoken to but he or she refuses or is unable to provide adequate information. Written information is not readable in a postal survey.
- 6b) The selected person, address or number was not contacted and his or her eligibility could not be determined. Might be (over-) sampled cases that were not contacted, phone rings but is never answered, reply envelope is never returned.
- C) Unknown according to CASRO, sum of the above. Category 3 in AAPOR.
- 7D) The selected person, address or number could be contacted and his or her ineligibility could be determined. Or, the selected person, address or number could not be contacted but his or her ineligibility could be determined anyhow. For example, by talking to the neighbors or family members (although there is always a reliability issue here). Ineligible mostly concerns cases which are not in the sample frame, for example, if a research project is about females males are ineligible, if it is about black people white people are ineligible, if it is about residential phone numbers business numbers are ineligible. Category 4 in AAPOR, 7 in ISER, D in CASRO.
- N) total number of sampled cases. You can give this number but you don't have to. This number, however, is important. Good book keeping should ensure that every sampled case is at the end of a research project coded and accounted for.
- e) Fraction/proportion of eligible cases. You can give this proportion if you have a very good reason for that; otherwise the program will calculate this proportion for you which is mostly the much-preferred way of doing things. Only one e is used, according to AAPOR, and not two, as in the ISER system.
Output.
Confidence Intervals.
The only thing that is fixed in a response rate calculation is the N, the number of sampled cases. All other variables are empirical variates that will fluctuate between measurements. Therefore, the researcher can ask him or herself the question: "if I would repeat this design in a similar situation, in between what values can I expect my estimates to be?" The procedure gives confidence intervals for the four main CASRO response categories that make up the four main proportions of all sampled cases. For some other measures confidence intervals are also presented.
CASRO response rates.
Response rates according to the CASRO procedure. The third response rate, the actual CASRO response rate, is the most important. In calculating the CASRO response rate it is assumed that the proportion of eligible cases in the unknown cases is equivalent to the proportion of eligible cases in the sum of cases in the sample of which the eligibility or ineligibility could be determined. This proportion is given in the output as the e for CASRO.
Response rates.
The response rates are the proportion of completed interviews in the total number of eligible respondents. The various versions of the response rate take the respondents of whom eligibility could not be determined as all eligible, as all not eligible, or a fraction "e" of the unknown respondents is considered to be eligible. Another question is whether to consider partial interviews as completed.
Cooperation rates.
The co-operation rate is the number of completed interviews in the number of contacted eligible respondents. In the case of the ISER cooperation rate an estimate is considered of the number of contacted eligible persons in the unknown category, in the case of the AAPOR cooperation rate unknowns are not considered.
Contact rates.
The contact rate measures the number of eligible persons which were contacted. ISER considers the number of contacted possibly eligible unknowns; AAPOR does not consider unknowns.
Refusal rates.
The refusal rate gives the proportion of eligible respondents who refused to give an interview. This is the least important rate.
Comment.
Instead of one standard we now have three. Note that the CASRO standard continuous to be at the basis of things and the CASRO response rate should continue to have the focus in any judgement of a research project. The more detailed AAPOR and ISER methods will make it easier to calculate the CASRO rate in a standard way. Both AAPOR and ISER present other measures as well, the co-operation rate, the refusal rate, and the contact rate. Although these rates can provide valuable information they are and continue to be considerably less important than the CASRO response rate! Note that for these additional rates you have to fill in all the eight AAPOR/ISER boxes, for the CASRO response rate you have to fill in the four CASRO boxes only. SISA does not have a preference for either the AAPOR or the ISER system. ISER seems to be more internationally applicable. AAPOR is closer to the original 4-category CASRO system. The major important difference is in the definition of the unknowns, were the split in contacted and not contacted respondents of whom eligibility could not be determined is used in calculating the ISER co-operation and contact rate, but not in the AAPOR system. For the AAPOR system it would suffice to only fill in the total number of unknown cases in the C box, this would not be appropriate in the ISER system. However, the overlap between both systems is considerable and to use both systems side by side, or even in combination, probably does not cause too many discrepancies.
References.
The American Association for Public Opinion Research. Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys. Lenexa, Kansas: The American Association for Public Opinion Research, 2000.
Lynn P, Beerten R, Laiho J, Martin J. Recommended Standard Final Outcome Categories and Standard Definitions of Response Rate for Social Surveys. Colchester, Essex: The Institute for Social and Economic Research, 2001.
