Getting the Most Out of Multiple Response Questions

Introduction

Communication between the researcher and the computer programmer is key to producing sensible data analyses that answer the needs of the former. Information such as research objectives, envisioned use of demographic data (if any), type of statistical analyses planned, layout of tables anticipated to be included in the report, and handling of missing data, greatly facilitates the tasks that the latter has to do to provide a fast and effective service.

One of the areas that is often overlooked during consultation on data analyses is the manner in which the researcher wants to handle the multiple response question (MRQ). The popularity of MRQ comes from its ability to capture as many of the possible pre-formatted answers to a question as possible. While its use may be convenient for the survey planners, results of data analysis usually vary and are oftentimes confusing to whoever will summarize and interpret the data. This article describes and discusses three approaches to analyzing MRQs and their implications for getting the most out of MRQ data analysis.

Methodology

The following questionnaire (Table 1) was derived from a survey that sought to understand the existing communications linkages among and between researchers, extension workers, and farmers. Although there was a total of 47 questions in the survey, I will use just one to illustrate the objective of this article. The statistical procedures from SAS^(R) were used in all data analysis.

Table 1.

Sample 6-Item Multiple Response Questionnaire

From a researcher's perspective, the above is one question with six possible answers. All he or she wants to have is the relative frequency counts of each item as a basis for deducing its importance as a confirmatory source for verifying the need for new practices. From the programmer's perspective, it is actually six questions, each requiring a dichotomous (i.e., yes/no) answer and subsequent dummy coding (i.e., "yes" numerically coded as 1, "no" is 0).

Depending upon the objective of the researcher, the programmer could handle the data analysis in one of two ways. First, if the objective of the research is to rank the six items individually in the order of importance (frequency), then a program can be developed to analyze the responses such that when an item is checked, a program module records the count and credits it to the same item. The procedure posts the frequency count for each of the six items separately (Table 2) without regard to combination responses.

An obvious advantage to this approach is its simplicity, which makes for easy discussion and interpretation of results. One objection associated with this procedure is that the contributions of the combination items in measuring a particular dimension are always confounded with the single-item responses and therefore are often ignored. In addition, this simplistic approach may be applicable for questions with limited number of items but may be impractical and programmatically tedious when one is dealing with many possible items. To perform this analysis, the programmer has to create as many variables as the number of possible choices, a task that could be overwhelming for MRQs with large arrays of items.

Table 2.

Frequency Analysis for a 6-Item Multiple Response Questionnaire Generated by Single-Item Analysis Module

* Each capital letter stands for the item with the same letter contained in the original questionnaire.

Second, if the objective of the research is to keep track of the counts of both the individual and combination items, then an entirely different procedure will have to be developed such that the checks for individual and combination items are completely accounted for as shown in Table 3. With numerous single and combinations items to deal with, the researcher would surely be confronted with a difficult task of explaining and interpreting the context of this analysis. Consequently, most of the non-significant variables would have to be dropped while only those with relatively high frequencies will be retained for further consideration. Nevertheless, this method has the advantage of separately accounting for the influence of the single and combination items characteristically present in real life systems.

Table 3.

Frequency Analysis for a 6-Item Multiple Response Questionnaire Generated by Multiple-Item Analysis Module

*Each capital letter stands for the item with the same letter contained in the original questionnaire; multiple letters indicate combination responses.

Putting the Objective In Sight

From the programmer's point of view, both techniques discussed earlier are valid statistical approaches. For the researcher, however, a useful guideline is to regard the procedure that addresses his/her research objective well as the most appropriate. By sticking to the guideline, it is very easy to establish and decide on the method of analysis to apply. However, it should be emphasized that the decision of which statistical design and analysis to use should be made prior to the conduct of surveys, as has always been recommended in most experimental methods.

Questionnaire Design Is Important

Earlier, I have demonstrated the use of single item analysis that apparently failed to capture some information (i.e., effect of combination responses), and that of combination-item analysis that generated extraneous information that the researcher may have no use at all. There is a middle ground to the aforementioned approaches but one that entails a change in questionnaire design at the early stage of survey development. Clearly, the anticipated discussion and presentation formats in a planned report should determine the choice of specific item-combinations to extract for use. To demonstrate how could this be done, the original questionnaire was modified to ask respondents to check the three most important items (out of six) instead of asking them to check all the items that apply. The modification made to the following questionnaire (Table 4) may appear trivial at first but its effect, as far as data analysis is concerned is significant.

Table 4.

A Multiple Response Questionnaire That Was Modified to Solicit Three-Item Combination Responses

By just looking at the instruction (i.e., check the three most important), an experienced programmer could easily determine how to proceed with analyzing the responses. The following printout (Table 5) was generated using a simple frequency analysis with little modification in the program. The result came out "lean" (no extraneous information), in 3-item combinations as desired, and in a format in which the researcher can plug in as it is as a supporting table for discussing the research dimension being measured. Unlike the two approaches discussed earlier, this method extracts just exactly the information that the researcher needs, making the procedure efficient and straightforward. Consequently, the result generated by this method make for an easy setup and write-up of reports for the survey dimension under study.

Table 5.

Frequency Analysis for a Multiple Response Questionnaire Designed to Solicit 3-Item Combination Responses

Conclusion

MRQ will always be a useful feature of Extension surveys. This type of question is unique in that it requires a different mode of data entry utilizing dichotomous variables and dummy coding. When properly handled, MRQ variables have the ability to relate with other numeric variables through regression and correlation, and may even be used as predictor variables in multiple regression analysis. However, before such versatility can be harnessed, the researcher has to do his or her part in making sure that the analytic approach to MRQ to be used by the programmer addresses the research objective(s) that he or she sought to achieve at the outset of the survey.

The sample questionnaire used in this article was derived from a survey conducted by Dr. Brigitte Peters for her doctoral dissertation in the Department of Agricultural Education of Texas A&M University in 1998.

SAS is a registered trademark of the SAS Institute Inc. in the USA and other countries. ^(R) indicates USA registration.