Intensity Weighted Ranking: A Methodology for Understanding What Clients Tell Us

Introduction

"Using Technology to Survey New Audiences," an Ideas at Work article in this issue, discusses our utilization of a computerized touch screen for collecting data about perceived client needs. While quite useful, the rank ordering of issues leaves a major question unanswered: "How strongly do you feel about each issue?" Respondents can only rank a limited number of discrete choices, and ranking tells us nothing about the intensity of client views on each topic. Unlike interval level scaling, rank ordering does not imply how much more important subsequently ranked issues are for a respondent.

This article introduces a tool for uncovering respondent's intensity of response and more specific information about respondents' issues of most concern. The methodology requires respondents to rank order a short set of concerns as they influence their community.

Methods

We were interested in what water quality issues Indiana residents see as most important and how strongly they feel about these issues. This information will assist us in planning educational activities and programs. To assess the concerns and educational needs of a general audience, we used a touch screen computer at the Indiana State Fair in 2003. Participants were asked to rate the strength of their concern for specific water quality topics. Seven general issues were broken down into component "sub-issues." The issues and sub issues were:

• Wastewater Treatment--septic systems, municipal sewage treatment, storm water runoff

• Supply of Clean drinking water--quantity and quality issues

• Community Planning Resources--land use/watershed planning, decision aids (maps, software, etc.)

• Ag Production Issues--animal production issues, nutrient management, sedimentation

• Surface Water Quality--maximum loading to streams (TDMLs), nutrient criteria

• Health of Aquatic Ecosystems--ecological metrics, disease vectors (e.g., West Nile)

• Water Based Recreational Opportunities--fishing, swimming, boating

Participants were asked to rate the sub-issues importance for their community on a scale of "no importance" (0) to "highest importance" (10). A score of 5 was an average value. The survey included four demographic questions, a ranking of seven water quality issues, and an analysis of the strength of the respondent's feelings about each issue. The water quality issues were determined from an open-ended e-mail survey of Indiana Extension staff (county and state).

A total of 434 people completed the survey through at least part of the rating of the strength of their concerns. (Details of the demographic information and results of the water quality issue rankings are presented in "Using Technology to Survey New Audiences," in this issue.)

Results were analyzed by comparing the average value given for each sub-issue. The sub-issues that respondents felt most strongly about received a higher value. Although the average values for sub-topic give some idea of the strength of concern, people tend to see every sub-issue as important in and of itself. Therefore, we developed a method to combine the issue ranking and the strength of concern for each sub-issue, which we call the "Intensity Weighted Rank," calculated by multiplying the importance rating by the inverse of the rank order of the corresponding major issue response.

For example, if an issue was ranked 4th and was given an importance rating of 5, the Intensity Weighted Rank would be 1.25 (5*0.25). An issue ranked 1st with an importance of 10, on the other hand, would have a Weighted Rank of 10 (10*1.0). This provides a measure of overall importance of each issue. The issues that people felt most strongly about had a higher rating, with 10 being the highest value possible.

Results

Community drinking water was given the highest average value (9.6) with well water quality next (9.1). Table 1 and Figure 1 show the average values given for each sub-issue. Participants felt that all sub-issues were important, with average values ranging from a high of 9.6 to 7.0.

Figure 1.
Sub-Issue Averages

The Intensity Weighted Rank values give good comparison of the issues. Values ranged from a high of 9.6 for Community Drinking Water down to 1 for Boating. Table 2 and Figure 2 show the Weighted Ranking results. Comparing Figures 1 and 2 shows the value of the Intensity Weighted Ranking method. While the results appear similar in Figure 1 (because respondents were generally concerned about all water quality issues), Figure 2 shows the issues of highest concern. In addition, the Intensity Weighted Rank provides an indication of how much more important some issues are than others. For example, the issue of community drinking water had a IWR that was 16% higher then the IWR for the issue of declining aquifers, although both were in the top rank category of Clean Drinking Water.

Figure 2.
Intensity Weighted Ranking

Although 434 people completed at least part of the survey asking how strongly they felt about the sub-issues, many stopped before completing all 20 sub-issues. There was a drop of 8% (to 92%) of the people who indicated the strength of their concern for the first issue but stopped at that point. A drop of from 0-2% was seen with each subsequent question. Most people completed the entire survey, however, with 318 (73%) people answering all 20 sub-issue questions, indicating how strongly they felt about each sub-issue.

Discussion

Using a computer survey we were able to obtain information from a variety of people with a minimum of staff involvement. Questions included ranking water quality issues and indicating how important 20 sub-issues were. Answering 27 questions may have became an onerous task for the average fairgoer. A possible change would be to ask participants to rank and rate only the seven major issues. This method, however, would lose the specificity that listing all 20 sub-issues provided.

The Intensity Weighted Ranking combines the average intensity rating and the issue rank, giving more complete information. For example, "septics" was ranked 13th by the straight averaging method (7.9) but was ranked 7th using the Intensity Weighted Ranking system. This additional information indicates that educational programs about septic systems may be more important to citizens than first appears. Note that although the strength of response values for the sub issues varied between a narrow range (7 to 9.6), dividing by the ranking can make a big difference (values from 1 to 9.6). Note, however, that the decreasing relative difference in the smaller fractions (1/4, 1/5, vs. 1 or 1/2) makes the lower ranked issues appear closer in importance.

Conclusions

Analysis of average responses and Intensity Weighted Rankings in a needs assessment effort is more useful in evaluating programming needs than a simple rank ordering of issues.