Content Appraisal as a Method for Measuring the Effectiveness & Usability of Online Content

Introduction

Content appraisal, a simple qualitative system, will provide Extension personnel and other website users with a method that is easy to use that ensures relevance and usefulness to clientele, increasing its effectiveness. Previous website analysis has focused mainly on the survey of users and the use of Web analytics software (Parish, 2011) or Neilsen's heuristics (Nielsen, 1994; Nielsen, 2005). Web analytics software, such as Google Analytics, provides detailed information regarding the use of a the website, including page views, average time users spend on pages, and bounce rate. Nielsen's heuristics are the 10 general principles for the design of the user interface, which focus on usability of a website. In contrast, the content appraisal system involves the thorough evaluation of content and results in a report that focuses on content strategy and identification of weaknesses and provides recommendations for improvement.

HorseQuest, an eXtension Community of Practice (CoP), used this method to evaluate content on their site. The criteria examined included knowledge level, interrelatedness, relevance, usability, actionability, and differentiation. HorseQuest was the first CoP to create a content appraisal report based on these criteria. The purpose of analyzing the content on eXtension.org was to review the existing site for overall effectiveness and adherence to HorseQuest's goals. In the strategic content foundation for this project, these three organizational goals were identified:

• Empower people to make smart decisions that improve the quality of their lives.
• Provide credible, reliable, research-based information, tools, and solutions people can use.
• Bestow the knowledge from the best minds at land grant universities to the community-at-large.

In addition, we've identified the following key guiding principles for content creation:

• Content developed by HorseQuest becomes part of that community.
• Content creators will be credited for their contributions, which may affect promotions and tenure.
• Content contributors should believe in collaborating on content—including letting others in the HorseQuest CoP make decisions about content they've created.
• Duplication of existing content should be avoided.
• Content contributors should strive to understand who their content is for—local audience vs. national audience; novice reader vs. expert reader.
• CoP content will need to change and evolve to meet the needs of the HorseQuest CoP audiences. (adapted from Brain Traffic's eXtension Qualitative Report, 2010)

Materials and Methods

The content appraisal report was organized by sections. Section one included the goals of the organization, a "what we did" section which described the scope, content selection, process of content appraisal, and appraisal attributes (criteria). Appraisal attributes included:

• Knowledge level—the level of specialized knowledge needed to understand website content.
• Interrelatedness—how well pieces of content link to other content on the website.
• Relevance—how current and of interest the contest is to the intended audience.
• Usability—the readability and effectiveness of the presentation of the content.
• Actionability—the explanation of a next step to be taken by the user.
• Differentiation—how the content fills a unique need or offers a unique perspective.

Section two focused on HorseQuest's content performance, including the format of the content (including answers from our experts, articles, audio, learning lesson, video, news, or other such as glossary terms); appraisal attributes, which included knowledge level, interrelatedness, relevance, usability, actionability, and differentiation; and other observations including overall key findings. Section three discussed the number of pages appraised, areas for improvement and strengths of the site.

Content Appraisal Results

Content appraisal for this report covered 213 webpages. The focus of these results is Section 2, related to content performance, which other Extension professionals will likely find to be most relevant. Percentages may not add up to 100 because of rounding to the nearest whole number.

Format of the material on the website was the first thing appraised (Table 1).

Table 1.
Format of the Material on the Website

Value	Meaning	TOTAL (of pages appraised)
Answer	Content in Answers from our Experts	0.94%
Article	Content that is solely or primarily text	37.09%
Audio	Content that is solely or primarily sound files	0%
Learning Lesson	Online lessons or tools	2.35%
News	Content in In the News	0.94%
Video	Content that is solely or primarily video	2.81%
Other	Glossary terms, other	55.87%

Subject matter knowledge level needed to understand or use the page was appraised (Table 2).

Table 2.
Subject Matter Knowledge Level Needed

Value	Meaning	TOTAL (of pages appraised)
1-LOW	No specialized knowledge is required or assumed.	45.07%
2-MEDIUM	Some knowledge is required.	48.83%
3-HIGH	A high level of subject matter knowledge is required.	6.10%

Interrelatedness (how the content fitted and linked to other content in the CoP) was appraised (Table 3).

Table 3.
Interrelatedness of Content

Value	Meaning	TOTAL (of pages appraised)
1-LOW	Lack of working links to other content	29.11%
	Tone is inconsistent with other content in a way that’s strongly distracting.
	Content is off-topic
2-MEDIUM	Links exist but some are broken	25.82%
	Tone is moderately inconsistent with other CoP content
	Tags are mostly absent (e.g., only one tag and it’s non-specific)
3-HIGH	Good links, tone, tags	45.07%

Relevance (timeliness and interesting to the intended audience) was appraised (Table 4).

Table 4.
Relevance of Content

Value	Meaning	TOTAL (of pages appraised)
1-LOW	Outdated, no longer relevant	1.88%
2-MEDIUM	Appears dated/stale or is of interest only to a small geographic area	5.63%
3-HIGH	Timely or "evergreen" content	92.49%

Usability, which focused on the content having clear, descriptive headings and subheadings, appropriate sentence and paragraph length, use of graphics, bullets, tables, charts, etc., where appropriate, and overall readability, was appraised (Table 5).

Table 5.
Usability of Content

Value	Meaning	TOTAL (of pages appraised)
1-LOW	Numerous minor problems, a few moderate problems, or one major problem that seriously interferes with the content.	6.10%
2-MEDIUM	Minor to moderate problem(s).	29.58%
3-HIGH	No significant problems.	64.32%

Actionability, which focused on whether there was a clear next step for the current audience of the content and a means of facilitation of that step, was appraised (Table 6).

Table 6.
Actionability of Content

Value	Meaning	TOTAL (of pages appraised)
1-LOW	No next steps given.	40.85%
2-MEDIUM	Next step is implied or vague; next step depends on a link that is broken; indirect or absent call to action (CTA).	15.02%
3-HIGH	Next step is clear, specific, and stated explicitly; effective CTA.	44.13%

Differentiation, which focused on whether the page filled a unique need, provided information or a perspective that is not readily available elsewhere, was appraised (Table 7).

Table 7.
Differentiation of Content

Value	Meaning	TOTAL (of pages appraised)
1-LOW	Topic is better covered elsewhere, and page doesn't contribute to a strong body of content for HorseQuest	29.10%
2-MEDIUM	Information is generally known or topic is covered elsewhere, but page provides some modest benefit over other sources, or contribute to providing comprehensive information in HorseQuest	53.52%
3-HIGH	Page offers a unique perspective, information not widely known, or better explanations than other sources.	17.38%

Conclusions

The content appraisal process helped the HorseQuest CoP analyze its existing online material and make improvements. It has also helped ensure that the website adheres to the CoP's organizational goals and guiding principles for content creation. Any content that received a low or medium rating is being examined and strengthened. The appraisal attributes used here are just a guideline; website creators can create their own set of standards. A solid content appraisal process can help address problem areas of a website as well as build upon strengths of the website to make content more usable to the audience.

References

Brain Traffic. (2010). eXtension Qualitative Report.

Nielsen, J. (2005). Ten Usability Heuristics. Retrieved from: http://www.useit.com/papers/heuristic/heuristic_list.html 

Nielsen, J. (1994). Enhancing the explanatory power of usability heuristics. Paper presented at the Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: Celebrating Interdependence, 152-158.

Parish J. A. (2011). Website usage information for evaluating beef cattle Extension programming. Journal of Extension [On-line], 49(5) Article 5TOT9. Available at https://www.joe.org/joe/2011october/tt9.php