A Librarian’s Approach to Reading a Wikipedia Article: The CRAAP Test

A. Introduction

As an academic librarian, I always caution patrons about properly using different types of sources (popular vs. scholarly, peer-reviewed, grey literature, and news sources). With regards to Wikipedia, I don’t discourage them from going there, as I use it all the time. Wikipedia, as a successful crowdsourcing project, has been surrounded by controversy due to its open editable model. In spite of the level of skepticism it has been receiving from scholarly communities for decades, Wikipedia still remains one of the popular choices for online information users. The perception that Wikipedia articles lack the quality control rigor of academia is now shifting, since now more users are relying on it for fact-checking and background information. Even academic institutions, such as libraries, are contributing to Wikipedia with content.

Wikipedia has shared some fundamental principles, called the Five Pillars:

Wikipedia is an encyclopedia
Wikipedia is written from a neutral point of view
Wikipedia is free content that anyone can use, edit, and distribute
Wikipedia’s editors should treat each other with respect and civility
Wikipedia has no firm rules

Out of all these pillars, the second one (in boldface) is of interest to me due to the aspects of currency, accuracy, and reliability, which are so important when determining if sources are good or not.

Selwyn and Gorard (2016) conducted a study on the treatment of Wikipedia by academia, pointing out a key finding regarding its usage: “Wikipedia mainly plays an introductory and/or clarificatory role in students information gathering and research.” But in spite of the great news this represents for students, one must practice caution when using Wikipedia for research papers and other forms of scholarship. The fact that anyone can write or contribute to a Wikipedia article…anonymously, should give you enough reason to do your due diligence as a scholar-to-be to make sure that the article you are reading can be verified from other reliable sources. `

Here’s a guide to help you evaluate a Wikipedia article, librarian-style. And how do we librarians assess the content of a source? By doing the CRAAP test. Yes, because librarians like myself do give a CRAAP about Wikipedia and see some value in what it offers to students in their initial phases of research.

B. The CRAAP Test

CRAAP is an acronym that stands for Currency, Relevance, Authority, Accuracy, and Purpose. This test was created by Sarah Blakslee (U of California, Chico), to serve as a way to assess sources’ accuracy and reliability. All five components are key in assessing any type of source you encounter, whether it comes from Google, Wikipedia, a governmental website, or in grey literature (e.g. pamphlets, conference proceedings, annual reports, etc.). Let’s take a quick look at each one of these components and the type of questions you could pose when reading a source, like a Wikipedia article.

Currency – The timeliness of the information

Questions we should pose when checking for Wikipedia article’s currency:

How was this page first developed?
How has it evolved?
How has its content been revised or updated?
Does this topic require current information?

Relevance – The importance of the information
for your needs

Questions we should pose when checking for Wikipedia article’s relevance:

How informative is this page?
Is there a diversity of contributors?

Authority –The source of the information

Questions to pose when checking for levels of authority present in Wikipedia article’s authority:

Who are the key contributors to this page?
Are they qualified to create and update content? If so, in what ways?
Has a contribution or a revision to the Wikipedia article been made anonymously?

Accuracy – The reliability, truthfulness and
correctness of the content

Questions to pose when checking for Wikipedia article’s accuracy:

How informative is the content presented?
Can the content be supported by evidence?
Has this content been reviewed or refereed by qualified authors?
Does the article provide with additional sources that verify its content?

Purpose – The reason the information exists

Questions to pose when determining why Wikipedia article was written:

Why was it produced?
What have been some of the main points of discussion among contributors and readers of this DH page?
How is this information going to be used?

C. The Digital Humanities Wikipedia page: Does it pass the CRAAP test?

A reliable source needs to pass all five criteria items. Just because an article is current and accurate, it does not mean that it is authoritative. Just like an article that is authoritative, it does not mean that is still holds currency if it was written decades ago and new findings have proven it to have obsolete ideas. So let’s give a CRAAP about Wikipedia and see if the following article passes the test:

The Digital Humanities Wikipedia Page (See Figure 1)

Figure 1: Screenshot of Digital Humanities Wikipedia Page. Image Credit: Nashieli Marcano

Use the following set of activities as a guide to deconstruct a Wikipedia article with the use of the CRAAP test. There are no right or wrong answers, and so the answers you will generate through these exercises do not have to match the ones I have provided as a guide. These exercises will only provide you with a partial way to determine if the DH Wikipedia page abides by the five criterion that comprise the CRAAP test.

1. Currency

Every Wikipedia article has a page history section that lists its previous iterations, each one with its corresponding time stamp contributor’s username, and a brief rationale statement. Article revisions are shown from newest to oldest, each one displayed one line at a time. Exploring the revision history of a Wikipedia article is key in understanding how its content has been revised and kept up-to-date. The following three activities will allow you to partially determine the currency of the DH Wikipedia page.

Currency Activity #1: View article’s modification history

To view the DH article edit history, click on the the “View History” tab, located on the top right to the DH page.

Answer the following questions:

When was the latest modification made?
Who was the contributor?
Why was the modification made?

Take a few moments to find the answer, then compare them with the following response:

The latest modification, was the reincorporation of three sources. This modification was made on November 2019, by username internateArchiveBot. (See Figure 2)

Figure 2: Screenshot of Wikipedia View History Page. Image Credit: Nashieli Marcano

Currency Activity #2: Describe the DH article’s first developments

Let’s now the first version of the DH article and see how the article was originally published. To accomplish this, click on the View History tab (top right of page), then click on “oldest” at the bottom of the page to select the oldest page of revisions, including the very first entry written on the DH Wikipedia page. (See Figure 3)

Figure 3: Screenshot of Wikipedia View History Page. Image Credit: Nashieli Marcano

Take a look at the very first entry and answer the following questions:

Who made the first contribution?

When was the article first published?

What were the major sections written?

Take a few moments to determine this information, then compare your answers with the following:

The first article, contributed by Dr. Elijah Meeks, included a very short definition of DH, with a See Also (Computers & Writing; Multimedia Literacy) and References (two entries) sections. Four days later, Meeks added a Table of Contents, and seven sections (Toolset, Lens, Document, Themes, See Also, References, and External Links). The next major changes that happened immediately after its creating involved expanding the definition of DH, adding items to the See Also and the External Links sections. These two sections have been expanded and modified ever since they were added.

Currency Activity #3: Describe major modifications

Select contributions made by JohnUnsworth, Elijameeks, GabrielBodard, Ammann, Teagrant or Willardmcarthy in the early iterations of the DH page. Take a few moments to explore some of ther major contributions, then compare notes with the following response:

A major modification took place in the DH page almost one month after its creation, when Meeks merged this page with the Humanities Computing page. From that moment, the number of entries under the External Links were consistently curated. In 2007, while Unsworth contributed to the newly-created section Software, Dr. William McCarty started contributing to the External Links section, then created the Objectives section in mid-2007. Towards the end of 2008, new sections were added, such as, Digital Projects. In 2009, the section Journals in the Field. In 2010, the See Also continued a rapid growth, thanks to the contributions of Dr. Gabriel Bodard, Dr. Michael Pidd, and Dr. Rudolph Ammann. In 2011, a contributor by the username Teagrant, curated the section Terminology, which went through most of its curation in 2012.

New sections, like Centers, Organizations and Institutions, Academic Disciplines, and Degree Programs and Certificates were also created in 2011. The most current iteration of the DH page contains more robust Projects and Criticism sections, each one containing respective sub-sections with a plethora of contents. There is also a new whole section devoted to Values and Methods in DH, which I can appreciate for it speaks of the shared principles of DH, amid the lack of a standard way of approaching DH and the vast diversity of projects and initiatives. I am a little underwhelmed by the Tools section, which I think should cover more ground on the different categories of digital tools that I’ve been learning in class (e.g. topic modeling, mapping, and networks).

2. Relevance

Relevance can be determined by finding out where the information comes from. Does the Wikipedia article include citations? Who is the intended audience? Is the content relevant for DH professionals, as well as for students interested in DH, or grant institutions? Is it diverse enough to capture the attention of different communities?

Relevance Activity #1: How informative and diverse is this page?

Browse through some of the different versions this page has experienced in the past 13 years. You may do this by visiting the “View History” tab of the DH page, selecting an edit line of interest, and clicking the (cur prev) links provided compare versions. To compare an old version with the current version, visit the old version then click cur. To a version with its previous iteration, click on prev. (See Figure 4 to view differences between editions)

Figure 5: Screenshot of Wikipedia Version Comparison Page — Figure 4: Screenshot of Wikipedia DH: Differences between versions page. Image Credit: Nashieli Marcano

As you do this with several versions, take a look at the diversity of contributors and contributions. Any surprising findings? Take a few moments to explore the different versions , then compare notes with the following response:

This page has information value in terms of the contributors’ cautious approach in defining DH and their acknowledgment of the difficulty in defining this field. The section on Values and Methods, although strangely included without any reference links (except for the last item), showcased this definitional complexity of DH. The two strongest sections of this page are Projects and Criticism.

The Projects page is very helpful in showcasing the amplitude of efforts undertaken by scholars in diverse contexts. It focuses not so much on the tools, but on the capabilities of DH initiatives in understanding humanistic phenomena and on the collaborative nature of these types of projects.I also really appreciate how responsive this page has been in addressing tough issues appearing in subsections like, Negative Publicity, Black Box, Diversity, and Issues of Access.

The most surprising part of this page is the lack of content under the Tools page, which is indicative of a focal shift from a technology-centric conceptualization to a more cultural-centric approach to DH. It also helps to have an introductory sentence or paragraph that explains the rationale behind each section.

If you have trouble finding the different versions of a Wikipedia article, please refer to the following source, which uses the example of the entry for “Heavy Metal Umlaut” to explain the evolution a of Wikipedia article:

Udell, Jon. “Heavy Metal Umlaut: the movie.” Strategies for Internet Citizens (blog), January 22, 2005.

3. Authority

In order to learn about the contributors of a Wikipedia page, you should become familiar with the User’s Page. If the edit was not made anonymously, we can check the User’s Page to see user’s biographical information, including interests, education, and professional experience.

Authority Activity #1: Find the User’s page of a contributor

Go back to the View History tab and go to the oldest page of the history log (back to 2007), and locate the individual by the username “Geertivp”. Click on “Geertivp” and identify the following:

Name, Work Experience, Education, Contributions
Can you locate the user’s website? Is this person appearing in social media outlets? Which ones? Could you contact this person if you needed to?

Compare notes with Figure 5:

Figure 4: Screenshot of Wikipedia User's Page — Figure 5: Screenshot of Wikipedia User’s Page. Image Credit: Nashieli Marcano

Authority Activity #2 Who were the early contributors of the DH Wikipedia page?

Take a few moments to identify the article’s earliest contributors, then compare notes with the following response:

The very first contributor was Dr. Elijah Meeks. Contributors from 2007 include: Dr. William McCarty (External links, Objectives), Dr. Gabriel Bodard (Objectives, See Also), SimonMahony (Objectives, Bibliography), and Unsworth (Software). Towards the end of 2008, GoldCorrie and Ms. Erin Belle contributed content (External Links) From 2011, this page has contributions from Teagrant and ARK (Terminology), and from Alf7e (Degree Programs, Organizations and Institutes). Sophia Chang, was also a contributor to this page (Centers and Institutes, Journals and Publications), until blocked in 2018.

Authority Activity #3: Identify the early contributors of the Wikipedia DH page, as well as the most recent ones

Take a few moments to identify some of the article’s contributors, then compare notes with the following response:

Some of the contributions made in the past three years include: Rudolph Ammann, who has added references to various sections (Objectives and History); Hectorlopez17 (Centers and Institutes, Definitions, and Project); MrOllie who has contributed as recent as October 2019, has mainly reverted edits made by other contributors; and Catsandthings, who had contributed with grammar and punctuation corrections, added and altered sections, modified titles, added and edited images, and added books to Further Reading sections. Also in 2016, Cheryl27 had also collaborated (History and Bibliography), and Librarian394 had added citations (Controversies) and fixed grammatical errors. The latest contribution was made by Geertivp this month by adding Wikimedia Projects to the article.

Authority Activity #4: Describe a biographical profile of some of the main contributors. What can we learn from their biographies?

It’s always a good exercise to learn a bit about contributor’s biographical profiles, as they provide a window into their projects and approaches to topics, in this case, the discipline of Digital Humanities.

Take a few moments to visit some of the contributors’ homepages or websites (external to Wikipedia article page) and learn more about about their biographies. Compare notes with the following response:

The originator of the Wikipedia DH page is Dr. Elijah Meeks, who currently serves as the Digital Humanities Specialist at Stanford University, specializing in developing humanistic research questions through spatial analysis, topic modeling, and visualizations. Meeks has been involved in key DH projects, including: The Digital Gazeteer of the Song Dynasty, ORBIS, and Mapping the Republic of Letters, among many other endeavors. Dr. Simon Mahony currently serves as Director of the UCL Centre for Digital Humanities, with research and teaching interests in the application of digital technologies in learning communities and collaborative environments.

Dr. Gabriel Bodard is a digital classicist (School of Advanced Studies, U. London), interested in methodologies and approaches of digital humanities in the subjects of ancient history, archeology, and classics. Mr. John Unsworth is the current Dean of Libraries at the University of Virginia. His research focus has been cyberinfrastructures for the humanities and he is credited for helping develop the HathiTrust Research Center. Dr. Willard McCarty is Professor of Humanities Computing (King’s College) and Digital Humanities (U. Western Sydney), with research foci in interrelations between DH and humanistic disciplines, interdisciplinary research, as well as computing historiography.

Matthew Kirchenbaum, who currently teaches and directs the Digital Studies Certificate Program at the Maryland Institute for Technology in the Humanities, claims to have successfully cohabited both the humanities and technology realms throughout his life and career. Erin Bell, MLIS, is the Technology Director of the Center for Public History + Digital Humanities at Cleveland State University. Other notable contributors include Rudolf Ammann, researcher at the UCL Centre for Digital Humanities; Amanda French, research and consultant with expertise in DH grant writing.

Authority Activity #5: Determine how these biographical profiles might influence they way these editors understand the topic?

Take a few moments to determine how their biographies might influence they way they treat the subject of Digital Humanities. Then compare notes with the following response:

These individuals have devoted their careers to the advancement of digital humanities as a field, as a practice, and as a discipline. Most have been innovators in DH with the creation of platforms, methodologies, and technologies that allow users to engage with humanities content.
Just to exemplify this influence, I would like to highlight a the possible influence of contributors, such as Bodard and McCarty, in expanding portions of the page, like the See Also section (2007) with the addition of topics (e.g. Digital Classicist and Digital Medievalist).

4. Accuracy

This CRAAP criterion can be partially correlated to one of Wikipedia’s Five Pillars,Wikipedia is written from a neutral point of view , which states:

“We strive for articles in an impartial tone that document and explain major points of view, giving due weight with respect to their prominence. We avoid advocacy, and we characterize information and issues rather than debate them. In some areas there may be just one well-recognized point of view; in others, we describe multiple points of view, presenting each accurately and in context rather than as “the truth” or “the best view”. All articles must strive for verifiable accuracy, citing reliable, authoritative sources, especially when the topic is controversial or is on living persons. Editors’ personal experiences, interpretations, or opinions do not belong.”

Let’s see how trustworthy and informative the DH Wikipedia page is, by (1) taking a look at the links provided, and by reading about the diversity of DH constructs provided by scholars.

Accuracy Activity #1: Indicate if the DH Wikipedia page can be trusted

Take a few moments to assess the accuracy of the DH Wikipedia page, then compare notes with the following response:

The content can be trusted to the point of gaining a panoramic grasp of DH. But to gain a deeper understanding of DH, I would advise making use of all the links provided, especially the ones listed on the References, Bibliographies, and Further Reading sections of the page. This page is an invitation, not to trust them, but to take an active role (as readers) to go visit those links they have provided, and to see for yourself what DH is about. I see this as a strategy that strengthens my trust in this page, as it doesn’t underestimate me as a reader.

Accuracy Activity #2: How informative is this page?

Let’s assess this in relation to definitions and discussions of digital humanities you may have explored in the past. Think of classical definitions offered by DH scholars, including Unsworth, Kirchenbaum, Terras, and McCarty. Reflect on how informative the DH Wikipedia page with regards to traditional definitions given by DH scholars. Does the evolution of the DH Wikipedia page reflect the definitional evolution of the concept Digital Humanities? If yes, in what ways?

Compare notes with the following response:

The iterations of the DH Wikipedia page possess a degree of correlation with the evolution of DH constructs defined by various scholars. The initial versions correspond with a second cycle of DH interpretations (2006-2008) that I read, a generation that started incorporating humanistic lexicon into their definitions. The statement given on the 2006 version of the Wikipedia page already expressed a focus on the “research and teaching of history, philosophy, literature, religious studies, or sociology.” Such statement addresses the same type of inclusivity described by DH scholars during that period.

Both Wikipedia contributors and DH scholars were also attempting to reconcile Humanities Computing and Digital Humanities, hence the Wikipedia Digital Humanities-Humanities Computing merge in 2006. We could also see these correspondences in the References section of the page, which included Unsworth and Schreibman as the initial sources. By the time the first iteration of the Wikipedia page was published, we observe the ways DH had already gone through a definitional shift from a computing-centric point of view to a state were the humanities were acquiring more real estate.

The next iterations of the page (2007) reflect Unsworth, Kirchenbaum, and McCarty’s contributions to the subject covered on Wikipedia. We start noticing a page that describes DH as a “highly interdisciplinary discipline” and as a medium for researchers to “integrate technology into their scholarly activities”. This takes me back to Unsworth’s shift in “the form of the attention… …from representation to abstraction”. The definition of DH in Wikipedia turns into one where practitioners get to do scholarship by keeping a balance between humanistic and computational frameworks. By 2011, Wikipedia had a more comprehensive approach to DH, which was described it as an “area of study, research, teaching, and invention”. This conceptual expansion sort of mirrors scholars’ perception of DH as a more abstract and integrative space of interactions, collaborations, and innovation. And while this iteration of the Wikipedia page has a more synthesized definition, we see sections like the See Also area expand with its newer sub-sections: Centers, Journals, Meetings, and Miscellaneous. This expansion the See Also section coincides with the reduction of the Bibliography page. This in my view is a moment of tension, which resembles the time period where DH thinkers were treating DH as a more complex networks of stakeholders, methodologies, and practices.

Perhaps, expanding the See Also section, while almost eliminating most entries in the References section was a way for Wikipedia contributors to exhort their readers to go beyond academic scholarship, to visit the diverse DH workflows, and to situate DH outside the ivory tower. In other words, let the readers discover their own interpretations of DH by visiting the websites of centers, journals, and meetings linked on the Wikipedia page).By 2014, we can see an evolved conceptualization of DH, more plentiful Reference and Bibliography sections, an Areas of Inquiry section, and even a unanimous “practical” definition of DH offered by scholar, Kathleen Fitzpatrick.

5. Purpose

Part of defining the purpose of information is by studying the major debates involving an issue and they ways in which consensus is reached among contributors and readers to resolve such issues. In the case of Wikipedia articles, it’s important to read about the different types of modifications that have taken place and the discussions that these generated. Do contributors provide with a rationale as to why modifications were needed? Were issues resolved for particular topics? Are there any biases or underlying motives that influenced the way the article content is presented?

There are two Wikipedia pillars that speak of how these discussions contribute to the open, editable model by which an article should be created and curated: Wikipedia’s editors should treat each other with respect and civility AND Wikipedia is free content that anyone can use, edit, and distribute

One of the ways to encourage contributors to work towards a common purpose in a respectful manner is the “Talk Page” feature. Every Wikipedia page includes “talk page”, where editors discuss improvements to articles and debate what stays and what goes. David Auerbach describes (and questions) the purpose of this “talk page” in his article “Encyclopedia Frown”:

“In the best case, which does occur reasonably often, spirited debate on the “talk” page of an article results in ongoing negotiations and refinement to an article until it is truly high quality. The editors who can work harmoniously in pursuit of this ideal goal of neutral ‘consensus’ can make editing Wikipedia a wonderful and productive experience.”

The following activities will help determine why content is created or modified, how it is contested by the Wikipedia DH community of discussants, and how disagreements have been resolved in ways that can sustain the main purpose of the DH page.

Purpose Activity #1: Study a major issue that generated a great deal of controversy on the DH page?

Take a few moments to identify topics discussed or debated regarding content revisions for this page. To do so, go through the edition history and select an edit line (which includes a date stamp, username, and type of edit). Click on the associated talk page, by clicking on the “talk” link. Compare notes with the following response:

The Further Reading section was the segment that probably generated the most interesting controversial topic. Back in March of 2017, a big dispute generated regarding the expansion of the Further Reading Section, between the Wikipedian-In-Residence (Thenewpulp) and an editor (MrOllie) The suggestion by the first party was to add a series of scholarly articles to enrich this section, but MrOllie called out a possible conflict of interest breach by posing the following question to the Wikepedian-In-Residence in an accusatory tone: “Would these happen to be scholarly articles that you have been paid to list on Wikipedia?” The newpulp responded with a denial of being paid to add such list and even warned MrOllie that making unpolite comments could result in being blocked from the Wikipedia community.

Purpose Activity #2: Explore some minor issues that generated a level of controversy on the DH page?

During the year 2017, other minor issues came up, including a suggestion made by contributor JonnieFox (January 2017) for the page to include content related to the access divide that exists based on race or socioeconomic status, questioning the “typically East-West” and high-access of DH in higher economic regions.One of the earlier issues raised had to do with the concept of Digital Humanities in general and the nature of the field as it is defined. This concerned was raised on February 19th, 2006, 20 days after Elijah Meeks contributed with the first article. The suggestion was to employ the concept of “humanities computing” instead of “digital humanities” if the contributors continue pointing to HC as a predecessor of DH. Meeks merges both the Digital Humanities page he had created and the already existing Humanities Computing. This is still reflected on the current page, with a statement that follows the page title: “(Redirected from Humanities Computing)”.Other minor topics of controversy that were generated over this page relates to neutrality in the treatment of page sections (2017), and the lack of sources that speak of DH projects approaching humanistic inquiries on race and gender (2016).

D. The Verdict: Does the Digital Humanities Wikipedia article pass the CRAAP test?

Your response is as correct as mine and I welcome your comments below. Here’s my verdict:

The Wikipedia Digital Humanities article has the potential for passing the test, in its currency, relatedness to the topic of Digital Humanities, the high qualification of its contributors, and the accuracy of its content.

The high number of edits has been made to this page, its diversity of collaborators, and the way the page has evolved, I can say have contributed an increase in its quality. In their study on Wikipedia . Wilkinson and Huberman (2007) established the following relationship between degree of collaborative editing and high-quality articles: “We examined all 50 million edits made to the 1.5 million English-language Wikipedia articles and found that the high-quality articles are distinguished by a marked increase in number of edits, number of editors, and intensity of cooperative behavior, as compared to other articles of similar visibility and age.” This correlation seems to be happening with the DH Wikipedia page. This is an article that has gone through 13 years of edits, curated by a group of high-caliber contributors, becoming an interesting venue of interactions and debates. The DH page is a testament on how the process of reaching consensus in such venue can help improve and move the field of Digital Humanities forward.

Feel free to use the CRAAP criteria when reading other articles of interests, regardless of the type of source. And speaking of sources, here are the sources I used for this analysis.

Sources:

Auerbach, David. “Encyclopedia Frown.” Slate, December, 11, 2014.

Selwyn, N., & Gorard, S. (2016). Students’ use of Wikipedia as an academic resource—Patterns of use and perceptions of usefulness.

Wilkinson, D. M., & Huberman, B. A. (2007, October). Cooperation and quality in wikipedia. In Proceedings of the 2007 international symposium on Wikis (pp. 157-164). ACM.

“Evaluating Information – Applying the CRAAP Test Meriam Library California State University, Chico”. https://library.csuchico.edu/help/source-or-information-good