Digital Medievalist 9 (2013). ISSN: 1715-0736.
© David G. Halsted, 2013. Creative Commons Attribution-NonCommercial licence

Accuracy and quality in historical representation: Wikipedia, textbooks and the Investiture Controversy

[ Skip to Abstract | Return to Top ]

Peer-Reviewed Article

Accepting Editor: Malte Rehbein, University of Passau.
Recommending Reader: James Coltrain, University of Nebraska.
Recommending Reader: Antonia Landois, Leibniz Universität Hannover.
Received: September 3, 2012
Revised: December 13, 2012
Published: October 11, 2013

[ Skip to Navigation | Return to Colophon ]

Abstract

Wikipedia’s popularity is unquestioned, but a perceived lack of accuracy and reliability in articles on historical topics prevents historians from embracing it more fully. This article argues that accuracy may be only one component of overall quality. While Wikipedia may have demonstrable shortcomings, it also has strengths in areas such as completeness and accessibility. These strengths appear when historical narratives in Wikipedia are compared to other sources of historical information readily available to American undergraduates. The article compares Wikipedia’s entry on the Investiture Controversy to current scholarship and textbook treatments of the theme. On a broader view of quality, Wikipedia appears in a more favorable light than it does when we employ a narrow focus on accuracy about specific dates and events.

Keywords: Wikipedia; Investiture Controversy; information quality; accuracy; completeness; western civilization.


[ Return to Navigation]

Introduction

§ 1    Wikipedia is hugely popular. As of this writing it is the fifth or sixth most-visited site on the Web, trading spots occasionally with the Chinese search engine baidu.com; it has been in the top 10 since March 2007 (Wikipedia.org is more popular than...). Wikipedia gets 500 million page views each day (Page Views for Wikipedia). At a little under four page views per individual user, according to Alexa.com (Wikipedia.org site info), that means 1.25 million people visit the site daily.

§ 2    A fair amount of the information served to this vast audience is historical. Of the top 1000 pages in the English-language version, over 200 have significant historical content, ranging from articles on individual countries (the United States, France) to articles on historical figures (Leonardo da Vinci, Julius Caesar) or periods (the Renaissance) (Wikipedia article traffic statistics). In the 30 days prior to this writing, the top 20 articles covering historical periods, figures or events from before 1700—titles like " Roman Empire", "Islam", "Henry VIII of England", "Isaac Newton"—had 5.4 million views. Wikipedia is putting historical information at the fingertips of a vast audience. It is by far the most efficient single means of distributing freely editable historical information that the world has seen.

§ 3    Despite Wikipedia’s power, it has not received universal accolades from historians. The primary issue has been accuracy (or lack of it) in articles on historical topics. A piece in Nature famously found that Wikipedia’s error rate was about on a par with Encyclopedia Britannica (Giles 2005), and a number of studies since then have reached similar conclusions in various disciplines (for some examples, see Brown 2011, 339). For historical subjects, Roy Rosenzweig found in 2006 that Wikipedia’s historical information was at least serviceably accurate, but Lucy Holman Rector’s 2008 study was more negative, concluding that "Wikipedia does not fare as favorably as do other reference resources under scrutiny for accuracy, comprehensiveness and reliability" (Rector 2008, 20). An historian, Neil Waters of Middlebury College, provided one of the more notorious Wiki-skeptical anecdotes when he noticed a group of students repeating erroneous information and traced the source to a factual error in a Wikipedia article in his field, Japanese history. In an incident that drew attention from the national and indeed global press, Waters forbade his students to use Wikipedia as a source in written assignments because of potential inaccuracies in its articles (Waters 2007, 15).

§ 4    In the field of history at least, Wikipedia’s "accuracy problem" cannot be said to have disappeared. Recent pro-Wikipedia articles by Sarah Baker (Baker 2012) and American Historical Society President William Cronon argue for its use despite the presence of errors: "Although the wide-open Wiki world sometimes harbors howling errors, even outright fraud, the overall quality of Wikipedia content is remarkably good," writes Cronon (2012). The real problem is that the on-going threat of inaccuracy is built in to Wikipedia’s model of (mostly) freely-editable content. Of course, on-going mechanisms for dealing with inaccuracy are also built in, but it is just not that hard to find an individual error of historical fact on Wikipedia.

§ 5    The accuracy question, in short, is not going to go away. But is it the right question? This article argues that we should take a broader view of information quality when we evaluate Wikipedia content. Instead of focusing on names, dates and events, we should look at other dimensions of quality, among them completeness and accessibility. I have taken Wikipedia’s article on the medieval Investiture Controversy as a case study. When we consider more dimensions and compare Wikipedia to other information sources readily available to American undergraduates, the appeal and power of Wikipedia—perhaps even for faculty—becomes more apparent. This broader view can help historians move forward with Wikipedia instead of fretting about whether it gets all the details right.

Project scope

§ 6    The most common method of checking the accuracy of historical information on Wikipedia is to compare its articles to other articles covering the same topic in traditionally edited sources like Encyclopedia Britannica or the American National Biography (Rosenzweig 2006 and Rector 2008 both follow the methodology of Giles 2005).

§ 7    This study will cast a slightly wider net. We are interested not just in comparing Wikipedia to other reference works, but instead, assessing Wikipedia as a part of the overall set of information available to American undergraduates. After all, undergraduates do not go only to a single source when they seek information for courses. In fact, the leading source of information is not a reference work or encyclopedia of any type, but the course readings assigned by instructors (Head and Eisenberg 2009, 18). Thus, we are interested in how Wikipedia operates within the information search space available to undergraduates. This means that we will be looking at course readings—textbooks aimed at undergraduate survey classes—in addition to reference works. By being more aware of how undergraduates use the various kinds of sources that are available to them, including both textbooks and online sources, we can help them learn to evaluate and synthesize all types of sources more effectively.

What is a Wikipedia article?

§ 8    A Wikipedia "article" is not a fixed entity in the way that an article in a research journal is, or a chapter in a textbook. Any user may create Wikipedia articles. An article will likely start out as a stub, with a few sentences giving a sense of direction for a future, more developed version. Articles can be converted to non-stub status without administrative intervention (Wikipedia:Stub). Content on Wikipedia is expected to adhere to three content principles: No Original Research, Neutral Point Of View, and Verifiability. It is not supposed to represent new knowledge, but a synthesis of existing knowledge, verifiable by reference to public sources.

§ 9    According to Wikipedia’s "Protection Policy" page, "Wikipedia is built around the principle that anyone can edit it,” though some pages have more restrictive editing policies to ward off so-called "edit wars" (Wikipedia: Protection policy). It is this openness that is most disturbing to historians. William Cronon writes that at first the whole idea "seemed at best dubious—and, more likely, laughably absurd" (2012). On Wikipedia, expertise and credentials do not necessarily win the day. "People of all ages, cultures and backgrounds can add or edit article prose, references, images and other media here. What is contributed is more important than the expertise or qualifications of the contributor," says Wikipedia’s About page (Wikipedia: About). This can be a hard pill to swallow for professors with PhDs, since academic culture is nothing if not a hierarchy based on expertise and qualifications.

§ 10    Wikipedia pages change over time, and those changes are all tracked and viewable by users. The obsession with tracking versions is a direct borrowing from IT culture. Programming organizations use version-control software to make sure they can manage the introduction of new code. Since any new edit may cause an unintended bug, it is important to be able to identify and, at need, eliminate or correct any bad code introduced by one of the programmers. If you look at the "View History" tab of any page on the English Wikipedia, you will see several functions borrowed from programming. The ability to compare two revisions of a Wikipedia article is based on a respectable old Unix utility called "diff." "Diff" allows you to compare two versions of a file so you can see what has changed. The "undo" function is like a version-control system’s ability to revert to an earlier version of code. Another version-control borrowing can be found under "Revision History Search"—the page you see there is labeled "WikiBlame." This is a reference to the "blame" function in code-management, a search across versions to answer the question "who gets the blame for putting this bad code in the build?"

§ 11    One of the more useful imports from software culture is the presence of revision numbers. Each line of the "View History" page represents a version of the article, with the date of the revision, the user name or IP address of the person who made it, the total file size and any note the editor may have made concerning the revision itself. Under the hood, each version is tagged with a revision number. For example, the June 16, 2012 revision of the article on the Investiture Controversy, which I’ll be discussing in more detail later in this piece, has an id of 497865484. The prior version has an id of 497736853. In a programming shop, having a unique number like this makes it possible to identify changes without ambiguity. For scholars, the revision number could be a useful way to overcome the changeability of a Wikipedia article in citations.

§ 12    If you want to read through a Wikipedia article as it appeared over time, click on the "View History" tab when the article is displayed. Then click on a revision date (highlighted with yellow below).

Figure 1: Screenshot showing the Wikipedia revision date. Screenshot showing the Wikipedia revision date.

§ 13    To navigate through versions, look for this at the top of the revision page:

Figure 2: Screenshot showing the top of the Wikipedia revision page. Screenshot showing the top of the Wikipedia revision page.

Clicking on the left-hand (diff) will show you the differences between the previous revision and the one you are looking at now. Clicking on the words "Previous revision" will show you the previous version in its entirety. The other links work in the same way, so to move forward in time, you just keep clicking on either "Newer revision" or, if you want to see exactly what was changed, the right hand (diff).

§ 14    What all of this means is that a Wikipedia "article" is not and does not intend to be a single, fixed text. It is closer to a collection of related texts that evolve over time, as you can see by scrolling through the revisions. This is why claims about accuracy on Wikipedia must always be limited; if a given revision is accurate, there is no guarantee at all that the next revision will still be accurate. If you catch an article at the right time, you may even see a fresh act of vandalism, such as the assertion that Plato "was an ancient hawaiian weather man and surfer, writer of cosmo girls . . . Plato is widely believed to have been a student of Barney the purple dinosaur and to have been deeply influenced by his dog, cutie" (McGlynn 2011). This system is always and essentially dynamic. It does not try to achieve an equilibrium or static point of balance. Strictly speaking, an article on Wikipedia is a collection of texts, not a single object. It is a set of its own versions over time, and a gateway to the family of related articles to which it is linked.

Measuring Information Quality: Reframing the debate about accuracy

§ 15    In information science, accuracy is not the only measure of information quality. As the following diagram suggests, possible dimensions or components of overall quality may include objectivity, completeness, representation (the presentation of material in a clear, comprehensible manner), and accessibility, but there is no one fixed definition of quality (Arazy and Kopak 2011, 91; Naumann and Rolker 2000).

Figure 3: Different aspects of Information Quality. Different aspects of Information Quality.

§ 16    The present study is primarily interested in accuracy and completeness. Because of Wikipedia’s emphasis on a neutral point of view, an objective tone tends to be preserved over time by the editing culture. As for representation, consistency of style is not necessarily Wikipedia’s strong suit, just as one might expect in an open, multi-author system. As Wikipedia articles are developed they do tend to converge on a clear structure. However much college professors might like to see Wikipedia’s writing improve, stylistic and organizational issues are clearly not costing Wikipedia audience share.

Accessibility

§ 17    Accessibility is the area where Wikipedia is so strong that comparisons to it become almost fruitless. Wikipedia is so easily accessible that authors may discount accessibility as an investigative object altogether in studies (Arazy and Kopak 2011, 90). This becomes an important factor when making multidimensional comparisons of Wikipedia to other sources of information, such as databases behind paywalls or textbooks that must be purchased, possibly even in physical form. An economist might express accessibility in terms of cost—the cost of obtaining a given piece of information—and argue that this cost must be taken into account when calculating a user’s optimal information-seeking path. Less-accurate but readily accessible information might be more valuable to a user than more accurate information that is too expensive, in terms of time, money or convenience, for him or her to retrieve easily. Although many undergraduates may have ready access to paywall-protected databases, depending on the resources their schools are able to devote to subscriptions, accessibility should be taken into account when considering the potential value of Wikipedia overall. Here we will not focus on it, so as to keep a clear view of the issues around accuracy and completeness—the areas in which historians have concerns.

Expertise, accuracy and trust

§ 18    Expertise does not just play a role in the creation of content for undergraduates. It also plays a role in evaluating it. User reactions to content differ depending on their expertise in the content area. In a recent study, Lucassen and Schraagen demonstrated that readers with domain expertise and novices establish trust based on different dimensions of information quality. Novices use surface and source features, such as style and perceived quality of references, to establish trust. In contrast, experts tend to weigh accuracy more heavily (Lucassen and Schraagen 2011). Lucassen and Schraagen were able to demonstrate this effect experimentally, and I believe their findings are confirmed by the behavior we see in students and in professors with respect to Wikipedia. Neil Waters’ article describes his almost visceral reaction to errors of fact in his area of expertise (Waters 2007). Sarah Baker writes that while her students understand that faculty does not want them to use Wikipedia as a source, they do not fully understand why (Baker 2012). Lucassen and Schraagen’s work helps us explain both phenomena. Professors like Waters are experts and for them, the dimension of accuracy looms large. Conversely, students are novices. They judge (because they must) based on surface features such as perceived objectivity and representation. It makes sense, then, that professors and students establish different trust relationships to the same content.

§ 19    Lucassen and Schraagen’s work helps us put accuracy in a larger context. We can re-cast the question about Wikipedia’s absolute accuracy with respect to historical names, dates and events as a question about the information quality and trustworthiness of Wikipedia’s content compared to other information sources undergraduates use in information seeking for courses. A 2009 study found that 84% of students use Wikipedia as a source of information for course work. However, course readings remain the most important source of information for them. More students reported using commercial research databases offered by vendors like ProQuest and online library catalogs than reported using Wikipedia (Head and Eisenberg 2009, 18). Wikipedia is part of an overall information search and retrieval space, a reference horizon within which students operate. In order to evaluate the role of Wikipedia as a source of information for undergraduate course work, we should not ask only about accuracy. Instead, we should take accuracy into account along with other dimensions of information quality and ask how Wikipedia stacks up to other sources of information used by undergraduates, especially materials provided for course readings or accessible via online databases at the library. As Jacobson and Mark argue, it is possible to teach students to interact with the materials in this search and retrieval space even in introductory classes (Jacobson and Mark 2000, 257). Looking beyond accuracy to other dimensions of information quality helps us see Wikipedia’s potential role in the active development of critical engagement with information from varied sources..

The Investiture Controversy

§ 20    The remainder of this paper is a case study in accuracy and completeness for a single Wikipedia article, on the medieval Investiture Controversy. As previously mentioned, I will focus primarily on revision 497865484, by user TaintedMustard at 13:24 on 16 June 2012. (This version is permanently available at <http://en.wikipedia.org/w/index.php?title=Investiture_Controversy&oldid=497865484>).

§ 21    As a topic in medieval European history, the Investiture Controversy is hardly one of Wikipedia’s more popular or frequently edited topics, but it is a central part of the narrative of the Middle Ages for survey courses and textbooks.

§ 22    The outlines of the Investiture Controversy are, of course, familiar, though work in the last thirty years has caused a revision of some of the narrative’s basic elements (Schieffer 1981, 114-176; Kempf 1982; Cowdrey 1998, 103-107; Blumenthal 1988, 120-121, Miller 2009). The Controversy stands for a whole range of questions about the relationship between secular and ecclesiastical power in the Middle Ages. Lay investiture is the ceremonial investiture of a bishop, archbishop or abbot by a layman with the staff and, from the time of Henry III on, the ring. In its simplest form, the narrative about the Investiture Controversy sees investiture as a primary driver of the conflict between Pope and Emperor. In this version, Gregory is seen as moving against lay investiture from the beginning of his papacy on, furthering his work as an advocate of a tradition of ecclesiastical reform stretching back to Cluny. At his Lenten synod in 1075, Gregory promulgated a decree against investiture by laymen. Henry IV, king of the Germans (not yet Emperor), defied Gregory by elevating Tedald as archbishop of Milan, upon which Gregory remonstrated with the king. In January of 1076, the king met with the bishops of Germany, who renounced their obedience to Gregory and sought to depose him. Gregory responded by excommunicating Henry and deposing him, releasing his subjects from their oaths of obedience. The conflict between Henry and the Pope caused the princes of Germany to invite Gregory to Germany, and Henry, seeing his position grow untenably weak, intercepted Gregory at Canossa in January 1077. After doing penitence for three days in the snow, and upon the intercession of, among others, Matilda of Tuscany and Abbot Hugh of Cluny, Henry was absolved by Gregory.

§ 23    The moment at Canossa is easy to see as an important symbol of the shift in power structures that leads into the High Middle Ages, with its weakened Empire, localized power in Germany, rising communes in the city-states of Italy, and an ascendant, bureaucratizing Church.

§ 24    But it did not end the conflict. A resurgent Henry created an anti-Pope, Clement III, who elevated Henry to the Imperial dignity. Henry pursued Gregory, who ended his life in exile in 1085. The controversy over investiture did not die away with Gregory or indeed with Henry, however. In the Concordat of Worms of 1122, both ecclesiastical and secular powers were given a role in the investiture process itself—in Germany, an oath of homage to the Emperor would precede ecclesiastical investiture, while in Burgundy and Italy it would work the other way around, and the secular power was given a voice in case of electoral disputes. Thus the settlement of the Controversy is seen as a step away from the theocratic concept of anointed kingship in the early Middle Ages and toward a state power conceived apart from outwardly sacral legitimization—a step in the long process of modernization by secularization. It is also seen as weakening the position of the Empire with respect to the German princes, and thus as a root cause of Germany’s "backwardness" in relation to the normative narratives of state-formation as we know them in England and France.

§ 25    The above account has come under pressure. In 1981, Rudolf Schieffer argued that the evidence for an explicit decree against lay investiture by Gregory in 1075 is not solid (Schieffer 1981, Blumenthal 1988, 120-121; Miller 2009). Instead, Schieffer argued, the prohibition of lay investiture did come to play a major role in the conflict between Gregory and Henry, but as a development rather than a cause. Gregory’s reforming zeal before 1075 was directed at simony, priestly celibacy and what we might think of as the command-and-control structure within the Church itself. The objections of the German bishops to what they perceived as Gregory’s high-handed intrusions into their spheres helps explain their resistance to him in January 1076. It was only later, in 1078 and in 1080, that Gregory and his supporters directly articulated a prohibition against lay investiture. Schieffer’s reviewer, Friedrich Kempf, though still holding that the sources are best explained if we posit a papal prohibition of lay investiture in February 1075, nonetheless agrees with Schieffer that lay investiture itself could not have been the primary cause of the conflict between Henry IV and Gregory VII (Kempf 1982, 414). Blumenthal (1988) and Cowdrey (1998) follow Schieffer. In her 2009 review article on the revisions of the Investiture Crisis narrative, Maureen Miller points out that historians have tried to tweak the narrative as a result of Schieffer’s work. She argues that it may be time to try to re-cast it, proposing several new directions for research, including a new emphasis on the active role of the laity, an attempt to get past the historical side-taking that is ultimately rooted in confessional allegiances, and a re-evaluation in light of recent studies of the dynamics of medieval lordship and "technologies of power" (Miller 2009). The question of what the controversy was really about, then, should have wider implications for the way we represent both it and the culture of the high Middle Ages, even at the level of survey classes.

Narrative versions of the controversy

§ 26    The Investiture Controversy is a long and complicated story that took nearly half a century to work itself out—and even then its conclusion was a compromise. In order to keep the scope of our investigation manageable we will focus primarily on the events of 1075-85 and on the Concordat of Worms in 1122, without trying to cover the intermediate events. We will look at two aspects: simple accuracy and completeness.

§ 27    In this study I looked at 3 major online reference works that should be available to most undergraduates: Wikipedia, NewAdvent.org’s online edition of the Catholic Encyclopedia, and the Academic Edition of the Encyclopedia Britannica. I also looked at 9 different Western Civilization textbooks from different publishers, taking a variety of historical perspectives (Coffin et al. 2011; Frankforter and Spellman 2012; Hunt et al. 2010; Kagan et al. 2010; Kishlansky et al. 2008; Levack et al. 2011; McKay et al. 2011; Sherman and Salisbury 2011; Spielvogel 2012). I want to examine how these online reference works and textbooks stack up against each other, taking into account their different purposes and intended audiences. One problem, of course, is the variation in scale. A scholarly account of Gregory’s career takes over 160 pages to get from Gregory’s election in 1073 to his death in 1085 (Cowdrey 1998, 75-241). Obviously, content like this will be more complete than a textbook entry of one or two pages. Reference works have yet a different purpose and format. Still, I think we should feel free to ask the same questions of all three types of information—course books, online reference works, and scholarly work—as we try to gauge the quality of historical representations for an undergraduate audience.

Accuracy

§ 28    Questions about factual accuracy can be vexed in the medieval period, since our source base is sometimes sparse and quite valid controversies may arise with respect to interpretation. Some facts are relatively easy to evaluate. For example, at the time the controversy started, in 1075-76, Henry IV had not yet been crowned Emperor. Thus, it is inaccurate, because it is anachronistic, to refer to Henry as the Emperor before this time. Yet 6 out of 9 of the textbooks make this error. None of the online reference works do. In this quite narrow—not to say pedantic—sense, the textbooks tend to be less accurate than the online reference works.

§ 29    However, the treatment of "facts" that depend heavily on interpretation are difficult to analyze across versions of a single narrative. The discussion above about the current revision of the position of lay investiture in the conflict between Henry and Gregory is a good example. Before Schieffer’s intervention, sentences like "Gregory promulgated a condemnation of lay investiture in 1075" or "The primary point at issue between Henry and Gregory was the question of lay investiture" would have seemed accurate. As a result of the revisionary work sketched above, however, the first sentence is at least seriously in doubt and the second does not match the results of recent scholarship, so should be regarded as factually inaccurate. In fact, the consensus at this point seems to be that both must be jettisoned (Blumenthal 1988; Miller 2009), although Cowdrey finds a middle way, positing a conditional prohibition of lay investiture in February 1075 that would still make the first sentence incorrect [Cowdrey 1998, 104-107]). Since Western Civilization courses themselves originated only in the 1920s, this revision has been in place for about a third of the period during which they have been taught (Stearns 2003, 12). Where, in our sample, is this finding taken into account?

§ 30    It turns out that is it hardly anywhere to be found. In fact, the only source in our sample that reflects this development in the scholarly world is Uta-Renate Blumenthal’s article in the Encyclopedia Britannica. Blumenthal, as it happens, has been one of the major contributors to the scholarly debate; the Britannica article on the Investiture Controversy is a nice example of the benefits of having research scholars write reference articles in their areas of specialty. The online version of the Catholic Encyclopedia has the older view, hardly surprising since it was written in 1910. We must conclude, then, that our sources are in general not accurate in their description of the main issue that brought king and pope into conflict.

§ 31    But what about the other aspects of the story?

§ 32    In the case of the Wikipedia, its account could be corrected or improved in several areas. It makes the role of investiture central to the conflict, just as do older works of scholarship and the textbooks. The January 1076 letter from Henry to Gregory is quoted before the elevation of Tedald in Milan is mentioned, giving a misleading impression about the sequence of events. The Dictatus Papae is incorrectly characterized as containing a prohibition of investiture. More seriously, a version of the article that was available from June 10 to June 15, 2012 gave the date of the Dictatus as 1076 rather than 1075, the year for which it is inserted in the Register (a look at that version will show that I made the correction myself). The description of the Concordat of Worms could use some work, although following the link to the main article on the Concordat will point the reader to a more expansive account.

§ 33    All in all, the Wikipedia article I reviewed is quite serviceable, though the writing could use a little polishing and there are a couple of flat-out errors.

§ 34    Is this better than the textbooks--even though these are not crowdsourced, but prepared by professionals? None of the textbooks I reviewed reflects recent scholarship. They all make investiture the driving issue between Gregory and Henry, though the Controversy itself may be characterized as a symptom of more fundamental historical conflicts (eg. Coffin et al. 2011, 259). The attribution of a fully-developed, public position against lay investiture to Gregory in 1075 is not the only problem. Some narratives conflate events, blurring the development of Gregory’s position in the years 1075-1080. Spielvogel, for example, quotes a passage from 1078 in a way that implies a date of 1075 (Spielvogel 2012, 288). Other errors include mistaken characterizations of the positions of figures in the controversy. Hunt et al asserts that Gregory thought Henry "was just a layman who had no right to meddle in church affairs" (Hunt et al. 2010, 319), though Gregory’s interactions with Henry make it quite clear that the pope sought a different balance between ecclesiastical and secular powers, not a sudden exclusion.

§ 35    Sometimes these errors are quite serious, as in the assertion in Frankforter and Spellman that "the new pope immediately made his position clear," followed by a summary of this ostensible position based on the Dictatus Papae. But Gregory was elected in 1073 and the Dictatus appears to date from two years later. In fact, the first two years of Gregory’s papacy were an important period of development, marked, tellingly, by a relatively cordial relationship to Henry. McKay et al. starts the narrative by attributing to 1075 the provisions of Gregory’s decrees in 1078 and 1080. The text then asserts that "[i]mmediately, Henry IV in the Holy Roman Empire, William the Conqueror in England, and Philip I in France protested" (which is pure fiction). It goes on to claim that, "when Gregory died in 1085, Henry invaded Italy, capturing Rome" (McKay et al. 2011, 253). In fact, Henry had invaded already in 1081, and Gregory died in exile. As a last example, two of the texts claim, incorrectly, that Hildebrand was a Cluniac monk (Frankforter and Spellman 2012, 264; Coffin et al. 2011, 259; cf. Blumenthal 1988, 114).

§ 36    As far as accuracy alone is concerned, then, the Wikipedia article I reviewed can certainly hold its own in this group. The Western Civilization texts all contain some discrepancies from recent scholarship, and many of them contain conflations that would make it difficult or impossible to reconstruct accurately the sequence of events in the crucial period 1075-1080. Some contain out-and-out errors, among them errors arguably more serious than the ones I identified in the Wikipedia article. Wikipedia is clearly behind Britannica in terms of its inclusion of recent research but it is not so clearly inferior to textbooks in terms of accuracy as one might expect. As we will see in the next section, Wikipedia comes out ahead in terms of completeness, our other comparative dimension.

Completeness

§ 37    We might think of the story about Investiture as a modular narrative; the narrator may choose to include or exclude various elements. The choice of elements affects the dimension of completeness, which we may regard as a dimension of overall information quality. Excluding a particular dimension of the base narrative may not diminish accuracy per se, but it may be regarded as a diminution of completeness. In the case of a complex narrative like the Investiture Controversy, completeness can become quite important.

§ 38    We can measure completeness in our accounts by identifying some elements that tend to appear in some narratives and not in others. The more of these elements a narrative has, the more complete it is. There is obviously a degree of arbitrariness in the identification of these elements, but this approach helps us get beyond dates and events in evaluating and comparing historical narratives.

§ 39    I chose twelve optional aspects of the Investiture narrative to consider. The choice of topics is based on accounts in the scholarly literature, rather than textbooks or summaries in reference works. While certainly at one level arbitrary, the list was made with Milller’s suggestions for further interpretive directions in mind (Miller 2009). Here is the list:

  • Hildebrand/Gregory’s relationship with the pataria movement in Milan
  • The conflict over the elevation of Tedald to archbishop in Milan
  • The resistance in Saxony to Henry’s rule, beginning in 1073
  • Episcopal resistance to Gregory’s reforms in Germany, predating the January 1076 conference with Henry IV
  • Canossa
  • Matilda of Tuscany
  • The intercession by Matilda and others between Henry and Gregory
  • Gregory’s exile in 1085
  • Controversies over investiture in England
  • Controversies over investiture in France
  • The Concordat of Worms, 1122
  • Regional differences in the Worms settlement (the process in Germany differed from that in Burgundy and Italy)

§ 40    Each of these could fill out the context of the Controversy. It will be noted that in some cases I have broken out single elements (Canossa, Worms) into two or more separate elements. For example, it is possible to mention Canossa without mentioning Matilda, and it is possible to mention Matilda without mentioning her role as an intercessor between Henry and Gregory. Each decision to include or exclude changes the story’s range of significance. Other, valid choices could have been made as well; for example, I ignored the anti-King Rudolf of Rheinfelden and the anti-Pope Clement III, the role of Gregory’s Norman allies, and two noblewomen who along with Matilda were in Gregory’s inner circle, the Empress Agnes (mother of Henry IV) and Beatrice of Tuscany. I also ignored the details of actual debates over investiture and disciplinary actions taken by Gregory against bishops whose elevations were examined for irregularities, although these historical actions help us understand the actual state of play as it was perceived by contemporaries.

§ 41    It can be seen at once how the inclusion or exclusion of these elements might affect the overall meaning of the narrative. For example, the pataria were a reform-minded tradesmen’s group in Milan. Excluding them reduces the range of social classes involved in the controversy and de-emphasizes the role of the laity, with a corresponding (and perhaps distorting) emphasis on ecclesiastics as the primary drivers of the reform movement. As Miller says: if we take a Church-centric view, our understanding of historical causality in the period is narrowed: "When causes are posited, they are Great Popes" (Miller 2009, 1574). Leaving out Matilda writes out another form of lay involvement in reform and leaves us with no female protagonists. Leaving out the role of the intercessors at Canossa, among them Matilda and Abbot Hugh of Cluny, reduces the events there to a personal confrontation between Henry and Gregory—better staging but worse history.

§ 42    Here are the results of the survey I undertook. On the left is the narrative element considered, followed by the number of occurrences in textbooks and in the online sources (Britannica, Catholic Encyclopedia, Wikipedia):

Narrative element Occurrences in textbooks (9 total) Occurrences in online reference works (3 total)
1. Gregory and the pataria 0 0
2. Conflict over Tedald 5 1
3. Resistance in Saxony prior to 1075 0 1
4. Episcopal opposition to Gregory prior to Jan 1076 meeting with Henry 0 0
5. Canossa 8 3
6. Matilda 4 1
7. Intercessions at Canossa 3 1
8. Gregory’s exile in 1085 5 1
9. Investiture in England 0 3
10. Investiture in France 0 3
11. Concordat of Worms, 1122 8 3
12. Distinction between regions in the Concordat 1 2

§ 43    This table shows a significant amount of "clumping" of narratives in the textbooks. It is easy to see at a glance that some elements, such as Canossa and the Concordat of Worms, are mentioned across the board, while others, such as the controversy around Tedald or Gregory’s death in exile, make frequent appearances. Matilda’s relative absence is a little surprising, as she is a compelling figure in her own right and adds a crucial element of gender balance to the story. Some of the "mentions" counted here only give her name, without giving a sense for this formidable personality. Some elements, such as the impact of the Controversy in England and France, get significantly better coverage in the reference sources online.

§ 44     While this aggregate view can give us a sense for the relative importance of the elements of the Investiture Controversy narrative across different information sources, it does not allow us to compare the strengths of individual sources. One way to measure the completeness of an individual publication is to count the number of elements that are present in it. When we do that, this is the result:

Source Element count
Kagan 3
Kishlansky 4
Coffin et al. 4
Sherman 5
Levack 3
Hunt 4
Frankforter 4
McKay 4
Spielvogel 4
Britannica 5
Catholic Encyclopedia 5
Wikipedia 9

§ 45    By this measure, Wikipedia is significantly more complete than any of the textbooks or the other online reference materials. However, there are some issues that must be addressed in evaluating this result. First, these numbers do not mean that the textbook accounts are in themselves incomplete. The textbooks I surveyed all have a version of the sequence of events from 1075 to 1022 that includes the key events in the conflict. If the Controversy is seen primarily as a conflict between two individuals representing two institutions, then these accounts are sufficient. Another limitation of this approach is revealed in the comparison between the online reference sources themselves. The Britannica article is very compact, accurate and by far the most reflective of current scholarship, but it does leave out context that might be of interest to an undergraduate information-seeker. Canossa is mentioned, but Matilda is not. The broader social and political worlds in Germany and Italy are left aside; the article’s very tight constraints deprive us of the context in which the protagonists moved and acted. The Catholic Encyclopedia article is long and comprehensive. It seems wrong to claim that it is less complete than Wikipedia, and in terms of contextualization it is far richer than the Britannica piece—certainly more so than this table implies. But it is still true that the Wikipedia article covers a lot of territory, and a lot of that territory is quite useful. Certainly it helps expand on the limited accounts in the textbooks.

§ 46    To sum up our findings here, in some important respects Wikipedia is actually superior to the textbooks that we give our classes. It is always possible to find errors in Wikipedia, and, as I have tried to show here, it always will be possible. A Wiki-skeptic could even introduce errors to see how long they last. But if we take a step back from focusing on the accuracy of names, dates and events to look at other aspects of information quality a different picture emerges. Wikipedia begins to look like a valuable part of the overall information space available to undergraduates, one we should recommend and embrace for what it is.

§ 47    In this article we have not particularly emphasized one of Wikipedia’s most obvious strengths; it is free and accessible to anyone with an Internet connection. This is in very sharp contrast to the textbooks we examined or online resources behind paywalls like Britannica. If we had added accessibility into our evaluation, for example by weighting information quality with a factor for accessibility as part of an overall information quality index, our results might have been quite different. For a student in an American college or university, the cost of accessing these different types of sources—course readings, for-pay online resources, or free sources—may not be very different, as their access may be supported by the institution. Accessibility may loom larger for people who do not happen to be students enrolled in institutions with budgets (albeit shrinking) for for-pay online subscription services, or loan programs to subsidize the cost of textbooks. Of course, this was not our point; we wanted to show that Wikipedia can play a valuable role for students with (we hope) ready access to many information sources.. But if we had wished to describe the resources available to all citizens of the online world, Wikipedia suddenly becomes a phenomenon of an entirely different order. This is the one of the most exciting and most truly revolutionary aspects of this remarkable resource.

Works cited

Appleby, R. Scott. 1999. How the pope got his political muscle. The Free Library. <http://www.thefreelibrary.com/How+the+pope+got+his+political+muscle.-a055671611>. Accessed June 15, 2012.

Arazy, Ofer, and Rick Kopak. 2011. On the measurability of information quality. Journal of the American Society for Information Science and Technology 62.1: 89–99. doi:10.1002/asi.21447.

Baker, Sarah. 2012. A place for Wikipedia or putting Wikipedia in its place. New Horizons for Learning 10.1. <http://jhepp.library.jhu.edu/ojs/index.php/newhorizons/article/view/23>. Accessed June 19, 2012.

Blumenstock, Joshua. 2008. Size matters: Word count as a measure of quality on Wikipedia. In Proceedings WWW 2008, 1095–96. Beijing, China.

Blumenthal, Ute-Renate. 1988. The Investiture Controversy: Church and monarchy from the ninth to the twelfth century. Philadelphia: University of Pennsylvania.

———. 2012. Investiture Controversy (Roman Catholicism): Introduction. Encyclopedia Britannica. <http://www.britannica.com/EBchecked/topic/292452/Investiture-Controversy>. Accessed June 15, 2012.

Brown, Adam R. 2011. Wikipedia as a data source for political scientists: Accuracy and completeness of coverage. PS: Political Science & Politics 44.2: 339–343. doi:10.1017/S1049096511000199.

Coffin, Judith, Robert Stacey, Joshua Cole, and Carol Symes. 2011. Western civilizations: Their history and culture. 17th ed. Vol. 1. Norton.

Cowdrey, H.E.J. 1998. Pope Gregory VII 1073-1085. Oxford: Oxford University Press.

Cronon, William 2012. Scholarly authority in a Wikified world. American Historical Association. <http://www.historians.org/Perspectives/issues/2012/1202/Scholarly-Authority-in-a-Wikified-World.cfm>. Accessed May 15, 2012.

Frankforter, Daniel, and William Spellman. 2012. The West: A narrative history. 3rd ed. Pearson.

Giles, Jim. 2005. Internet encyclopedias go head to head. Nature 438.15: 900–901.

Head, Alison, and Michael Eisenberg. 2009. Lessons learned: How college students seek information in the digital age. Project Information Literacy Progress Report. Seattle, Washington: The Information School, University of Washington.

Hsu, Jeremy. 2012. Thrun leaving Stanford for online startup - Technology & Science - Innovation - Msnbc.com. MSNBC. <http://www.msnbc.msn.com/id/46138856/ns/technology_and_science-innovation/t/professor-leaving-stanford-online-education-startup/#.T9onHOJYuAw>. Accessed June 14, 2012.

Hunt, Lynn, Thomas Martin, Barbara Rosenwein, and Bonnie G. Smith. 2010. The making of the West: Peoples and cultures. 3rd ed. Vol. 1. Bedford/St. Martin’s.

Jacobson, Trudi, and Beth L. Mark. 2000. Separating wheat from chaff: Helping first-year students become information savvy. The Journal of General Education 49.4: 256–278.

Kagan, Donald, Steven Ozment, and Frank Turner. 2010. The western heritage. 10th ed. Vol. 1. Pearson [Prentice Hall].

Kempf, Friedrich. 1982. Review of Schieffer, Die Entstehung des päpstlichen Investiturverbots, 1981. Archivum Historiae Pontificum 20: 409–415.

Kishlansky, Mark, Patrick Geary, and Patricia O’Brien. 2008. Civilization in the West. 7th ed. Pearson [Longman].

Kupferberg, Natalie, and Bridget McCrate Protus. 2011. Accuracy and completeness of drug information in Wikipedia: An assessment. Journal of the Medical Library Association 99.4: 310–313. doi:10.3163/1536-5050.99.4.010.

Levack, Brian, Edward Muir, and Meredith Veldman. 2011. The West: Encounters and transformations. 3rd ed. Pearson [Longman].

Löffler, K. 1910. Conflict of investitures. New Advent Catholic Encyclopedia. <http://www.newadvent.org/cathen/08084c.htm>. Accessed June 15, 2012.

Lucassen, Teun, and Jan Maarten Schraagen. 2011. Factual accuracy and trust in information: The role of expertise. Journal of the American Society for Information Science and Technology 62.7: 1232–1242. doi:10.1002/asi.21545.

McGlynn, Katia. 2011. The funniest acts of Wikipedia vandalism ever. May 5. <http://www.huffingtonpost.com/2010/04/06/the-funniest-acts-of-wiki_n_522077.html#s78737&title=Plato_Was_A>. Accessed August 7, 2013.

McKay, John, Bennett Hill, John Buckler, Clare Haru Crowston, Merry Wiesner-Hanks, and Joe Perry. 2011. A history of western society. 10th ed. Bedford/St. Martin’s.

Miller, Maureen C. 2009. The crisis in the investiture crisis narrative. History Compass 7.6: 1570–1580. doi:10.1111/j.1478-0542.2009.00645.x.

Naumann, Felix, and Claudia Rolker. 2000. Assessment methods for information quality criteria. In Proceedings of 5th Annual Conference on Information Quality, 148–162.

Page views for Wikipedia, non-mobile, normalized. 2012. <http://stats.wikimedia.org/EN/TablesPageViewsMonthly.htm>. Accessed June 20, 2012.

Pellegrini, Mark, and Guang Gao. 2009. The effect of primary authorship on Wikipedia article quality. University of Delware. <http://www.eecis.udel.edu/~mpellegr/wiki.pdf>.

Rector, Lucy Holman. 2008. Comparison of Wikipedia and other encyclopedias for accuracy, breadth, and depth in historical articles. Reference Services Review 36.1: 7–22. doi:10.1108/00907320810851998.

Rosenzweig, Roy. 2006. Can history be open source? Wikipedia and the future of the past. Journal of American History 3.1: 117–146.

Schieffer, Rudolf. 1981. Die Entstehung des päpstlichen Investiturverbots für den deutschen König. Stuttgart. Hiersemann. Schriften der Monumenta Germaniae Historica, Bd. 28.

Sherman, Dennis, and Joyce Salisbury. 2011. The West in the world: A history of western civilization. 4th ed. McGraw-Hill.

Spielvogel, Jackson J. 2012. Western civilization. 8th ed. Wadsworth, Cengage Learning.

Stearns, Peter. 2003. Western civilization in world history. Themes in World History. New York and London: Routledge.

Waters, Neil. 2007. Why you can’t cite Wikipedia in my class. Communications of the ACM 50 (9): 15–17.

Wikipedia article traffic statistics. 2012. Grok.se. <http://stats.grok.se/en/top>. Accessed June 14, 2012.

Wikipedia contributors. 2012. Investiture controversy. Wikipedia, the Free Encyclopedia <http://en.wikipedia.org/wiki/Investiture_Controversy>. Accessed June 21, 2012.

Wikipedia contributors. 2012. Dictatus Papae Wikipedia, the Free Encyclopedia. <http://en.wikipedia.org/wiki/Dictatus_Papae>. Accessed June 21, 2012.

Wikipedia contributors. 2012. Wikipedia:About. Wikipedia, the Free Encyclopedia. <http://en.wikipedia.org/wiki/Wikipedia:About>. Accessed June 21. 2012.

Wikipedia contributors. 2012. Wikipedia:No original research Wikipedia, the Free Encyclopedia. <http://en.wikipedia.org/wiki/Wikipedia:No_original_research>. Accessed June 21, 2012.

Wikipedia contributors. 2012. Wikipedia:Protection Policy. Wikipedia, the free encyclopedia. <http://en.wikipedia.org/wiki/Wikipedia:Protection_policy>. Accessed June 20, 2012.

Wikipedia contributors. 2012. Wikipedia:Stub Wikipedia, the Free Encyclopedia. <http://en.wikipedia.org/wiki/Wikipedia:STUB>. Accessed June 21, 2012.

Wikipedia.org is more popular than.... 2012. <http://meta.wikimedia.org/wiki/Wikipedia.org_is_more_popular_than>... Accessed June 21, 2012.

Wikipedia.org site info. 2012. <http://www.alexa.com/siteinfo/wikipedia.org>. Accessed June 21, 2012.