In the Library with the Lead Pipe is pleased to welcome guest authors Iris Jastram, Danya Leebaw, and Heather Tompkins.  They are reference and instruction librarians at Carleton College, a small liberal arts college in Minnesota.

Becoming forensic librarians

Image by smwright

“Wait, this is information literacy?” a rhetorician at our workshop exclaimed in excited surprise. “But this is so cool!” And we wanted to respond “YES!” not only from joyful pride but also out of recognition. After all, we too had had very similar reactions to our own work with information literacy, and not that long ago. We too had realized that information literacy could be different than we had originally thought (or that the ACRL information literacy standards had led us to believe). Information literacy could be more alive and integrated within the discourse of academic work. It could be more applicable across disciplines and genres and rhetorical goals. And these revelations remapped our practice.

Just two summers earlier we had pored over some sample papers pulled from Carleton College’s Sophomore Writing Portfolio submissions, debating whether we could see information literacy at work in those papers and if so, exactly what we could see. We couldn’t see the processes by which the students arrived at their final work or the assignments that prompted and guided them. All we had were the completed papers and a nagging sense of unease about what we could meaningfully say about information literacy in student writing based solely on samples of student writing.

As the hours ticked by, though, realization began to dawn. We had always said that information literacy was more than a discrete set of research skills, but when it came right down to it we had nearly always taught a set of research skills (cf Jacobs 2008, Simmons 2005, or Swanson 2004). We worked with students to help them develop researchable questions, formulate search strategies, evaluate what they find, and cite sources. We collaborated with faculty to help them design assignments that would lead students through these complex and iterative steps. These practices were good and valuable, but we now recognized them as only the beginning. Reading the finished papers themselves, we realized not only that research skills were hard to observe with any consistency, but also that we could trace the far richer information literacy habits of mind. We could be forensic librarians reconstructing our students’ understanding of the ways sources function in academic work based on the often subtle patterns left woven through the finished writing. These patterns coalesced around three dimensions, Attribution, Evaluation, and Communication, that we codified into a rubric and used to help us investigate our students’ habits of mind.

Attribution

We were surprised to find that we couldn’t really assess how well students followed citation style guidelines (one of the things we originally thought would be especially easy to see) because there are just too many citation styles and because many professors tell their students that “it doesn’t really matter as long as you’re consistent.” However, we found that we could see how well students guided their readers through the distinctions between their own thoughts and the thoughts of others and how well they helped their readers understand the nature of their sources. It became clear that teaching attribution as a habit of mind rather than citation as a rote skill would not only improve our students’ writing, but would also help them understand how sources function in academic writing in the first place.

So now when we teach, we help students understand citation as context. We emphasize that students can build contexts for themselves by paying attention to the contexts other scholars have built for them in the literature. Then we talk about how it is the students’ job to build similar contexts for their readers, and that this can help them decide what sources and citations belong in their papers. They can decide whether something counts as “common knowledge” by putting themselves in their readers’ shoes and wondering whether their readers would like to have the option of knowing more about that topic, and if so, leave them a citation to use as a starting place. Thinking of their own classmates as their “community of inquiry” we have them develop citation styles that would be instant context-building tools for their community, privileging information that matters to their classmates and leaving out extraneous identifiers. Then we explain how the citation style for their discipline performs that same function. This shift toward concentrating on the uses and functions of attribution breathes life into an otherwise stultifying topic, but more importantly it places students in the role of helpful knowledge creators rather than information compilers and potential plagiarists.

Evaluation

When we turned to the question of how students evaluate sources, the pattern emerged again. It turned out that we could tell very little about whether students had managed to uncover core resources or spread their wings beyond JSTOR. Instead, what we could see was whether or not students made compelling cases for their sources being the right sources for their papers. What’s more, once we knew what we were seeing, we could trace these same intellectual habits through papers that only included primary sources since selecting those sources is also an intellectual choice that involves matching evidence and claim. This insight helped us shift our instruction yet again.

Suddenly we realized that we could work with professors who often prefer not to include a research paper but still want to include an information literacy component in their courses, and with this realization, whole expanses of the curriculum opened up to us in ways that had seemed impossible just months earlier. We could work more closely with our language and literature departments, which place great emphasis on reading and writing about literature and far less emphasis on research. In these “non-research” classes students can analyze secondary literature that makes claims using similar types of sources to see what aspects of those sources are important to skilled scholars. We teach students to explore sources that will help them understand their primary sources well enough to see what might constitute an interesting question to ask of the source.

Even with more traditional research-based assignments, we shifted our teaching after realizing that students had been understanding the research process as one of gathering “everything” related to their topics and reporting on what they had gathered. Now we discuss bibliographies as representations of intellectual choice designed to present the most convincing claims possible, guiding the reader toward agreement with the claim by presenting the most convincing evidence possible. Bibliographies are rhetorical tools, too, not simply lists.

Communication

While all of the dimensions we identified have to do with communication, this dimension is distinct in that it focuses on how well students use the evidence that they’ve found instrumentally in the service of their own goals rather than ceding the main thrust of the paper to outside voices. In strong papers, students marshaled their evidence while maintaining their own voice and their own sense of purpose. In weaker papers, on the other hand, patch writing¹ and excessive citation signaled fundamental confusion about the sources themselves and the purposes for drawing on the works of others in the first place. Oddly, one of our most transformative findings felt the most obvious: students have to actually read and understand their source material, really integrate it into their thinking, before they can synthesize those sources into their own arguments effectively.

Of course, we aren’t content specialists. Yet, this insight helps us continuously improve how we teach familiar topics, like literature reviews. We recognized that students don’t actually know what a “literature review” means and what it is, really, that they are being asked to do in their papers. Backing up and deconstructing these as much as possible, we connect the notion of literature reviews with creativity, intellectual choice, and disciplinary conventions by showing an in-class video about originality, teaching students mindmapping, or having a class create research journals using Google Docs. Students also get a laugh out of demonstrations of bad literature reviews as conversations in which one person simply mimics or paraphrases another person. Acknowledging and summarizing previous points in a conversation is important but simply listing those points is socially and academically weak.

Looking toward the future

Reading papers and working with faculty and students in these new ways have opened up opportunities for more integrated and enlivened collaborations both with departments having deeply entrenched information literacy curricula and with departments that have not typically seen information literacy as highly relevant for their students. Emphasizing a “habits of mind” approach rather than a skill set approach, we are remapping our practice in ways that resonate more strongly with faculty and students across disciplines and courses on our campus. Information literacy is truly a “critical literacy” now, encompassing “the ability to read, interpret, and produce information valued in academia” (Elmborg 2006).

So yes, this is information literacy. And yes, information literacy is so cool.

Many thanks to our colleagues in the Carleton College Gould Library Reference & Instruction Department who together created this rubric, the design of the Information Literacy in Student Writing study, and with whom we shaped the thoughts and practices discussed here. Thanks also to Lead Pipers Ellie Collier and Eric Frierson and to Steve Lawson for helpful comments and edits.

Works Cited:

Elmborg, James. 2006. “Critical Information Literacy: Implications for Instructional Practice.” Journal of Academic Librarianship 32 (2): 192-199.

Howard, Rebecca Moore. 1999. Standing in the Shadow of Giants: Plagiarists, Authors, Collaborators. Stamford, CT: Ablex Publishers.

——, Tricia Serviss, and Tanya K. Rodrigue. 2010. “Writing from sources, writing from sentences.” Writing & Pedagogy 2 (2): 177-192.

Jacobs, H. L. M. 2008. “Information Literacy and Reflective Pedagogical Praxis.” The Journal of Academic Librarianship 34 (3) (May): 256-262.

Simmons, Michelle Holschuh. 2005. “Librarians as Disciplinary Discourse Mediators: Using Genre Theory to Move Toward Critical Information Literacy.” portal: Libraries and the Academy 5 (3): 297-311.

Swanson, Troy A. 2004. “A Radical Step: Implementing A Critical Information Literacy Model.” portal: Libraries and the Academy 4 (2): 259-273.

1. Patch writing refers to the practice of gathering verbatim passages from various sources and then piecing them together, much like a patchwork quilt, with connecting words and sentences. The term was coined by Rebecca Moore Howard in her 1999 work Standing in the Shadow of Giants.

In the Library with the Lead Pipe is pleased to welcome guest author Derek Rodriguez. Derek serves as a Program Officer with the Triangle Research Libraries Network where he supports collaborative technology initiatives within the consortium and is project manager for the TRLN Endeca Project. He is a Doctoral candidate at the School of Information and Library Science at The University of North Carolina at Chapel Hill, and is the Principal Investigator of the Understanding Library Impacts project.

Value for money in higher education

These are challenging times for colleges and universities. Every week it seems a new article or book is published expressing concerns about college costs,[1] low graduation rates, and what students are learning.[2] We also don’t have to look very hard to find reports computing the economic benefits of a college education to individuals.[3] Clearly, U.S. colleges and universities are under pressure to demonstrate that the value of an undergraduate education is worth its cost.

Graduation rates are important measures.  Personal income is  certainly a measure that hits home for most of us during these difficult economic times. However, stakeholders in higher education have had their eyes on a different set of metrics for many years: student learning outcomes. A recent example is A Test of Leadership, better known perhaps as the Spellings Commission report in which the U.S. Department of Education raised concerns about the quality of undergraduate student learning. The report called for measuring student learning and releasing “the results of student learning assessments, including value-added measurements that indicate how students’ skills have improved over time.” [4] In recent years, higher education has responded with new tools to assess and communicate student learning such as the Voluntary System of Accountability.[5]

As colleges and universities grapple with this challenge, academic libraries are also seeking ways to communicate their contributions to student learning. The recently revised draft Standards for Libraries in Higher Education from the Association of College and Research Libraries (ACRL) signals the importance of this issue for academic libraries.[6] The first principle in the revised standards, Institutional Effectiveness, states that:

“Libraries define, develop, and measure outcomes that contribute to institutional effectiveness and apply findings for purposes of continuous improvement.”[7]

And an accompanying performance indicator reads:

“Libraries articulate how they contribute to student learning, collect evidence, document successes, share results, and make improvements.”[8]

While libraries have made significant progress in user-oriented evaluation in recent decades, libraries still lack effective methods for demonstrating library contributions to student learning. Unless we develop adequate instruments (and generate compelling evidence) libraries will be left out of important campus conversations.

In this post I review current approaches to this problem and suggest new methods for addressing this challenge. I close by introducing the ‘Understanding Library Impacts’ protocol, a new suite of instruments that I designed to fill this gap in our assessment toolbox.

The challenge of linking library use to student learning

Demonstrating connections between library use and undergraduate student achievement has proven a difficult task through the years.  Several authors have suggested outcomes to which academic libraries contribute such as:  retention, grade point average, and information literacy outcomes.[9] I review a few of these efforts below.

Retention

Retention is a measure of the percentage of college students who continue in school and do not ‘drop out.’ A handful of studies have investigated relationships between library use and retention. Lloyd and Martha Kramer found a positive relationship between library use and persistence as students who borrowed books from the library dropped out 40% less often than non-borrowers.[10] Elizabeth Mezick explored the impact of library expenditures and staffing levels on retention and found a moderate relationship between expenditures and retention.[11] Several authors report a different ‘library effect’ on retention: holding a job in the library.[12] This finding is supported by evidence that holding a campus job, especially in an organization that supports the academic mission, is related with “higher levels of [student] effort and involvement”[13] in the life of the university and should logically lead to increased retention. Those of us who have worked in academic libraries have probably observed this mechanism at work with students we have known.

However, I believe relying exclusively on this measure is problematic. First, numerous factors influence retention and it can be difficult to isolate library impact on retention without extensive statistical controls. Second, retention is an aggregate student outcome; it is not a student learning outcome. Retention is an important metric in higher education and we should seek connections between library use and this measure, but it does not satisfy our need to know how libraries contribute to student learning.

Grade point average

Several authors have attempted to correlate student use of the library with grade point averages (GPA). Charles Harrell studied many independent variables and found that GPA was not a significant predictor of library use.[14] Jane Hiscock, James Self, and Karin de Jager, among others switched the dependent and independent variables in their studies and found limited positive correlation between library use and GPA.[15] Shun Han Rebekah Wong and T.D. Webb reported on a large-scale study with a sample of over 8,700 students grouped by major and level of study. In sixty-five percent of the groups, they found a positive relationship between use of books and A/V materials borrowed from the library and GPA.[16]

However, GPA-based studies have their problems. As Wong and Webb note, studies that use correlation as a statistical method cannot assure causal relationships between variables; they can only show an association between library use measures and GPA. As the old adage goes, ‘correlation does not imply causation.’ Do students achieve higher GPAs because they are frequent users of the library? Or do students who make better grades tend to use the library more? Without adequate statistical controls it is impossible to conclude library use had an impact on GPA. Also, as noted by Wong and Webb, it can be difficult to gain access to student grades to carry out this type of study.

Information Literacy Outcomes

Information literacy outcomes assessment is the most fully developed approach we have for demonstrating library contributions to undergraduate achievement. Broadly speaking, information literacy skills encompass competencies in locating and evaluating information sources and using information in an ethical manner. Instruction in these skills is a core offering in academic libraries and findings from Project Information Literacy suggest there is still plenty of work to do![17] ACRL has also created a suite of information literacy outcomes[18] to guide the design and evaluation of library instruction programs. Numerous methods have been used to assess information literacy skills including fixed-choice tests, analysis of student work, and rubrics.[19]

It is tempting to rely solely on student achievement of information literacy skills to demonstrate library contributions to student learning. However, a recent review of regional accreditation standards for four-year institutions suggests there is uneven support for doing so. Laura Saunders found three of six regional accreditation agencies specifically name information literacy as a desired outcome and assert the library’s prominent role in information literacy instruction and assessment of related skills. [20] Others rarely use the term “information literacy” in their standards. Instead, competencies such as “evaluating and using information ethically” appear in these standards as general education outcomes to be taught and assessed throughout the college curriculum. In part, I think this reaffirms for us that many in higher education associate information literacy outcomes with general education outcomes such as critical thinking.

While it may be encouraging for information literacy outcomes to be integrated into the college curriculum, I think this poses real difficulties when we attempt to isolate library contributions to these outcomes. If information literacy and critical thinking skills are inter-related, how are we to assess one set of skills, but not the other?  Heather Davis thoughtfully explored this issue in her post “Critical Literacy? Information!” finding that these competencies are intricately related and it is extremely difficult to teach (and assess) them independent of one another. If information literacy skills are taught across the curriculum, when, where, and by whom should they be assessed?  Where does faculty influence stop and library influence begin?

Information literacy outcomes are integral to undergraduate education, but these are not the only learning outcomes that stakeholders are interested in. And information literacy is not the library’s sole contribution to student learning.

I believe we need to shift course in our assessment practices and tackle ‘head on’ the challenge of connecting library use in all its forms with learning outcomes defined and assessed in courses and programs on college and university campuses. We should also link our efforts to the learning outcomes frameworks used in the broader academic enterprise. Broadening our perspective will provide a better return on our assessment dollar.

Where to begin

We can improve our ability to detect library impact on important student learning outcomes by carefully choosing our units of observation. Fortunately we can look to the literature of higher education assessment for clues.[21] My conclusion is that a one-size-fits-all approach to assessment is not likely to work for higher education or for library impact. Instead, our instruments should respect differences in students’ experiences. We should focus on the ‘high-impact’ activities in which faculty expect students to demonstrate their best work. Capstone experiences and upper level coursework within the academic major seem to fit the bill for four year institutions.

The academic major

Students majoring in the arts and humanities, the sciences, and the social sciences acquire different bodies of knowledge and learn different analytical techniques. We also know that learning activities, reward structures, and norming influences vary by discipline. This suggests the academic major plays a significant role in shaping expectations for student learning outcomes and the pathways by which they are achieved.[22] Shouldn’t student information behaviors vary by academic major as well? Our assessment tools should be sensitive to these differences.

The capstone experience and upper level coursework

Capstone courses are culminating experiences for undergraduate students in which they complete a project “that integrates and applies what they’ve learned.”[23] I think we should be studying information use during these important times for several reasons. First, there is ample evidence that the time and energy students devote to college is directly related to achieving desired learning outcomes.[24] Students who work hard learn more. Furthermore, students exposed to high-impact practices such as capstone experiences are more likely to engage in higher order, integrative, and reflective thinking activities.[25] Finally, there is strong evidence that student learning is best detected later in the academic career.[26]

If faculty expectations are at their highest and student effort is at its peak during the capstone experience and in upper-level coursework, shouldn’t studying student information behaviors during these times yield valuable data about library impact?

Speaking the language of learning outcomes

Assessing information use during upper-level and capstone coursework in the academic major is only part of the puzzle. We also need to link library use to student learning outcomes that are meaningful to administrators and policy-makers. I’d like to share two frameworks for student learning outcomes which I think hold great promise.

The Essential Learning Outcomes and the VALUE Rubrics

The Association of American Colleges & University’s (AAC&U) Liberal Education and America’s Promise (LEAP) project[27] defined fifteen ‘Essential Learning Outcomes’ needed by 21^st century college graduates such as critical and creative thinking, information literacy, inquiry and analysis, written and oral communication, problem solving, quantitative literacy, and teamwork. These outcomes are applicable in all fields and highly valued by potential employers. A companion AAC&U project called VALUE (‘Valid Assessment of Learning in Undergraduate Education’) generated rubrics that describe benchmark, milestone, and capstone performance expectations for each outcome. These rubrics are intended to serve as a “set of common expectations and criteria for [student] performance” to guide authentic assessment of student work and communicate student achievement to stakeholders.[28]

Tuning

Some student learning outcomes are discipline-specific. For instance, one would expect students majoring in chemistry, music, or economics to acquire different skills and competencies. A process called Tuning is intended to generate a common language for communicating these discipline-specific outcomes.[29] First developed as a component of the Bologna Process of higher education reform in Europe,[30] Tuning is a process in which teaching faculty consult with recent graduates and employers to develop common reference points for academic degrees so that student credentials are comparable within and across higher education institutions.[31] Expectations are set for associate, bachelor, and master degree levels. Generic second cycle or bachelor degree level learning expectations as defined by the European Tuning process are noted below. Recent work funded by the Lumina Foundation has replicated this work in three states to test its feasibility in the U.S.[32]

Subject-specific learning expectations for second cycle graduates[33]

• Within a specialized field in the discipline, demonstrates knowledge of current and leading theories, interpretations, methods and techniques;
• Can follow critically and interpret the latest developments in theory and practice in the field;
• Demonstrates competence in the techniques of independent research, and interprets research results at an advanced level;
• Makes an original, though limited, contribution within the canon and appropriate to the practice of a discipline, e.g. thesis, project, performance, composition, exhibit, etc.; and
• Evidences creativity within the various contexts of the discipline.

The VALUE rubrics are currently being evaluated in several studies[34] and colleges and universities have begun using them internally to articulate and assess student learning outcomes.[35] While the Tuning process hasn’t yet ‘taken off’ in the U.S., the Western Association of Schools and Colleges recently announced a new initiative to create a common framework for student learning expectations among its member institutions.[36] As colleges and universities experiment with and adopt these frameworks, we should incorporate them into our library assessment tools.[37]

New tools for generating convincing evidence of library impact

As part of my doctoral research I created the Understanding Library Impacts (ULI) protocol, a new suite of instruments for detecting and communicating library impact on student learning outcomes. The protocol consists of a student survey and a curriculum mapping process for connecting library use to locally defined learning outcomes and the VALUE and Tuning frameworks discussed above. Initially developed using qualitative methods[38] the protocol has been converted to survey form and is undergoing testing during 2011. I illustrate how it works with a few results from a recent study.

A pilot project was conducted during the spring of 2011 with undergraduate history majors at two institutions in the U.S., a liberal arts college and a liberal arts university. Faculty members provided syllabi and rubrics regarding learning objectives associated with researching and writing a research paper in upper-level and capstone history courses. History majors completed the online ULI survey after completing their papers.

First, students identified the types of library resources, services, and facilities they used during work on their research papers, including:

• Electronic resources, such as the library catalog, e-resources and databases, digitized primary sources, and research guides.
• Traditional resources, such as books, archives, and micro-formats.
• Services, such as reference, instruction, research consultations, and interlibrary loan.
• Facilities and equipment, such as individual and group study space, computers, and printers.

The forty-one students who participated in the pilot project collectively reported 590 types of library use during their capstone projects ranging from e-journals, digitized primary sources, books, archives, research consultations, and study space. Electronic resource use dominated, but traditional resources, services, and facilities made a strong showing.

Students then identified the most important e-resource, traditional resource, service, and library facility for their projects and when each was found most useful.  At one study site over 60% of students said library-provided e-resources were important when developing a thesis statement.  And 90% of students said both library-provided e-resources and traditional resources were important when gathering evidence to support their thesis.  Over 80% of this cohort reported library services were important during the gathering stage.  These services included asking reference questions, library instruction, research consultations, and interlibrary loan.  These data help link library use to learning outcomes associated with capstone assignments and to the VALUE and Tuning frameworks.

Students reported next on helpful or problematic aspects of library use. For instance, students at both study sites extolled the convenience of electronic resources and the virtues of interlibrary loan, while several complained of inadequate quiet study space and library hours. Information overload and ‘feeling overwhelmed’ were also frequent problems. Time savings and ‘learning about sources for my project’ were mentioned often in regard to library services.

A series of open-ended questions ask about a challenge the students faced during the project. Almost fifty percent of the student-reported challenges were related to finding and evaluating sources and almost as many were related to managing the scope of the paper and issues with writing. Faculty and librarians can ‘drill down’ into these rich comments to understand challenges students face and shape collaboration faculty-librarian collaboration to meet the needs of future student cohorts.

Open ended questions also elicit powerful stories of impact. When asked what she would have done without JSTOR, one student replied:

“I honestly have no idea. I may have been able to get by with just the books I checked out and Google searching, but those databases, JSTOR specifically, really helped me.”

I hope these glimpses of recent pilot study results demonstrate the value of focusing our attention on important and memorable academic activities in students’ lives. Using both quantitative and qualitative methods helps us understand how and why libraries support students when the stakes are highest. Authentic user stories coupled with links between library use and student learning outcomes serve as rich evidence of library impact to support both advocacy efforts and internal improvements.

Future uses

The Understanding Library Impacts protocol is not designed to assess student learning; teaching faculty and assessment professionals fulfill this role. The protocol is intended to link library use with existing assessment frameworks. ULI results can then be used in concert with other assessment data enabling new partnerships with teaching faculty and assessment professionals. For example:

• The AAC&U Essential Learning Outcomes map well to general education outcomes at many colleges and universities.  The protocol’s use of the VALUE rubrics creates a natural vehicle for articulating library contributions to these outcomes.
• Understanding Library Impacts results may also integrate with third-party assessment management systems (AMS). As Megan Oakleaf noted in the Value of Academic Libraries Report, integrating library assessment data with AMSs allows the library to aggregate data from multiple assessments gathered across the library and generate reports linking library use to a variety of outcomes important to the parent institution.[39]

It is critical to find ways to connect library use in all its forms with learning outcomes important to faculty, students, and stakeholders. Doing so will bring the library into campus-wide conversations about support for student learning.

Thanks to Ellie Collier, Hilary Davis, and Diane Harvey for their comments and suggestions that helped shape and improve this post.  Thanks also to Hilary and Brett Bonfield for their help preparing the post for publication. I also want to thank the librarians, faculty members, and students at the study sites for their support and participation in this pilot study.

As librarians, we invest a great deal of time and effort instructing researchers on how to use our materials. This is especially true for special collections librarians, as we attempt to familiarize researchers with our unique resources and intricate collection arrangements. At the end of that instruction investment, we often wonder if we have been effective and what our students have truly learned. Have we taught them lasting research skills? If so, how do we illustrate the value of this service to cost-cutting administrators? How do we quantify the skills gained from working with our materials? Most importantly—how do we know if our instruction is making a difference for the researcher?

University Archives Photograph Collection, NCSU Special Collections Research Center

Last year, we had the opportunity to collaborate with the Association of Research Libraries (ARL) in the production of SPEC Kit 317: Special Collections Engagement. SPEC Kits, produced annually, survey the 124 ARL member institutions and collect data on current practices and policies of libraries.

We surveyed member institutions about the ways special collections are engaging students, faculty, and researchers through exhibits, events, and curricular involvement, and found that over 95% of respondents are involved in these activities (Berenbak et al., 2010, 16). A core component of many of these outreach efforts was instructional engagement in the use of special collections materials.

As we began the work of analyzing the survey results, a recurrent theme surfaced: the inconsistency of instructional engagement assessment. We began to ask ourselves questions about the concepts of evaluation and assessment of instruction, and how those terms are articulated and understood in the context of special collections. For example, when conducting a one-time instruction session, should evaluation focus on the librarian’s presentation skills, the use of archival collections by participants after a session, or the number of participating students or classes?

Although special collections are attempting to assess their instruction in a variety of ways, these efforts are not consistent, not standardized, and often not driven by a “need for information that fosters targeted change” (Ariew, 2007, 508). Many special collections would like to move assessment beyond use counts and anecdotal feedback, but the majority of ARL special collections have no plan or policy for outreach or engagement, and few have dedicated outreach staff (Berenbak et al, 2010).[1] Under these circumstances, how do special collections conceptualize what success looks like, or what measurements will convey when success has been achieved?

What IS being assessed?

University Archives Photograph Collection, NCSU Special Collections Research Center

Currently, most research libraries contribute annual statistics to government agencies and organizations such as the Association of Research Libraries (ARL).[2] These statistics include the size of each library’s collections, circulation, staff. Additionally, ARL asks libraries to describe instructional engagement efforts, reporting on the number of presentations that are given to groups, the number of participants in those groups, and the number of reference transactions. ARL provides the following definitions for its categories:

Presentations to Groups. Report the total number of sessions during the year of presentations made as part of formal bibliographic instruction programs and through other planned class presentations, orientation sessions, and tours . . . Presentations to groups may be for either bibliographic instruction, cultural, recreational, or educational purposes . . . the purpose of this question is to capture information about the services the library provides for its clientele. (Kyrillidou, & Bland, 2009, 100, emphasis added)

Participants in Group Presentations. Report the total number of participants in the presentations. For multi-session classes with a constant enrollment, count each person only once. Personal, one-to-one instruction in the use of sources should be counted as reference transactions (Kyrillidou, & Bland, 2009, 100).

Reference Transactions. A reference transaction is an information contact that involves the knowledge, use, recommendations, interpretation, or instruction in the use of one or more information sources by a member of the library staff. The term includes information and referral service. Information sources include (a) printed and nonprinted material; (b) machine-readable databases (including computer-assisted instruction); (c) the library’s own catalogs and other holdings records; (d) other libraries and institutions through communication or referral; and (e) persons both inside and outside the library . . . . (Kyrillidou, & Bland, 2009, 100).

Special collections departments are asked to contribute their numbers to their library’s general pool; ARL does not differentiate between general library instruction and the instructional efforts of special collections departments, a practice that makes the compiled statistics less useful for both ARL and the responding institutions.

It is clear from the results of our SPEC Kit findings that few institutions are doing any assessment of instructional engagement beyond what is required by ARL. Most responding institutions do not have formal evaluative measures. [3] Instead, these institutions tend to rely heavily on feedback and conversations with students, faculty, and researchers (Berenbak et al, 2010, 78, 79,91,92). [4] Special collections tend to either quantify the usefulness of their instruction when patrons mention they “learned” or “got something” from the instruction, or when they count how many items were checked out to patrons. And while counting items is arguably important for certain kinds of assessment, without measuring against a desired and stated outcome, what does a number like this really tell a special collections about its practices?

We know that very few special collections departments have any sort of formalized planning or policies guiding their instructional programming (Berenbak et al, 2010, 15). Different circumstances in each special collections contribute to this situation. In some cases, staff are short on the time and energy to devote to this activity (or, more commonly, staff tasked with this activity are a luxury most special collections cannot afford). In others, the responsibility of instruction is delegated at the time of need to the staff person whose background most closely aligns with the subject area of the instruction, limiting the consistency of the instruction. Sometimes the institution simply has not considered or not yet formally developed a plan for instructional engagement that fits into the overall activities of that special collections.

Whatever the circumstances, the results of our survey showed that most (80%) special collections are engaging in instructional sessions on a steady basis, and will likely continue to do so in the future — perhaps at an even greater frequency than their current rates (Berenbak et. al, 2010, 13).[5] If special collections are going to direct more focused efforts at planning their instructional engagement, they will need articulated and useful assessment metrics. After all, we cannot know if our engagement planning is a worthwhile investment if we are not assessing the outcomes of that engagement.

Though we recognize a need for better assessment, we are struggling to respond to this need. Determining which metrics will provide useful information about instruction is a conundrum that is keeping many special collections frustrated or hesitant to try assessment at all. A few institutions provide evidence that assessment is not daunting for everyone — one special collections, for example, looks for citations of materials from their holdings in student papers as an indication of the success of their instruction; some look for any citations of primary source materials; and some have undertaken short surveys and faculty interviews. [6] But by and large, most special collections seem uncertain as to what to collect or how to collect it.

What are we teaching?

University Archives Photograph Collection, NCSU Special Collections Research Center

When we do an instruction session with patrons in special collections, what are our objectives? Aware that specific objectives will vary from one session to another — informed by the needs, topics, or other parameters that may frame a session — there are still general objectives that we, as instructors, are always hoping to meet. Helping patrons find exactly what they need is possibly the most successful outcome we can achieve, but the many steps along the path to discovery are the components of instruction that perhaps most need to be measured in order to gauge the effectiveness of our instruction. Before patrons can find exactly what they are looking for, they first have to learn how to find it. From our perspective as instructors, a successful journey is more indicative of our instructional impact than arrival at the destination.

Why is the journey so important in special collections? Elizabeth Yakel, in her article “Listening to Users,” describes archives as a tabula rasa for researchers (Yakel, 2002, 122). She makes the important point that, unlike libraries where the “paradigm for assistance, access tools, and rules” has been learned by users from childhood at their public and school libraries, archives are considered a great unknown (Yakel, 2002, 122). The intricacies of the different rules, different materials, and different access tools often stump even the most experienced library user or researcher. Some archivists have correctly compared a successful special collections instruction session to an “archaeological dig” (Schmiesing & Hollis, 2002). Since the majority of special collections materials are not reflected on an item-by-item basis in either the library catalog or a finding aid, researchers must “dig” through boxes of materials, digital images, or artifacts. Because of the nature of this type of research, and because materials are not individually pre-selected for consumption, users must constantly reformulate their queries as they discover new materials. This often necessitates close collaboration with the special collections staff throughout the research process.

Unfortunately, this type of instruction is not accurately reflected in our measurements. Certainly limited head counts and use statistics do not paint an accurate picture of this work, nor do brief reactionary evaluations. ^[7] These evaluations are important and necessary, especially when reporting to organizations outside the library, but they fail to assess whether or not learning objectives are being met.

What’s out there now?

Academic libraries recognize that the reactionary evaluation of instruction often falls short, and have developed tools to help libraries make sure students and users are meeting learning objectives. These include guidelines such as ACRL’s “Information Literacy Competency Standards for Higher Education,” “Standards for Proficiencies for Instruction Librarians and Coordinators: A Practical Guide,” and skills tests such as Project SAILS. These guidelines give a framework for conducting meaningful evaluation for instruction librarians. And although there is no shortage of literature to be found on the subject of library instruction and assessment, we are only beginning to see similar literature and tools dealing with evaluating instruction in archives and special collections. A good example of this emerging interest can be found in Michelle McCoy’s article, “The Manuscript as Question: Teaching Primary Sources in the Archives – The China Missions Project.” McCoy details methods for the planning, instruction, and innovative assessment of a collaborative effort between the special collections and archives department at DePaul University and Professor Warren Schultz’s undergraduate History 199 Historical Concepts and Methods class.

Arguably the most important current project appearing in the assessment literature for archives and special collections is the Mellon-funded Archival Metrics Project, which includes models for assessing instruction (discussed at length below). In addition to the products themselves, Archival Metrics investigators have produced papers detailing initial studies for the project such as Duff and Cherry’s “Archival orientation for undergraduate students: An exploratory study of impact.“

Although we appear to be making progress, current assessment practices of efforts to instruct patrons on the use of special collections resources — both the materials themselves and the many discovery tools we’ve created (finding aids, databases, and subject guides) — would probably not receive a passing grade. Measuring and quantifying the journey is a daunting task.

While we should not stop collecting the statistics that are needed by ARL, the general library community — and especially special collections — should have a clear understanding of what these numbers actually represent. Any instruction or reference librarian will tell you that a headcount for their curricular sessions or a tally mark for a reference transaction does not adequately measure what they do or the instruction they provide. Especially when tally sheets obscure the difference between a quick question lookup and an hour-long research consultation at the desk.

We face a number of difficulties in achieving the goal of both establishing and collecting useful assessment metrics. In addition to a lack of policies or plans regarding curricular outreach and engagement, special collections often do not have positions designated to conduct instructional outreach. As discussed earlier, these duties often fall to the person in the department with the greatest subject knowledge, or the most available time.[8] It will be difficult to take on additional duties — especially when there are no easy answers and many special collections are short on staff and funding — but we offer some suggestions for ways that special collections might start.

Assessing the Journey

University Archives Photograph Collections, NCSU Special Collections Research Center

First, we must share. Some special collections are reaching students and evaluating their work with them in innovative ways, and the success of these efforts needs to be promoted.

Some of these innovations include:

• Making early contact with graduate student instructors so that they have experience working with special collections before they enter faculty positions;
• Working with subject librarians to incorporate relevant material into their teaching efforts;
• Giving awards to undergraduate research projects that make extensive use of the collections;
• Working with students to create virtual and physical exhibits highlighting materials used in special collections.

Assessment examples include:

• Monitoring use statistics of particular collections after an instruction session;
• Asking classes to donate copies of student papers to review the citations as a tool for better understanding the effectiveness of instruction;
• Using student focus groups to evaluate video tutorials;
• Monitoring books and articles published, performances given, and theses written;
• Tracking number and value of grants received;
• Examining web server statistics;
• Feedback forms and surveys;
• Monitoring number of graduate and practicum students using the collections;
• Soliciting and compiling one-on-one feedback from professors and students. [9]

The assessment practices that generate the most useful results are multipronged in their approach. The China Missions Project, for example, was organized in such a way that the students included a self-assessment of their experience using archival materials and research methods as part of their class research papers (McCoy, 2010, 55). Copies of these papers were deposited with the University Archives, and then staff conducted a qualitative survey of the papers to assess their responses. Recognizing that self-assessment in a graded paper might encourage students to write positive responses regardless of actual understanding, staff further scrutinized the papers’ citations. “Students who used a total of four citations or fewer or relied heavily on Wikipedia or other Web sources whose reliability cannot be verified were moved to a neutral position and not included in the positive total” (McCoy, 2010, 55). This approach—as well as other methods listed above—have plusses and minuses, but becoming aware of what other special collections are trying gives the rest of us a jumping off point.

A variety of special collections have noted their relationships with outreach and subject liaison librarians. Developing these close relationships can be beneficial for everyone involved. Understanding the holdings in a special collections, and illustrating how those materials might be incorporated into the curriculum, creates a great opportunity for instructing students in the value of primary sources. Drawing on the skills and backgrounds of subject specialists and instruction librarians can help special collections staff (often untrained in these areas) to develop sound instruction techniques.

Additionally, our colleagues in outreach and instruction have done an extraordinary amount of work related to best practices for evaluating instruction. In a 2007 article, Ariew and Lener state that one of the main insights gained in their study “Evaluating instruction: developing a program that supports the teaching librarian” was that teaching evaluation forms should be “tailored to specific classes, objectives and learning outcomes.” Most importantly, the group learned that “effective assessment requires a variety of assessment procedures be used” (Ariew, & Lener, 2007, 512). From teaching portfolios to 3-2-1 cards to surveys, the literature yields a great deal of information about what works and what doesn’t for each type of instruction. Although not all of these practices can be used to evaluate special collections instructional engagement practices, they provide guideposts to start from.

Fortunately, some people are starting to address the problem of how to assess the engagement work being done by special collections departments. The Archival Metrics Toolkits[10], for example, attempt to standardize evaluation in archives. This work recognizes that the “administration and use of primary sources are sufficiently different from libraries that they deserve tools that appropriately measure service to users” (Yakel, & Tibbo, 2010, 221). This creation of a standardized survey tool for archives could relieve a large part of the assessment burden, which is particularly important for archives with small staff. It also begins to answer the call for standardized evaluation that was so apparent in the results of our SPEC Kit survey.

The Archival Metrics Toolkit is particularly useful in laying out a set of standard questions about archives use, and they provide clear instructions on how to gather, compile, and analyze the data from the surveys. This information provides a basis for making comparisons across institutions, and could give special collections a better chance of identifying best practices and trends.

However, even the best of surveys have drawbacks such as rate of completion (particularly difficult in archives due to small numbers)[11], survey fatigue, and a focus on perceptions. Supplementing surveys by seeking evidence of skills mastered, such as citation analysis or testing, seems a more well rounded method to determining “what students have learned as opposed to how they feel about what they have learned” (Barclay, 1993, 198).

Special collections must clearly state engagement goals in order for any type of evaluation to be meaningful. Good practice in evaluating instructional engagement starts “with the learning objectives of the instructor” (or the department), and uses those to shape the tools being applied for evaluation (Areiw, & Lener, 2007, 512). As the libraries at Virginia Tech discovered, evaluation, when possible, should be unique to specific classes and desired student and faculty outcomes and will likely require that a variety of assessment procedures be used (Ariew, 2007, 512).

Conclusions

Today more than ever, library administrators are being asked to describe in a quantifiable way the value of their academic libraries and their practices. Therefore, special collections must be able to articulate to administrators why current evaluation methods are insufficient. Simple forms, tally marks, and baseline ARL statistics will never be able to get at the information we need to improve our practices. Specials collections need to make the case for developing more appropriate evaluation methods — even though this will require a commitment of valuable staff resources — and then make the commitment to using the results of these evaluations to enhance services. Ultimately, more meaningful data will help us provide better service to the students, faculty, and researchers who rely on special collections, and it will better equip us to tell their story and our own.

University Archives Photograph Collection, NCSU Special Collections Research Center

Huge thanks to our editors and advisors: Kathy Brown, Hyun-Duck Chung and Brett Bonfield. Your thoughtful comments have made this a much better post,` and sparked ideas for future avenues of exploration. And of course, thank you so much to all of our SPEC Kit co-authors, Adam Berenbak, Claire Ruswick, Danica Cullinan and Judy Allen-Dodson.

[2] The ARL states that these data “describe collections, staffing, expenditures, and service activities” of the 114 university libraries and 10 public, governmental, and nonprofit research libraries that collectively form ARL (Association of Research Libraries, 2008).

[3] This is not to imply that no one is attempting to assess instruction, but it is not standardized, and based on the survey responses, in general, it is fairly ad-hoc (Berenbak et al., 2010, 78, 79, 90, 91).

[4] Question 35 in the ARL Spec Kit did not specifically ask about the evaluation of instructional engagement but more broadly inquired, “What measure(s) have been used to evaluate special collections engagement with faculty/scholars/researchers who are affiliated with your institutions.” Many of the responses were similar to the more directed question 28 “What measure(s) are used to evaluate student use of unique materials in research projects.” The following types of statements made up the bulk of the responses: “no evaluation,” “much to few [sic],” “no particular measures have been used,” “nothing systematic,” “little evaluation has been done,” and “none to date.”

[5] The number of respondents actively working to engage students for curricular purposes is even higher at 99% (Berenbak et. al, 2010, 62).

[6] These examples are taken from the responses to the question “What measure(s) are used to evaluate student use of unique materials in research projects?” Responses include examining the “extent and breadth of primary resources and collections in any format,” a “learning outcomes survey,” and “discussion with faculty of results” (Berenbak et. al., 2010, 78, 79 and 80).

[7] Reactionary refers to a short survey after a presentation that often focuses on a students’ perception of the presentation rather than on whether or not new skills have been developed.

[8] When respondents were asked who had primary responsibility for coordinating curricular engagement, 15% had one individual who held primary responsibility, 15% said one individual leads a team or staff, 31% stated that all (or most) special collections staff shared the responsibility, and 39% noted that it varied depending on the project (Berenbak et al, 2010, 64).

[9] These examples are drawn from the responses to questions 28 and 35 of the ARL SPEC Kit 317 “What measure(s) are used to evaluate student use of unique materials in research projects” and “what measure(s) have been used to evaluate special collections engagement with faculty/scholars/researchers who are affiliated with your institution”(Berenbak et. al, 2010 78, 79, 80, 91). The respondent’s institutions are kept anonymous in SPEC Kit publications, so although these are specific examples, we are unable to point out specific schools for the purposes of this post.

[10]The toolkit includes sections for “Researchers” (A user-based evaluation tool for on-site researchers to evaluate the quality of services, facilities, and finding aids in university archives and special collections), “Online Finding Aids” (A user-based evaluation tool for visitors to evaluate the quality and usability of online finding aids in university archives and special collections), “Websites” (A user-based evaluation tool for visitors to evaluate the quality and usability of websites in university archives and special collections), ”Student Researchers”(A user based evaluation tool for students use the archives or special collections as part of a class and participate in archival orientations), and a “Teaching Support” section (A user-based evaluation tool for instructors who have used the university archives and special collections to evaluate its services.)

[11] Small numbers can make it difficult to obtain an appropriate sample size.

Resources

Ariew, S., & Lener, E. (2007). Evaluating instruction: developing a program that supports the teaching librarian. Research Strategies, 20. Retrieved from http://www.sciencedirect.com/science/article/B6W60-4MWXT97-2/2/e3d2a22ec51f17a15bc53a77240d49e7 doi: 10.1016/j.resstr.2006.12.020

Association Of Research Libraries, (2010). Association of Research Libraries: SPEC Kits. Retrieved Sep. 15, 2010, from http://www.arl.org/resources/pubs/spec/index.shtml.

Association of Research Libraries. (2008, February 4). Association of research libraries: annual surveys. Retrieved from http://www.arl.org/stats/annualsurveys/index.shtml

Barclay, D (1993). Evaluating library instruction: Doing the best you can with what you have, RQ 33 (2), pp. 195–202.

Berenbak, Adam, Putirskis, Cate, O’Gara, Genya, Ruswick, Claire, Cullinan, Danica, Dodson, Judy Allen, Walters, Emily, & Brown, Kathy (2010). Spec kit 317 special collections engagement. Washington, DC: Association of Research Libraries.

Knight, L. (2002). The Role of assessment in library user education. Reference Services Review, 30(1), Retrieved from http://www.emeraldinsight.com/journals.htm?articleid=861677&show=html

Kyrillidou, Marth, & Bland, Les. (2009). Arl statistics 2007-2008 Washington, DC: Association of Research Libraries. Retrieved from http://www.arl.org/stats/annualsurveys/arlstats/arlstats08.shtml

McCoy, M. (2010). The Manuscript as question: teaching primary sources in the archives – the china missions project. College and Research Libraries, 71(1), Retrieved from http://crl.acrl.org/content/71/1/49.full.pdf+html

Schmiesing , Ann, & Hollis, Deborah. (2002). The Role of special collections departments in humanities undergraduate and graduate teaching: a case study Libraries and the Academy, 2(3), Retrieved from http://muse.jhu.edu/journals/portal_libraries_and_the_academy/v002/2.3schmiesing.html

Yakel, E. (2002). Listening to users. Archival Issues, 26(2), 111-127.

Yakel, E., & Tibbo, H. (2010). Standardized survey tools for assessment in archives and special collections. Performance measurements and metrics, 11(2), Retrieved from http://www.emeraldinsight.com/journals.htm?articleid=1871188&show=abstract doi: 10.1108/14678041011064115

Treemap of mammals courtesy of Flickr user Arenamontanus

Introduction

I’m a slave to spreadsheets. Trying to decide between a stacked column bar chart and a 3-D area chart is par for the course in my work. Microsoft Excel© is great for many practical needs, but it doesn’t always support the need to create simple, compelling and interactive graphical data visualizations that are critical for libraries to best express value, communicate trends, and test assumptions about library services and collections. Data visualization is the study of strategies and methods for conveying information, as captured by data, in an efficient, functional way that leads to insights about a process or system. Good data visualization can drive home a point quickly and have lingering impact. Data visualizations can help you see something that you hadn’t noticed before. These days, libraries can’t afford to not be wise and impactful with the data that is collected and conveyed about patrons, services and collections. Many libraries are reporting declines in reference desk queries against the backdrop of massive surges in use of computers and other tech-related services. Most libraries are undergoing comprehensive reviews of journal and database usage (among other metrics) with the aim to cut collections to comply with shrinking budgets. To express these kinds of trends, to seek support, or to simply try to assess library collections and services, many libraries fall back on the use of tables with a few pie charts and bar graphs thrown in for added measure. When I started having conversations with my library colleagues about data visualization tools and techniques, I was humbled by what I didn’t know and embarrassed that I hadn’t heard about, much less tested, some of the data visualization tools that are surfacing. So, I decided to start exploring what I’ve been missing while hiding behind the ubiquity of Microsoft Excel© graphs and charts. In this post, I present some examples using a few popular data visualization tools and I give an overview of some inspirational guides for creating compelling data graphics that may help you better express your own library metrics. First, let’s explore a little further why data visualization matters for libraries.

Library data in context

Libraries serve users at the reference desk, circulation desk, and special collections centers. Library staff engage with constituents through committees and working groups, at the library security gate, and through online chat. Librarians attempt to expose valuable services and collections via library catalogs, carefully-crafted subject guides, during bibliographic instruction sessions, and via long lists of databases and online journals. Libraries assess usage and patrons needs via web statistics, gate counts, circulation transactions, LibQual surveys, usage statistics, and feedback forums. Why do we measure these experiences? To show value for money or time and to understand the uptake of our collections and services. Library value has been a popular topic since at least the 1930s and libraries have gotten better at showing return on investment (ROI). We’re not completely there yet, as the recent $1,000,000 IMLS sponsorship of “Lib-value” grant suggests. Libraries are pretty adept at measuring lots of different kinds of interactions, so how can we be so bad at demonstrating our worth and making our point? What if part of our problem in demonstrating value lies in how we attempt to showcase library value? Libraries also want to make good, sound decisions in the context of their user communities. Libraries collect a lot of data that encompass complex networks about how users navigate through online resources, which subjects circulate the most or the least, which resources are requested via interlibrary loan, visitation patterns over periods of time, reference queries, and usage statistics of online journals and databases. Making sense of these complex networks of use and need isn’t easy. But the relationships between use and need patterns can help libraries make hard decisions (say, about which journals to cut) and creative decisions to improve user experiences, outreach, achieve efficiencies, and enhance alignment with organizational goals.

Not another library ROI article, please!

Relax, this isn’t another post about how calculate library ROI nor is it about how to collect data that show library worth. This post is an exploration of visualization techniques that can help libraries make a compelling case to stakeholders and get insight about how data visualization can help libraries make more informed decisions. Disclaimers: I’m not an expert on visualization techniques; I’m part of the slew of librarians who need to know how to better illustrate what we do and learn how to better allocate resources. Visualization strategies have made their debut at library conferences already (e.g., 2009 Computers in Libraries; 2008 Computers in Libraries; 2009 NASIG Conference). However, I haven’t seen a groundswell of examples indicating that libraries have taken these strategies and these conference presentations to heart. What I have experienced is a few really good ideas popping up in conversations with colleagues about how to make the case for libraries in simple, compelling, visual ways. I want to share what I’ve learned so far in my exploration and open the door to some more good ideas.

Data needs to be “humanized”

During various conversations about how to represent library collections and expenditures data, one of my very smart colleagues, Cory Lown, introduced me to the work of John Tukey and Edward Tufte. Cory explained that Tufte’s aim is to encourage the use of as much data as possible (“to clarify, add detail”) and to use visualization techniques that “fit” the data.

“Often bar charts and pie charts (which tend to have low data to ink density) obscure more nuanced and interesting data. It’s not just about new and interesting tools, but matching the data to the right visualization so we can make use of data we have.” (Lown, 2009, pers. comm.)

This isn’t a trivial process by any means due to the uniqueness of each set of data due to variation in methods for collection, data clean-up, analysis and so on. But, according to Tufte’s principles, focusing on giving as much attention to the data in a chart, graph or image (aka “maximizing the data-ink ratio“) while reducing the “fluff” (aka “chartjunk“) (e.g., chart borders, text legends, background fill, decorations) can aid in getting the point across.

In the spirit of the work of Tukey and Tufte, a recent book, aptly named Beautiful Data (2009, edited by Toby Segaran and Jeff Hammerbacher) brings together a great compilation of data visualization, data handling and data sense-making strategies. In one chapter, Nathan Yau, also author of a terrific blog called FlowingData (to which I’ll refer a little later in this post), describes the development of a simple, user-friendly tool to track and measure what he calls “personal data” (e.g., eating, sleeping, travel habits). Yau is interested in creating tools for people to distill their personal data into stories that can help them understand patterns about their personal habits and eventually help relate people to the bigger picture about their impact on their environment and vice versa. This concept of creating a way for a person to relate to the bigger picture through data is an important lesson for libraries.

“Data has to be presented in a way that is relate-able; it has to be humanized.  Oftentimes we get caught up in statistical charts and graphs, which are extremely useful, but at the same time we want to engage users so that they stay interested. . .Users should understand that the data is about them and reflect the choices they make in their daily lives.” (Yau, 2009)

All of those interactions with patrons that libraries collect and track – circulations, journal usage statistics, cost/use metrics, etc. – are about the patron. However, most of the metrics that libraries present to make the case to patrons, aren’t presented in a way that relates the patron to the data. An example: Academic libraries spend a lot of money on journals. In fact, the NCSU Libraries spent around $6 million on journals during 2008-2009, but how many of our patrons know that when they download a journal article that it’s paid for by the NCSU Libraries? That $6 million dollars doesn’t necessarily “translate” to a user when they download an article. We tell them how many articles were downloaded, but is there a better way to make the connection between the user and the cost of resources? For the most part, library metrics aren’t good at telling stories that keep our users interested and help them inform the choices that they make. We are in need of some great ideas and examples from the field.

Simple data visualization tools

While several other visualization tools exist, I want to focus on three of the most popular tools and demonstrate what is possible using a few datasets that I’ve created using the kind of library metrics that you might be dealing with in your own library. After trying a few different types of library metric datasets in Google Gadgets for spreadsheets, ManyEyes and Swivel, my favorites are Google Gadgets and ManyEyes because of their ease of use and diversity of visualization styles.

Google Gadgets: First, I have to give props to Cory Lown for making me aware of Google Gadgets for spreadsheets. It’s really quite simple to use. If you have a Google account (e.g., Gmail), then you can use Google Gadgets. Log into your Google account, choose Google Documents, Create New Spreadsheet, then add your data (just as you would in Excel). Once your data is ready, go to the Insert menu and choose Gadget. As of the time of this writing, there are over 35 different visualizations you can choose from: everything from the standard bar charts to motion graphs to piles of money. The upside: You can experiment with the different visualizations and pick one that fits the point that you’re trying to make or the audience that you’re trying to reach. You can share your visualizations with a simple URL that you plug into an email, or into your website or blog. The downside: You don’t have a lot of control over font size or positioning of elements on the charts.

Google Gadget Motion Charts are excellent for showing change in values over time. They are the primary visualization mode for sites like GapMinder to illustrate changes in global issues over time. Below is an example using data that I collected from the data from the Association of Research Libraries on research library expenditures plotted against university expenditures spanning from 1982 through 2006. Try the motion chart with the default variables, then try changing them. You’re welcome to access the dataset itself to create your own data visualization.

ManyEyes: As part of the agreement to use ManyEyes with your own data, any data you upload is made publicly and freely available for others to use. After signing up with an email address, the process is easy and straightforward. You can be up and running with several visualizations of your data within a few minutes and you can share your visualizations with links in emails, or embed them in your website or blog. The upside: the choice of visualizations is pretty extensive: Word Tree, Phrase Net, Wordle, Tag Cloud, Bar Chart, Block Histogram, Bubble Chart, Network Diagram, Scatterplot, Matrix Chart, Treemap for Comparisons, Treemap, Pie Chart, Country Map, US County Map, World Map, Stock Graph, Line Graph, Stack Graph. The downside: if you want to compare more than two variables, you have limited options. The example that I’ve included here is data that I collected on the publication and citation patterns of NCSU scholars. Researchers at an academic university will almost always have more citation activity than publication activity in a journal. But just how much more? This visualization illustrates the scale of citations for journals in which NCSU scholars publish 0 times, 2 times, 3 times, on up to 41 times. Try the visualization below and experiment with the dataset to create other ManyEyes visualizations.

Swivel: With Swivel, you have a choice to let the data that you upload be freely available to others or to keep your data private. If you choose to keep your data private, be prepared to commit to a fee of $12/month. For most of us who use Excel to prepare data for upload into a tool like Swivel, an Excel toolbar is available from the Swivel Confectionary. The upside: You have a little more control over things like font size and font face (compared to Google Gadgets for spreadsheets); it’s just as easy to share data and visualizations (email or embedding in websites or blogs); and if you want your audience to be able to interact with your charts, Swivel makes that a trivial process. The downside: The choice of graphs is limited (Bar, Line, Area, Stacked Bar, Stacked Area, Scatter, and Pie) and the site isn’t very responsive with larger sets of data (e.g., I tested it with a dataset of over 1900 rows and it had trouble switching between different types of graphs). In this example, I’ve uploaded a small dataset of usage of the major types of digital collections provided by the NCSU Libraries. Try interacting with the pie chart and download the dataset if you want to use it to experiment with your own visualizations (you’ll need to create an account before you can do much with the data in Swivel).

Visualization inspiration

There are some excellent resources that help provide some insight into what is considered good and bad data visualization practices. These sites are filled with examples of interesting data visualizations to inspire your own work and in some cases (e.g., GapMinder) also offer datasets with which to experiment.

FlowingData: The FlowingData blog is one of the most compelling, idea-filled blogs I’ve come across – ever. Authored by Nathan Yau (UCLA PHD student in statistics focusing on data visualization), this blog highlights great examples of how to make a compelling point with data and visual creativity. FlowingData offers a great deal, but I want to point out 5 specific
visualization categories:

• Statistical Visualization – strategies for visualizing different types of statistics
• Infographics – examples of aesthetically pleasing and intellectually captivating modes for presenting data in graphical format
• Mapping – examples of data mapped to geographic representations
• Artistic Visualization – examples of data as art
• Network Visualization – examples of visualizations showing networks or relationships between entities

Not only does Yau collocate examples of how to display data to different audiences, but he also provides thoughtful analysis about why a visualization is effective (or not) and what could be improved about it.

Visualization ad

Infosthetics: Authored by Andrew Vande Moere (faculty member of Architecture, Design and Planning at the University of Sydney in Australia), Infosthetics acts much like the FlowingData blog, but tends to focus more on data as art. There’s overlap between Infosthetics and FlowingData, but you’ll find a slightly different perspective in Infosthetics – one that deals with data visualization from the design and interaction approach.

Visual Complexity: Manuel Lima uses the Visual Complexity blog to bring together examples and ideas around the study of the visualization of complex networks such as data from library systems, the social web, biological systems, and transportation patterns.

3D Dewey Data Visualization

GapMinder: GapMinder is an organization that runs a website for displaying trends in global issues such as poverty plotted against inequality indices or oil consumption plotted against oil production. Its main visualizations are based on Google Motion Charts, and have been featured in the famous TED Talks.

Information Dashboards: Information dashboards are user interfaces that serve the need of providing critical information at a glance. A book aptly named Information Dashboard Design (2006, by Stephen Few) promises to teach readers how to use graphs discriminately to enhance communication. Some excellent examples of information dashboards that might fit in library contexts are the Indianapolis Museum of Art (IMA) Dashboard (thanks to Adrienne Lai for sharing this site with me) and the Sprint Now Dashboard.

Indianapolis Museum of Art Dashboard

The IMA Dashboard presents simple, compelling data in a graphically aesthetic way. It tells a visitor things like how many plants are in the gardens, how many visitors are at the IMA, how much energy is being consumed by the IMA, and the number of active memberships. Each widget window leads to a little more information about the IMA, drawing the visitor in to learn more without overwhelming him/her with too many options or underwhelming with too few avenues to explore. The Sprint Now Dashboard, on the other hand, creates a slightly different experience. There’s a lot going on that isn’t necessarily relevant here – from the creepy voice-over to the number of eggs being produced or the number of people stuck in elevators – but the concept of surfacing this kind of real-time information is compelling.

The possibilities in libraries for these kinds of information dashboards are obvious. An external audience might find it helpful to know which books are being checked out (similar to Seattle Public Library display of circulating materials), real-time locations of available computers, how many journal articles are being downloaded, how many e-books are being read, the number of devices (e.g., laptops, ipods, Kindles) that are checked out, the latest articles by your campus researchers, upcoming community events, and maybe even an ROI metric on the value of library services and collections per tuition dollar (or tax dollar) per hour. Tack on a catalog search box, real-time webcam views of the coffee shop wait line and the Info Commons, and you’ve got a mode for making a case for the value of library services and collections while providing real-time information all in one view.

An internal audience of library staff and decision makers might find it helpful to see in a dashboard view the “health” of the library budget, cost/use metrics based on circulation data or electronic journal or e-book usage statistics, hourly gate counts, keywords searched in the catalog, cataloging activity, and a current snapshot of the composition and use of the collection broken down by format or by material type plotted against community demographics (e.g., number of full-text journal downloads per graduate student in the Chemical Engineering Department) among other things. Other than the Seattle Public Library, I am not aware of any libraries presenting this kind of (more or less) real-time, dynamic information dashboard to the public, but I suspect that any data displayed for public consumption would require that personally-identifying information be excluded.

Final thoughts

The ultimate goal of libraries is to help patrons make smart decisions about the information they use and create. As an extension of that goal, Jason Casden, one of the reviewers of this article noted that data visualization techniques should be adopted to be part of a library’s organizational culture for assessment and justification to not only best serve patrons, but also to help guide the allocation of limited resources. Investing in ways to leverage the data that libraries collect to show value, communicate trends, and test assumptions about library services and collections is part of the solution for making the library be all about the patron. Try out some of the visualization tools and sample datasets used in this post or share your own data visualization creations via the Comments.

Learn more

• Few, Stephen. 2006. Information Dashboard Design: the effective visual communication of data.
• Fichter, Darlene. 2008. “Data Visualizations.” Presented at Computers in Libraries 2008 Conference.
• Fichter, Darlene and Jeff Wisniewski. “Harnessing New Data Visualization Tools: Say It Visually.” Presentation at Computers in Libraries 2009 Conference
• Kurt, Lisa and Will Kurt. 2009. “Making Usage Data Understandable with Visual Representation.” Presented at the North American Serials Interest Group 2009 Conference.
• Legrady, George. 2005. “Making Visible the Invisible: Seattle Library Data Flow Visualization.” Presented at International Cultural Heritage Meeting 2005.
• Seattle Public Library Data Visualization: “Making the Visible Invisible.”
• Segaran, Toby and Jeff Hammerbacher. 2009. Beautiful Data: the stories behind elegant data solutions.
• Tenopir, Carol. 2009. “Value, Outcomes, and Return on Investment of Academic Libraries (Lib-Value).” IMLS Awards 3-Year Grant.
• Tufte, Edward. Statistician and author of data visualization books such as “Beautiful Evidence,” “Envisioning Information,” and “Visual Display of Quantitative Information.”
• Tukey, John. Statistician and author of Exploratory Data Analysis (1977).

Thanks

Thanks to the people who’ve opened my eyes to the possibilities and who reviewed this post and offered valuable feedback: Cory Lown, Jason Casden, Brett Bonfield, and Kim Leeder.