In the Library with the Lead Pipe is pleased to welcome guest author Derek Rodriguez. Derek serves as a Program Officer with the Triangle Research Libraries Network where he supports collaborative technology initiatives within the consortium and is project manager for the TRLN Endeca Project. He is a Doctoral candidate at the School of Information and Library Science at The University of North Carolina at Chapel Hill, and is the Principal Investigator of the Understanding Library Impacts project.

Value for money in higher education

These are challenging times for colleges and universities. Every week it seems a new article or book is published expressing concerns about college costs,[1] low graduation rates, and what students are learning.[2] We also don’t have to look very hard to find reports computing the economic benefits of a college education to individuals.[3] Clearly, U.S. colleges and universities are under pressure to demonstrate that the value of an undergraduate education is worth its cost.

Graduation rates are important measures.  Personal income is  certainly a measure that hits home for most of us during these difficult economic times. However, stakeholders in higher education have had their eyes on a different set of metrics for many years: student learning outcomes. A recent example is A Test of Leadership, better known perhaps as the Spellings Commission report in which the U.S. Department of Education raised concerns about the quality of undergraduate student learning. The report called for measuring student learning and releasing “the results of student learning assessments, including value-added measurements that indicate how students’ skills have improved over time.” [4] In recent years, higher education has responded with new tools to assess and communicate student learning such as the Voluntary System of Accountability.[5]

As colleges and universities grapple with this challenge, academic libraries are also seeking ways to communicate their contributions to student learning. The recently revised draft Standards for Libraries in Higher Education from the Association of College and Research Libraries (ACRL) signals the importance of this issue for academic libraries.[6] The first principle in the revised standards, Institutional Effectiveness, states that:

“Libraries define, develop, and measure outcomes that contribute to institutional effectiveness and apply findings for purposes of continuous improvement.”[7]

And an accompanying performance indicator reads:

“Libraries articulate how they contribute to student learning, collect evidence, document successes, share results, and make improvements.”[8]

While libraries have made significant progress in user-oriented evaluation in recent decades, libraries still lack effective methods for demonstrating library contributions to student learning. Unless we develop adequate instruments (and generate compelling evidence) libraries will be left out of important campus conversations.

In this post I review current approaches to this problem and suggest new methods for addressing this challenge. I close by introducing the ‘Understanding Library Impacts’ protocol, a new suite of instruments that I designed to fill this gap in our assessment toolbox.

The challenge of linking library use to student learning

Demonstrating connections between library use and undergraduate student achievement has proven a difficult task through the years.  Several authors have suggested outcomes to which academic libraries contribute such as:  retention, grade point average, and information literacy outcomes.[9] I review a few of these efforts below.

Retention

Retention is a measure of the percentage of college students who continue in school and do not ‘drop out.’ A handful of studies have investigated relationships between library use and retention. Lloyd and Martha Kramer found a positive relationship between library use and persistence as students who borrowed books from the library dropped out 40% less often than non-borrowers.[10] Elizabeth Mezick explored the impact of library expenditures and staffing levels on retention and found a moderate relationship between expenditures and retention.[11] Several authors report a different ‘library effect’ on retention: holding a job in the library.[12] This finding is supported by evidence that holding a campus job, especially in an organization that supports the academic mission, is related with “higher levels of [student] effort and involvement”[13] in the life of the university and should logically lead to increased retention. Those of us who have worked in academic libraries have probably observed this mechanism at work with students we have known.

However, I believe relying exclusively on this measure is problematic. First, numerous factors influence retention and it can be difficult to isolate library impact on retention without extensive statistical controls. Second, retention is an aggregate student outcome; it is not a student learning outcome. Retention is an important metric in higher education and we should seek connections between library use and this measure, but it does not satisfy our need to know how libraries contribute to student learning.

Grade point average

Several authors have attempted to correlate student use of the library with grade point averages (GPA). Charles Harrell studied many independent variables and found that GPA was not a significant predictor of library use.[14] Jane Hiscock, James Self, and Karin de Jager, among others switched the dependent and independent variables in their studies and found limited positive correlation between library use and GPA.[15] Shun Han Rebekah Wong and T.D. Webb reported on a large-scale study with a sample of over 8,700 students grouped by major and level of study. In sixty-five percent of the groups, they found a positive relationship between use of books and A/V materials borrowed from the library and GPA.[16]

However, GPA-based studies have their problems. As Wong and Webb note, studies that use correlation as a statistical method cannot assure causal relationships between variables; they can only show an association between library use measures and GPA. As the old adage goes, ‘correlation does not imply causation.’ Do students achieve higher GPAs because they are frequent users of the library? Or do students who make better grades tend to use the library more? Without adequate statistical controls it is impossible to conclude library use had an impact on GPA. Also, as noted by Wong and Webb, it can be difficult to gain access to student grades to carry out this type of study.

Information Literacy Outcomes

Information literacy outcomes assessment is the most fully developed approach we have for demonstrating library contributions to undergraduate achievement. Broadly speaking, information literacy skills encompass competencies in locating and evaluating information sources and using information in an ethical manner. Instruction in these skills is a core offering in academic libraries and findings from Project Information Literacy suggest there is still plenty of work to do![17] ACRL has also created a suite of information literacy outcomes[18] to guide the design and evaluation of library instruction programs. Numerous methods have been used to assess information literacy skills including fixed-choice tests, analysis of student work, and rubrics.[19]

It is tempting to rely solely on student achievement of information literacy skills to demonstrate library contributions to student learning. However, a recent review of regional accreditation standards for four-year institutions suggests there is uneven support for doing so. Laura Saunders found three of six regional accreditation agencies specifically name information literacy as a desired outcome and assert the library’s prominent role in information literacy instruction and assessment of related skills. [20] Others rarely use the term “information literacy” in their standards. Instead, competencies such as “evaluating and using information ethically” appear in these standards as general education outcomes to be taught and assessed throughout the college curriculum. In part, I think this reaffirms for us that many in higher education associate information literacy outcomes with general education outcomes such as critical thinking.

While it may be encouraging for information literacy outcomes to be integrated into the college curriculum, I think this poses real difficulties when we attempt to isolate library contributions to these outcomes. If information literacy and critical thinking skills are inter-related, how are we to assess one set of skills, but not the other?  Heather Davis thoughtfully explored this issue in her post “Critical Literacy? Information!” finding that these competencies are intricately related and it is extremely difficult to teach (and assess) them independent of one another. If information literacy skills are taught across the curriculum, when, where, and by whom should they be assessed?  Where does faculty influence stop and library influence begin?

Information literacy outcomes are integral to undergraduate education, but these are not the only learning outcomes that stakeholders are interested in. And information literacy is not the library’s sole contribution to student learning.

I believe we need to shift course in our assessment practices and tackle ‘head on’ the challenge of connecting library use in all its forms with learning outcomes defined and assessed in courses and programs on college and university campuses. We should also link our efforts to the learning outcomes frameworks used in the broader academic enterprise. Broadening our perspective will provide a better return on our assessment dollar.

Where to begin

We can improve our ability to detect library impact on important student learning outcomes by carefully choosing our units of observation. Fortunately we can look to the literature of higher education assessment for clues.[21] My conclusion is that a one-size-fits-all approach to assessment is not likely to work for higher education or for library impact. Instead, our instruments should respect differences in students’ experiences. We should focus on the ‘high-impact’ activities in which faculty expect students to demonstrate their best work. Capstone experiences and upper level coursework within the academic major seem to fit the bill for four year institutions.

The academic major

Students majoring in the arts and humanities, the sciences, and the social sciences acquire different bodies of knowledge and learn different analytical techniques. We also know that learning activities, reward structures, and norming influences vary by discipline. This suggests the academic major plays a significant role in shaping expectations for student learning outcomes and the pathways by which they are achieved.[22] Shouldn’t student information behaviors vary by academic major as well? Our assessment tools should be sensitive to these differences.

The capstone experience and upper level coursework

Capstone courses are culminating experiences for undergraduate students in which they complete a project “that integrates and applies what they’ve learned.”[23] I think we should be studying information use during these important times for several reasons. First, there is ample evidence that the time and energy students devote to college is directly related to achieving desired learning outcomes.[24] Students who work hard learn more. Furthermore, students exposed to high-impact practices such as capstone experiences are more likely to engage in higher order, integrative, and reflective thinking activities.[25] Finally, there is strong evidence that student learning is best detected later in the academic career.[26]

If faculty expectations are at their highest and student effort is at its peak during the capstone experience and in upper-level coursework, shouldn’t studying student information behaviors during these times yield valuable data about library impact?

Speaking the language of learning outcomes

Assessing information use during upper-level and capstone coursework in the academic major is only part of the puzzle. We also need to link library use to student learning outcomes that are meaningful to administrators and policy-makers. I’d like to share two frameworks for student learning outcomes which I think hold great promise.

The Essential Learning Outcomes and the VALUE Rubrics

The Association of American Colleges & University’s (AAC&U) Liberal Education and America’s Promise (LEAP) project[27] defined fifteen ‘Essential Learning Outcomes’ needed by 21^st century college graduates such as critical and creative thinking, information literacy, inquiry and analysis, written and oral communication, problem solving, quantitative literacy, and teamwork. These outcomes are applicable in all fields and highly valued by potential employers. A companion AAC&U project called VALUE (‘Valid Assessment of Learning in Undergraduate Education’) generated rubrics that describe benchmark, milestone, and capstone performance expectations for each outcome. These rubrics are intended to serve as a “set of common expectations and criteria for [student] performance” to guide authentic assessment of student work and communicate student achievement to stakeholders.[28]

Tuning

Some student learning outcomes are discipline-specific. For instance, one would expect students majoring in chemistry, music, or economics to acquire different skills and competencies. A process called Tuning is intended to generate a common language for communicating these discipline-specific outcomes.[29] First developed as a component of the Bologna Process of higher education reform in Europe,[30] Tuning is a process in which teaching faculty consult with recent graduates and employers to develop common reference points for academic degrees so that student credentials are comparable within and across higher education institutions.[31] Expectations are set for associate, bachelor, and master degree levels. Generic second cycle or bachelor degree level learning expectations as defined by the European Tuning process are noted below. Recent work funded by the Lumina Foundation has replicated this work in three states to test its feasibility in the U.S.[32]

Subject-specific learning expectations for second cycle graduates[33]

• Within a specialized field in the discipline, demonstrates knowledge of current and leading theories, interpretations, methods and techniques;
• Can follow critically and interpret the latest developments in theory and practice in the field;
• Demonstrates competence in the techniques of independent research, and interprets research results at an advanced level;
• Makes an original, though limited, contribution within the canon and appropriate to the practice of a discipline, e.g. thesis, project, performance, composition, exhibit, etc.; and
• Evidences creativity within the various contexts of the discipline.

The VALUE rubrics are currently being evaluated in several studies[34] and colleges and universities have begun using them internally to articulate and assess student learning outcomes.[35] While the Tuning process hasn’t yet ‘taken off’ in the U.S., the Western Association of Schools and Colleges recently announced a new initiative to create a common framework for student learning expectations among its member institutions.[36] As colleges and universities experiment with and adopt these frameworks, we should incorporate them into our library assessment tools.[37]

New tools for generating convincing evidence of library impact

As part of my doctoral research I created the Understanding Library Impacts (ULI) protocol, a new suite of instruments for detecting and communicating library impact on student learning outcomes. The protocol consists of a student survey and a curriculum mapping process for connecting library use to locally defined learning outcomes and the VALUE and Tuning frameworks discussed above. Initially developed using qualitative methods[38] the protocol has been converted to survey form and is undergoing testing during 2011. I illustrate how it works with a few results from a recent study.

A pilot project was conducted during the spring of 2011 with undergraduate history majors at two institutions in the U.S., a liberal arts college and a liberal arts university. Faculty members provided syllabi and rubrics regarding learning objectives associated with researching and writing a research paper in upper-level and capstone history courses. History majors completed the online ULI survey after completing their papers.

First, students identified the types of library resources, services, and facilities they used during work on their research papers, including:

• Electronic resources, such as the library catalog, e-resources and databases, digitized primary sources, and research guides.
• Traditional resources, such as books, archives, and micro-formats.
• Services, such as reference, instruction, research consultations, and interlibrary loan.
• Facilities and equipment, such as individual and group study space, computers, and printers.

The forty-one students who participated in the pilot project collectively reported 590 types of library use during their capstone projects ranging from e-journals, digitized primary sources, books, archives, research consultations, and study space. Electronic resource use dominated, but traditional resources, services, and facilities made a strong showing.

Students then identified the most important e-resource, traditional resource, service, and library facility for their projects and when each was found most useful.  At one study site over 60% of students said library-provided e-resources were important when developing a thesis statement.  And 90% of students said both library-provided e-resources and traditional resources were important when gathering evidence to support their thesis.  Over 80% of this cohort reported library services were important during the gathering stage.  These services included asking reference questions, library instruction, research consultations, and interlibrary loan.  These data help link library use to learning outcomes associated with capstone assignments and to the VALUE and Tuning frameworks.

Students reported next on helpful or problematic aspects of library use. For instance, students at both study sites extolled the convenience of electronic resources and the virtues of interlibrary loan, while several complained of inadequate quiet study space and library hours. Information overload and ‘feeling overwhelmed’ were also frequent problems. Time savings and ‘learning about sources for my project’ were mentioned often in regard to library services.

A series of open-ended questions ask about a challenge the students faced during the project. Almost fifty percent of the student-reported challenges were related to finding and evaluating sources and almost as many were related to managing the scope of the paper and issues with writing. Faculty and librarians can ‘drill down’ into these rich comments to understand challenges students face and shape collaboration faculty-librarian collaboration to meet the needs of future student cohorts.

Open ended questions also elicit powerful stories of impact. When asked what she would have done without JSTOR, one student replied:

“I honestly have no idea. I may have been able to get by with just the books I checked out and Google searching, but those databases, JSTOR specifically, really helped me.”

I hope these glimpses of recent pilot study results demonstrate the value of focusing our attention on important and memorable academic activities in students’ lives. Using both quantitative and qualitative methods helps us understand how and why libraries support students when the stakes are highest. Authentic user stories coupled with links between library use and student learning outcomes serve as rich evidence of library impact to support both advocacy efforts and internal improvements.

Future uses

The Understanding Library Impacts protocol is not designed to assess student learning; teaching faculty and assessment professionals fulfill this role. The protocol is intended to link library use with existing assessment frameworks. ULI results can then be used in concert with other assessment data enabling new partnerships with teaching faculty and assessment professionals. For example:

• The AAC&U Essential Learning Outcomes map well to general education outcomes at many colleges and universities.  The protocol’s use of the VALUE rubrics creates a natural vehicle for articulating library contributions to these outcomes.
• Understanding Library Impacts results may also integrate with third-party assessment management systems (AMS). As Megan Oakleaf noted in the Value of Academic Libraries Report, integrating library assessment data with AMSs allows the library to aggregate data from multiple assessments gathered across the library and generate reports linking library use to a variety of outcomes important to the parent institution.[39]

It is critical to find ways to connect library use in all its forms with learning outcomes important to faculty, students, and stakeholders. Doing so will bring the library into campus-wide conversations about support for student learning.

Thanks to Ellie Collier, Hilary Davis, and Diane Harvey for their comments and suggestions that helped shape and improve this post.  Thanks also to Hilary and Brett Bonfield for their help preparing the post for publication. I also want to thank the librarians, faculty members, and students at the study sites for their support and participation in this pilot study.

Courtesy of Flickr member cambodia4kidsorg

“The indicators that served as benchmarks in the past, such as number of volumes and number of journal subscriptions, are no longer sufficient because of the more expansive role that the contemporary library has assumed” (Weiner, 2005).

“The measurement of quality will come back to the questions of who are the users, what are the inputs, what are the outputs, do we produce the outputs in a way that meets the needs of the users, and what do those outputs contribute to the productivity and accomplishments of those users?” (Pritchard, 1996).

It’s almost a sure bet that your friends, family and colleagues are looking at ways to save money and, in general, are tightening the purse strings a little more these days. The post on January 14, 2009 in this blog reviewed the state of libraries during recessions and pointed out the growing news pieces that remark at the huge surge in library usage. People are realizing real savings by relying more and more on libraries. Ask yourself how much do you spend at bookstores and music shops such as Amazon and Barnes and Noble each year? Magazine subscriptions? Internet service? Entertainment like movies and concerts? If you had to go without one or more of those services, think about how much you could save by relying on your local library to provide access to those services and content streams. At the same time, libraries of all types are faced with the inevitability of budget cuts due to the recession and must justify the use of existing funds for programming, staff, services and collections. Take a second and google ‘return on investment and libraries’ to get a sense of the importance of demonstrating library value.

In order to have the financial ability to continue providing those services and content streams, libraries need to prove to their funding sources, whether tax-payers, private donors, universities, governments, schools, or corporate parents, that those services, programs and collections are meeting users’ needs. Moreover, libraries must prove without a doubt that the funds provided to libraries to develop those services, programs and collections provide a good return on investment.

While there are many metrics for assessing library value (e.g., LibQual, circulation trends, gate counts, usage statistics trends, ARL Annual Statistics, etc.), this article aims to explore the return on investment (ROI) approach used by libraries to demonstrate value.

What is Return on Investment (ROI) and how is it used by libraries?

Return on investment (ROI) is how much you get back for what you put into something. Strictly speaking, ROI is based on dollars and cents. So, you need to be able to quantify how much money was invested in something and then you need to compare how much money is gained or lost as a result of how the investment was handled. There are two kinds of questions that ROI is good at answering. One is: how much money will be gained by investing in a particular financial asset? The other is: will putting resources into a project or service yield a measurable benefit? Let’s take a look at a quick example using a baseball card. If you bought a baseball card in 2000 for $50 and now in 2009, it’s worth $500, what is your return on investment?

In libraries, ROI is measured in many different ways. ROI can be used to measure the costs (investment) and the outcomes (the return on investment) from the perspective of library users, the parent organization, or from the perspective of the library itself. Costs are typically dollars spent on a service or resource and/or time spent to provide or access a service or to acquire or use a resource. The returns on an investment can be either outputs (the result of a service or resource such as expanded journal collection), uses (how the service or resources are used), or outcomes (indirect results of the output or the use such as time saved) (Jose-Marie Griffiths, 2007).

Motivations for using ROI in libraries

ROI can be an integral part of the process for evaluating a library’s services, collections, staffing levels, planning for new services and resources, or measuring how valuable your library is to your community and stakeholders.

For example, for libraries supported with public tax dollars, one way to use ROI is to measure tax dollars (the investment) against the benefits (savings by not having to pay elsewhere for the use of library materials and services). Library ROI studies consistently suggest that public libraries give a high return on investment, providing anywhere from $2 to $10 in return for every tax dollar received.

• In Florida: for every $1.00 of taxpayer dollars spent on public libraries, income (wages) increases by $12.66
• In South Carolina: In return for an investment of $77.5 million, public libraries pump $347 million into the state’s economy
• Can you say your library users derive more than $4.00 in benefits for every $1.00 spent of taxpayer money? St. Louis Public Library can!

[From Trustee Blog: Library ROI - What's Your Community rating]

Strategies for measuring ROI in libraries

Calculating a return on investment may seem straightforward until one considers the kinds of costs and returns associated with libraries. Measuring returns first requires that the organization have a clear sense of its mission and objectives. It is not possible to measure benefits unless one can identify the value an organization aims to provide. There are several classes of returns, direct and indirect, and individual and collective. In general, direct, individual benefits are easier to measure and quantify than indirect and collective benefits. This poses a challenge for libraries, as many of the benefits libraries provides are indirect and collective, such as the value of having a better-educated citizenry.

Jose-Marie Griffiths has conducted several return on investment studies in public and special libraries. In a presentation at the Special Library Association Annual Conference (2007), Griffiths outlined a method for calculating ROI in special libraries that demonstrates the kinds of factors that should be taken into consideration. Costs are generally straightforward, and include overhead and the costs of the users’ time associated with utilizing the library, but returns are trickier. Griffiths argues that libraries should use contingent valuation for assessing benefits. Contingent valuation is a method for evaluating goods and services that are not priced. It involves assessing the effect of taking the service away. This could mean attempting to calculate the costs that would be borne by users if they could not use the library. Griffiths also notes that changes in productivity and information needs that would go unanswered should also be considered.

Using ROI, libraries can try to place a value on the services they provide and the collections that they make accessible. For instance, many ROI studies compare the cost associated with borrowing an item from the library versus individuals having to purchase that item on their own. Consider the costs associated with the library providing a DVD that is worth $20 that circulates 50 times in a year. If each library user had to purchase that item, the collective cost would have been $1,000. Of course, there are additional costs borne by the library for providing the DVD than just the $20 investment, including staffing, storage, and preservation.

Courtesy of Flickr member allaboutgeorge

Libraries can also be valued in terms of savings of entertainment costs to a community—public film viewings, author readings, and workshops are freely offered services provided by libraries. Many libraries also offer classes, which can be viewed as a cost savings to the community as well. Classes on Microsoft Office programs, general computing, financial planning, and job hunting strategies are often offered for free at libraries. Libraries are also valuable to communities as employers of citizens and as contributors to the local economy.

Examples from different library contexts

Public libraries

Most ROI studies in libraries have focused on public libraries. A fantastic inventory and review of value-demonstration methods and metrics is available from the Americans for Libraries Council: Worth their Weight: An Assessment of the Evolving Field of Library Valuation (2007). We will highlight a few examples here.

The “Public Library Benefits Valuation Study” conducted by the St. Louis Public Library in 1999-2001 used cost-benefit analysis techniques. First, they calculated a comparison between local taxes invested in library services and direct benefits provided to users. Across the five urban, public libraries included in the study, for each $1 of annual taxes invested in the library, library users received between $1.30 and $10 in benefits. This varied widely among the libraries in the study. Second, the study looked at returns in terms of capital investment. This compares the total investment in a library’s capital (buildings, vehicles, furniture, and other assets) with the benefits received by users. Annual returns on capital investments ranged from 5% to 150%, again varying widely among the public libraries in the study.

The State Library and Archives of Florida conducted a taxpayer return on investment study of Florida public libraries. Overall, the study found that for every $1 invested in the library at least $6.54 is returned. Other findings show that for every $1 invested in the library the Gross Regional Product increases by $9.08 and wages increase by $12.66. They also estimate that for every $6,488 invested in the library, one job is created. Statewide, the study estimates that the Gross Regional Product is increased by $4 billion as a result of taxpayer investment in Florida public libraries.

Many other public libraries have reported return on investment information, including New York public libraries, Pennsylvania public libraries, South Carolina public libraries, and Wisconsin public libraries, among others. Return on investment results from these studies ranged from $2.38 in Indiana to $5.50 in Pennsylvania. Among the five states with published ROI studies, Indiana, Pennsylvania, South Carolina, Vermont, Wisconsin, and Florida, the average ROI is $4.99 for every $1 invested.

Another approach some libraries have taken is to provide calculators on their websites that let users estimate how much value they are getting from the library based on their own use of its services (e.g., Rhode Island).

A potential hazard of ROI studies is that they produce what appears at face value to be a simple metric that can be compared across libraries.

Courtesy of Flickr member msmail

It is critical to reiterate that the ROI studies that we use as examples specifically state caveats that those ROI metrics are estimates that are based on surveys of their own local users combined with metrics that are relevant to their own budget systems. Any attempt to compare ROI metrics across these boundaries doesn’t make sense and is not relevant. The new LJ Index, while not specifically ROI-focused, attempts to correct the problem of peer comparison by removing the metrics that are specific to local contexts. Instead the LJ Index focuses only on measurable outputs such as circulation per capita, visits per capita, program attendance per capita, and internet use per capita. The LJ Index is one of several models for ranking public library quality, and is unique in that it enables statistically valid comparison across libraries. However, it removes the context-sensitive framework that enables libraries to show a return on investment of money and resources.

For ROI library metrics, the point isn’t that putting more and more money into libraries will yield ever increasing returns. The point is to show that libraries are providing value for the money that is invested in them. Those investments should be commensurate with the needs of the communities they serve.

Academic libraries

University libraries have fewer models to emulate. In 1996, Sarah Pritchard described the problems associated with assessing the value of academic libraries. She claims that the lack of standards and repeatable methods for demonstrating value (such as support of accreditation reviews, educational assessment and outcomes, ranking of graduate programs, success of job attainment after graduation, success in attracting donors, and faculty research productivity as measured by grants and publications) makes it impossible to conduct studies that compare library value across institutions.

A recent study (Luther, 2008) at the University of Illinois at Urbana-Champaign addresses some of Pritchard’s concerns about demonstrating ROI in academic libraries. Unlike public libraries, which focus on the value of services, the Illinois study examines the library’s contribution to revenue-generating activities of faculty by examining the role of library-sourced citations in grant applications. The model for the study is that the library’s investment in materials increases researchers’ productivity. This increase in productivity produces a measurable increase in grant receipts due to increased citations, as well as recruitment and retention of productive faculty.

To calculate the dollar value returned to the University of Illinois in the form of grants due to investment in the library, the faculty was surveyed to determine the importance of citations in securing grants, the percentage of faculty who use citations in grant applications, and the percentage of citations obtained through library-subscribed resources. The model also accounts for the proportion of grants that are funded. The study found that for every $1 invested in the library, $4.38 in grant income is generated for the university. This model purposefully avoids attempting to measure the social value or increases in productivity attributable to outcomes from use of the library resources.

Special libraries

Special libraries such as those found primarily in the government and corporate sectors tend to focus their ROI metrics on time saved for employees by using library resources and expertise, increases in revenues, decreases in research and development expenses, productivity gains, and cost savings. Roger Strouse, Director of Outsell, Inc., writes extensively about the value and application of ROI studies for special libraries. Outsell conducts studies of market trends in the publishing, education, and information industries. In its 2007 study on corporate, government, and medical libraries, Outsell found that the average time saved for users was 9 hours per library visit/interaction. They also reported that not only do corporate libraries save $3,107 per use of library resources and services, but also that $6,570 worth of revenues were generated with the aid of library resources and services.

“Being good at what you do and at the services you provide is no longer good enough. Very good information centres will be cut, and may be outsourced or offshored, not because of their inability to provide good services, but because of their inability to demonstrate an ROI or provide evidence of the impact they make on their organizations” (Boyd Hendriks and Ian Wooler, 2006).

Outsourcing remains a serious threat to many special libraries. Integrating and aligning the work of special libraries with the risks associated with the parent organization is one of the key recommendations of corporate library strategists. Collaborating on the reduction of risk, delays, and workplace injuries is seen as a way to position a library as a value-added partner and a key component in the success of an organization. Strouse (2003) provides an example survey for special libraries to use in measuring ROI that includes questions about types of projects for which library services and resources were used such as patents, new technologies, new product acquisitions, and changes in marketing strategies. The Special Libraries Association maintains a summary of articles and presentations on ROI metrics and value-demonstration strategies (the full bibliography is available to SLA members).

A Few Caveats

There are some reasons why ROI might not be the best tool for demonstrating library value. In some cases, a strict ROI metric may demonstrate that a library is not providing a good return on investment. Elliott, et al. (2007) describe the pros and cons of conducting an ROI or cost-benefit analysis. Many of the benefits they describe are covered earlier in this article. However, one of the disadvantages of ROI or cost-benefit analyses described by Elliott, et al. is that these metrics cannot be used for peer-comparison. The metrics are created using value systems and context-sensitive data that pertain to individual libraries. Pritchard (1996) echoes this problem in the realm of measuring academic library quality and effectiveness:

“The difficulty lies in trying to find a single model or set of simple indicators that can be used by different institutions, and that will compare something across large groups that is by definition only locally applicable—i.e., how well a library meets the needs of its institution.”

Elliot, et al. (2007) also warn against applying these kinds of metrics to small libraries: “Efficient operating costs do not appear to rise proportionally with cardholder population and collection size. Thus, benefit-cost ratios for well-managed larger libraries tend to be higher, in general, than those for well-managed smaller libraries. Larger libraries also are more likely to be able to accommodate the expense and technological requirements associated with a CBA [cost-benefit analysis] study.”

There are more subtle reasons to not rely on ROI metrics alone, and to be careful about interpreting ROI. Organizations need resources to survive. Not-for-profit organizations, whose missions are based on soft values or moral ideas rather than monetary profit, must be supported by private donations, government, or by the organizations that they support. The values of the library—ubiquitous access, preservation, and organization of information—are prone to differing interpretations of importance. Put bluntly, the library must show that the Internet has not rendered it obsolete. Libraries will be stronger if they can demonstrate their value in terms which those that provide its funding understand. In the culture and time in which we live, “value” is understood most readily in monetary, economic terms.

Making it even more difficult for academic libraries to demonstrate their worth, the mission of the library is tied to the mission of the university at large. Academic libraries must demonstrate their contributions to the mission of the organization of which they a part. And they must also attempt to make a long-term argument about preservation of information and investment in human capital to an audience that is focused on the present bottom line. Can we articulate the importance of what we do in terms that non-librarians can understand? If we cannot articulate what we are doing then we must first go back and redefine what our mission and purpose is, clearly and succinctly, before we can attempt to measure our effectiveness and value.

It is, however, essential to remember that there is a reason why libraries do not operate for a profit (with the possible exception of libraries that charge back to users for services), and that they came into being for reasons other than generating monetary wealth. There are also good reasons to be skeptical of measurements of library quality, performance, and relevance presented in purely economic terms. Discussing the role of non-profit organizations in general, Eikenberry claims:

Because of their inherent value, it is extremely important for nonprofit organizations to focus on their organizational missions…They are more than just tools for achieving the most efficient and effective mode of service delivery; they are also important vehicles for creating and maintaining a strong civil society. (Eikenberry)

Libraries must strike a balance between focusing on their mission and on their desire to prove worth in terms of high performance and ROI. For many libraries, ROI simply doesn’t measure the indirect benefits they provide.

Librarians must be particularly creative in the ways that they think about how their libraries perform and what they contribute to the populations they serve. For example, does having an appealing library make a university more appealing to potential students? One study suggests that libraries have a significant influence on students’ decisions to go to a particular university—53%—second only to “Facilities for Major” (e.g., labs, studios, etc.) at 73%.

U.S News and World Report’s rankings of colleges and universities, has transformed the way students select schools. Traditional ROI studies do not account for a library’s impact on the reputation of its university or college as a whole, but a study by Weiner in 2009 makes a case for the library’s contribution to the reputation of the university it serves. The study finds that library expenditures are a significant predictor of institutional reputation.

Weiner argues that this finding means that, in fact,

“the disintermediation caused by the rapid increase in online access to information does not seem to have displaced the library. These results suggest that libraries in the doctoral institutions included in the study may have adjusted and found solutions to unprecedented external pressures.”

Weiner sees the libraries’ position as one of “boundary spanning,” meaning that it brings together researchers and students from across the campus in ways that no other organization on the campus can. Studies such as this one are vital for libraries, as they give evidence that libraries contribute to their parent organizations in unexpected ways. By contributing to an institution’s overall reputation, libraries also impact real economic outcomes from that reputation: such as attracting better students, retaining top-notch faculty, and attracting donors. Being able to articulate this kind of impact may help university administrators listen a little more closely.

It is vital that libraries demonstrate both the monetary value and as well as the social value of their services. ROI is one part of a suite of tools librarians can use to demonstrate performance and value. Relying on ROI alone to communicate and demonstrate the value of libraries may very well undermine the core purposes libraries serve and the indirect benefits they bring. Libraries undertake many tasks that are invisible to the casual user. They handle licensing of journals, negotiating with vendors and publishers for access to content, selecting resources. Libraries also contribute to the prestige of the institutions they serve by helping to attract top researchers, faculty, and students. Academic libraries and public libraries, especially, serve as unique third places within their communities, where people who would otherwise not interact come to work and learn in the same space. It’s up to us to convince our users and our sources of funding that we’re worth it. ROI studies aside, one of the best things we can do to show our worth is to provide great services that help our users work more effectively.

We hope this post will generate some lively discussion about the role of ROI in libraries, and also generate ideas about what libraries can measure to demonstrate their value. What has your library done to measure its value?

Thanks especially to Brett Bonfield, Greg Raschke and Katie Wheeler for their comments and for reviewing and editing various drafts of this article.

Conferences, Seminars, Upcoming Events on ROI and Libraries

• Computers in Libraries – Track E (Planning and Managing): What’s the Return on Investment for Your Library? April 1, 2009
• ALA Annual Conference Surviving in a Tough Economy – An Advocacy Institute Workshop July 10, 2009

Further Reading

• Abram, Stephen. 2007. The Value of our Libraries: Impact, recognition, and influencing funders: http://www.sirsidynix.com/Resources/Pdfs/Company/Abram/ArkansasLA_Value.pdf
• ALA. 2009. Articles and Studies Related to Library Value (Return on Investment): http://www.ala.org/ala/aboutala/offices/ors/reports/roi.cfm
• Cain, David and Gary L. Reynolds. 2006. The Impact of Facilities on Recruitment and Retention of Students. Facilities Manager, March/April: http://www.appa.org/files/FMArticles/fm030406_f7_impact.pdf
• Eikenberry, A.M. and J.D. Kluver. The Marketization of the Nonprofit Sector: civil society at risk? Public Administration Review, vol. 64 (2): 132-140.
• Elliott, Donald S., Glen E. Holt, Sterling W. Hayden, Leslie Edmonds Holt. 2007. Measuring Your Library’s Value: How to Do a Cost-Benefit Analysis for Your Public Library. ALA Editions (978-0-8389-0923-2)
• Hendriks, Boyd and Ian Wooler, 2006. Establishing the return on investment for information and knowledge services: A practical approach to show added value for information and knowledge centres, corporate libraries and documentation centres. Business Information Review, v. 23 (1): 13-25.
• Holt, Glen E., Donald S. Elliott, Leslie E. Holt, Anne Watts. 2001. Public Library Benefits Valuation Study (St. Louis Public Library): http://www.slpl.lib.mo.us/libsrc/valuationc.htm
• Imholz, S. and Arns, J.W. 2007. Worth Their Weight: An Assessment of the Evolving Field of Library Valuation. Americans for Libraries Council: http://www.bibliotheksportal.de/fileadmin/0themen/Management/dokumente/WorthTheirWeight.pdf
• Lance, Keith Curry and Ray Lyons. 2008. The New LJ Index: Why these measures matter. Library Journal, http://www.libraryjournal.com/article/CA6566452.html
• Luther, Judy. 2008. University Investment in the Library: What’s the Return? A Case Study at the University of Illinois at Urbana-Champaign: http://libraryconnect.elsevier.com/lcn/0601/lcn060103.html
• O’Hanlon, Nancy. 2007. Information Literacy in the University Curriculum: Challenges for Outcomes Assessment. portal: Libraries and the Academy 7, no. 2: 169-89.
• Pritchard, Sarah M. 1996. Determining Quality in Academic Libraries. Library Trends, vol. 44 (3): 572-594.
• SLA. 2009. Additional Value Resources: http://www.sla.org/content/learn/members/ipvalue/additionalValue.cfm
• Strouse, Roger. 2003. Demonstrating Value and Return on Investment: The Ongoing Imperative. Information Outlook, v. 7 (3): 15-19. http://findarticles.com/p/articles/mi_m0FWE/is_3_7/ai_99011610
• Strouse, Roger. 2007. ROI for libraries remains high: http://www.outsellinc.com/store/insights/3538
• Weiner, Sharon Gray. 2005. Library Quality and Impact: Is there a Relationship between New Measures and Traditional Measures? Journal of Academic Librarianship, 31(5):432-7.
• Weiner, Sharon. 2009. The Contribution of the Library to the Reputation of a University. Journal of Academic Librarianship, vol. 35 (1): 3-13.
• Yackle, Anna. 2007. What is Your Library’s ROI? (NSLS): http://www.nsls.info/articles/detail.aspx?articleID=137