The Yahoo! Search team by Yodel Anecdotal (explored on Sep 16, 2009) / CC-BY

As it turns out, librarians aren’t the only ones competing with Google. In fact, we’re not even the only ones offering an alternative to Google when it comes to helping people find information.

There’s Microsoft’s Bing, of course. And Yahoo! Search, at least until 2012, when Bing will begin providing Yahoo’s search results (though some testing has already started). Combined, Microsoft and Yahoo! provide about 30% of the search results in the United States, but only roughly 10% of the search results overall; Google, at 63% U.S. and 85% overall pretty much owns search.

Google’s dominance is one of the reasons many people get excited about alternative search engines. Choice is important, especially in something as important as access to Web-based information, and so is competition, which often leads to innovation. There’s often excitement leading up to the introduction of well funded and reputedly innovative search engines, such as Powerset (quickly acquired by Microsoft) and Cuil, both of which debuted in 2008, and Blekko, which is currently in closed private beta, but earned a positive review from Michael Arrington at the influential TechCrunch. Innovation in search is a good thing for many reasons, not least of which is the issue Paul Ford recently called, “the Barnes & Noble problem”:

Until I was about 26 almost everything I wanted to read was in Barnes & Noble. Eventually they had less and less of what I wanted. Now B&N’s a place I go before a movie, and I get my books anywhere else. I’m increasingly having B&N moments with full text search ala Google. It’s just not doing the job; you have to search, then search, then search again, often within the sites themselves. The web is just too big, and Google really only can handle a small part of it. It’s not anybody’s fault. It’s a hard, hard problem.

It’s possible that many ways exist to avoid the Barnes & Noble problem in Web search, but the two ways most companies seem to be trying at the moment are represented exceptionally well by Endeca and DuckDuckGo. Endeca, which provides search for Borders, Walmart, Home Depot, and many other large corporations and institutions (as well as North Carolina State University Libraries), will “guide users through asking and answering any question;” DuckDuckGo tries to out-google Google by adding features people want, removing annoyances, and finding out what’s working by engaging its users in a fun, ongoing conversation about their interests.

I recently had a chance to interview Endeca co-founder, Pete Bell, and DuckDuckGo founder Gabriel Weinberg about their companies and their thoughts on search.

Both of your companies provide search for specialized collections. Do you believe that people want a single, universal interface that will work everywhere or do they want an interface that’s been built to suit the collection they’re using?

Gabriel: I think that vertical search engines can work if they are compelling enough, e.g. Kayak, which aggregates prices on airline tickets, hotel rooms, car rentals, and helps people find good deals on travel. However, there are only so many verticals where they can be compelling due to business model, i.e. high transaction value.

In general, I believe people want the “single, universal interface that will work everywhere.” At DuckDuckGo, I have a longer term goal to help people navigate towards vertical engines that may be better for them. I’m doing this currently in a completely self-selected basis via !bang syntax.

If you look at each vertical, there is usually a search engine out there that produces better results than Google for that vertical. But no one is going to go to each of these hundreds of sites in specific situations.

Pete: Good experiences are always designed around tasks—around specific users searching for specific content. And I’m using the word “search” to mean much more than the search box—I’m talking about all the navigation, visualizations, and content that helps people find what they need. Now, if you ask people what they want, they’ll say they just want a Google box. But if you test that against a task-built experience—say, image search at Jupiter Images—they’ll overwhelmingly pick the latter. Marti Hearst tested a great example of this as part of her Flamenco Search Interface Project on faceted search User Interfaces (UIs).

Is it important for search interfaces to match the way people think or will people adjust their thinking to suit search interfaces?

Pete: There’s a difference between zero-training and easy-to-use. Zero-training means it has to match the way people think, and for any popular public-facing website, it has to be fluid. On the other hand, there can be easy-to-use sites that take a few minutes to learn. They better become fluid after those few minutes though. For example, we’ve built some search applications for manufacturers that give their design engineers thousands of facets. They’re willing to spend a couple of minutes to orient themselves to get power-user features. First time I switched from a PC to a MAC, I was surprised that there was still a learning period, but it faded fast.

Gabriel: If you want fast, low-cost adoption, I believe the interface should be as fluid and simple as possible. However, sites like Amazon have proved that you can push through User Experience (UX) with enough money. By which I mean basically what Pete said, in that if you are allowed to train people for a few minutes then you can end up with a better UX overall. Amazon has done it essentially via brute force, i.e. push through by simply being around long enough that people end up spending those few minutes over time.

How do you weigh precision versus recall? Has your thinking changed along the way?

Gabriel: I’ve been pretty much about precision from the beginning, in part because I rely on external APIs for the long-tail; that is, for less popular searchers, I rely mostly on the raw search results I get from Bing API 2.0 and Yahoo! “Build your Own Search Service”. I think my value-add for those types of queries is in added precision. But more generally, there are just so many Web pages out there and people don’t look at many of them (they choose from just the top few results), so precision is most important for general search. For specialized search I think it can reverse depending on the vertical.

An example of a vertical in this context would be searching for bug reports. There are usually very few pages out there that have the exact output of your bug report, and if they exist, you want to find them. For things like that, we rely on Yahoo & Microsoft to have crawled those pages. For less specific queries we layer on top of those APIs some Neuro-Linguistic Programming (NLP) stuff that, among other things, tries to extract the concepts/entities in the query and gives you pages more associated with them. For other queries where we know a vertical engine will give you better info, e.g. weather or complicated math, we will automatically query an API and display the better results—I think this is another form of recall.

Pete: You can cheat the precision vs. recall trade-off. At Endeca, we’ve become disciples of the Human Computer Information Retrieval school, and all that Gary Marchionini and Daniel Tunkelang have done to popularize the HCIR model.

When we started, it was orthodoxy that there was a trade-off between precision and recall. That assumes people make a query into a black box, get back a ranked list of results, and then either accept one of those top results or recompose their query. It’s the TREC evaluation model. But ranking is dubious—it conflates many dimensions of relevancy into a single score.

With HCIR, there is no strict trade-off between recall and relevancy. Instead, you engage the user in a multi-step “conversation” with the data, as in a faceted search. You start with a probe query that returns a set of results. And then the system characterizes the set—it tells you the attributes and facets associated with that set. That helps you refine to a subset, then lather, rinse, repeat. The trick is to treat search as a set retrieval problem instead of a ranked list retrieval problem.

For example, if your task were to find a photo of dogs with kids to illustrate a book jacket, and all you had was a classic search box, you’d probably maximize for recall with some searches like “dogs kids jpg” or “dogs children photos” and then eyeball the results. But with HCIR, the system has a chance to teach you about the results. Back to Jupiter Image search, we could search for “dogs,” and then discover facets about ages, concepts, and image technique, and use those to whittle down. You’re returning a set of results, and then learning about subsets. The effect is that you get unexpected results that you could never hope to discover with keywords.

What usability testing methods do you find most informative?

Pete: Agile testing is best. Make mistakes often and learn from them quickly. I’m with Jared Spool—you can learn a lot, inexpensively, by testing a small set of people and iterating.

Gabriel: I find natural feedback coming through the site to be most informative. Often this kind of feedback comes from users who have put in a lot of thought. I’ve also found Reddit comments from ads to be particularly informative, especially for first impressions. Finally, I’ve gotten use out of PickFu. I have plans to investigate usertesting.com and feedbackarmy.com as well, but haven’t done so yet.

Can you expand on “natural feedback”? And how you’ve used Reddit and PickFu?

Gabriel: By natural feedback I mean feedback that flows from real users using your site. On DuckDuckGo, there is a feedback button on every search result page (in the lower right corner). Most of our feedback comes through there and is in a “natural” context of searching for something particular.

I posted a PickFu review on my blog. Basically, it is good way to get quick opinions on two choices. People vote which one they like better, but more importantly they give you their take on why, which provides some insight into what people were thinking.

Reddit is more straight advertising, but with each ad there is also a comment thread. Reddit users are known to actually check out things and report back in comments, and they luckily do this for Reddit’s ads as well. But that’s not all, because you can actually engage with Reddit users as well, and have conversations about your product. All in all, it is a great feedback experience.

Guest question for Gabriel (courtesy of Andrew Nagy¹): “How are you positioning DuckDuckGo differently from Google in terms of user interface and user experience? What sort of new UI concepts are you evaluating that Google is not already doing?”

Gabriel: On a feature level, our about page attempts to answer this question directly: But at a higher level, I’m trying to make DuckDuckGo results pages more readable and understandable. A lot of the features are in this vein. For example, I put Zero-click Info on top, which is readable topic summaries (sometimes full paragraphs) from crowd-sourced sources like Wikipedia and Crunchbase. Other examples are labeled official sites, human-edited link titles and descriptions (also from crowd-sourced sources), disambiguation pages, and fewer useless sites in our results pages. Another angle is discovery. I provide related topics (as opposed to related searches) and category pages, which are groupings of topics of a similar theme.

How closely do you think profitability aligns with quality? In evaluating your competition, do you get the sense that it’s the better engineered search products or the better run businesses that are succeeding?

Pete: Just to set the context for Endeca, in our market, our customers want to customize a search experience for their specific users and content. There’s a healthy market for one-size-fits all sites generated by inexpensive appliances, but that’s not our market. NCSU, WalMart.com, and ESPN have different experiences from each other. We call these search applications.

There are a few ways to go about that. You could invest many, many, many millions on in-house developers, like Amazon and eBay did. But our customers choose the platform route—they’re buying Endeca’s “Legos,” and partnering with our services team to design their site.

Now, that’s a complex project. It brings together teams from two companies that haven’t worked together before. And it mixes a lot of specialties—user experience, application development, information architecture—that might not understand a lot about each other. My friend Joseph Busch does high-end taxonomy and document management projects, and he likes to joke that he’s 5% a library scientist, 95% a social worker.

People tend to focus on technology when they’re planning a new site. But with projects like these, business process, user experience, support, professional services, education, and so on all matter, too. So to answer your question, in the search applications market, technology is part of it, but execution matters just as much.

Gabriel: I think it is product for the most part, at least for general search and with a few caveats. Google’s share just kept climbing and climbing, and I think that is largely due to its product. Recently, Bing canceled their cashback program after tons of money because it presumably didn’t yield new customers. That’s more evidence of product dominance.

The first caveat is distribution deals. A lot of people use what is in front of them, and sometimes have no choice. It’s very hard (if not impossible) for a startup to capture those distribution deals since Microsoft and Google have so much money behind them.

The second caveat is, without distribution it is very hard to get people to switch search engines. All the recently well-funded search startups who failed are evidence of this fact. I think they didn’t wow people enough in the product, however. But the bar is pretty high.

The third caveat is brand. Google did a study comparing its results with its competitors’ and found a huge implicit trust from using the Google logo at the top. They earned that, but that is additionally hard to overcome for a startup (or even for Microsoft).

What are your thoughts on expert search features, such as specialized syntax or regular expressions?

Gabriel: I’ve been trying to “walk the line” in this arena, by offering specialized syntax that I think could get mainstream support from power users. I think regular expressions are a bit out there for the normal user although I did already incorporate them in some capacity already (though probably not what you meant): http://duckduckgo.com/?q=regexp+/(.*%3F)+(.*%3F)+(.*)/+duck+duck+go.

Something I think more walks that line is the !bang syntax I created where you input !amazon x in the search box and it searches for x in amazon. I think that’s easy to grasp and it is useful. Additionally, I think it can help market to specific groups of users, e.g. I also added hex color codes and unicode query responses.

Pete: You know the rule of thumb that 90-odd percent of users never change the defaults. Whatever the number is, it’s increasing. That said, it’s not fair to round down to zero and say that the few people that do use expert features don’t count. They tend to be some of the most valuable users. We’ve got extensive XQuery hooks into our engine that make it possible to build up some great queries.

What do you think of Wolfram Alpha?

Gabriel: As a collection of cool data that gets aggregated usefully in response to queries, I love it! As a standalone product, however, I worry that it will die for lack of a business model. I think a lot of what they’ve done would be great in a search engine, and I’ve tried to integrate it as much as possible into Duck Duck Go (see Duck Duck Goodies).

Pete: There’s a continuum of search tasks that range from fact finding on one end to discovery on the other. (Fact finding: Who wrote Ulysses? Discovery: Which Irish writer should I read on the beach this afternoon?) Wolfram Alpha is really cool for fact finding, and lousy for discovery. You can’t have discovery without human input—HCIR.

What do you think of WorldCat.org?

Gabriel: I had not heard of it until this moment, so this is a first impression. I’m not the target customer since I haven’t checked out something from a library since college :). But I imagine this could be really useful for people who do check stuff out from libraries, i.e. students, researchers, etc. The implementation seems a bit cluttered and I’m not sure how big that market is. I suppose the business model is clicking through to Amazon or whatever; it’s an empirical question on how much that actually converts.

Pete: I enjoy WorldCat. They’ve done an impressive job on their primary mission. That’s sincere—I’m not damning them with faint praise. But if you want me to focus on search and give constructive criticism, there’s a lot more they could do.

If you hold up some great sites as the bar, you’ll see ideas WorldCat should adopt on user experience, relevancy, text mining, and visualizations. Just to name a couple of sites, IEEE Explore and Food Network both have ideas that could improve WorldCat.

And if you expect OCLC to take a leadership role, they should push the bar on searching digital collections: full text, images, multi-media. We’ve been working with the JFK Presidential Archive on their next generation site to search their digital archives. That’s given me a real appreciation for how big the challenges are on searching digital collections. There’s a lot of work to do, and it would be good to see OCLC start experimenting.

If you’re interested in hearing more from Pete Bell, I recommend his always interesting contributions to Endeca’s excellent Search Facets blog as well as a very good interview with him conducted by Steve Arnold. For more on Gabriel Weinberg, I recommend his superb blog, book (still a work in progress, but we get to follow its development online), and the DuckDuckGo community for educators and librarians.

Thanks to Pete Bell and Gabriel Weinberg for participating in the interview, to Andrew Nagy for his question and his assistance with the article, and to my Lead Pipe colleague, Ellie Collier, for her comments.

1. Andrew Nagy, an open source evangelist and library technologist, joined Serials Solutions in late 2008 where he has been an evangelist for Discovery services and seminal in the development of [Summon] (http://www.serialssolutions.com/summon/). Prior to joining, he was the Technology Development Specialist for the Falvey Memorial Library at Villanova University where he was responsible for developing many innovations, including VuFind , an internationally adopted open-source Library Discovery solution.

Teapots In a Tempest by GaijinSeb / CC-BY-NC-ND

Only a few information technology organizations predict the future by inventing it.¹ One of the canonical examples is Xerox PARC, which in the early 1970′s produced the first mouse, pioneered Graphical User Interfaces, invented Ethernet, and developed the first laser printer, along with dozens of other innovations. Among contemporary organizations, the inheritor of this lineage appears to be Google.

The Graduate Library School at the University of Chicago during its early years is probably the most widely accepted Xerox PARC analog within librarianship. If libraries have a Google equivalent, a contemporary organization that is both synthesizing the best work in the field and shaping its future, it’s North Carolina State University Libraries. Under Susan Nutter’s directorship, NCSU Libraries became the first university library to win the Association of College and Research Libraries’ Excellence in Academic Libraries Award and received the American Library Association’s Library of the Future award; Susan Nutter was Library Journal‘s Librarian of the Year in 2005; and it places someone in LJ‘s Movers & Shakers list pretty much every year. Observe NCSU Libraries from afar and you can’t help but be impressed. Study it up close, as I did two years ago this week, and you get a sense of what it must have been like to work at Xerox PARC or, I expect, what it’s like to work at Google.

Two years ago, I was a library school student enrolled in Steven Bell’s Academic Librarianship course at Drexel University. The major assignment for the class was to conduct a field report on a library, and Susan Nutter allowed me to spend a day interviewing her management team, mostly individually or in small groups. One of the major themes I noticed was how fortunate they felt to work with each other. They believed they were working more hours than their colleagues at peer institutions, but they also believed they were having more fun (in my experience, both beliefs seem to be accurate). As complimentary as they were toward all of their colleagues, when they began listing the colleagues who they most admired, who drove them the hardest, who made them feel like what they were doing was important–and just about every member of the management team cited just about everyone else by name–inevitably they started that list with Kristin Antelman.

The sense I got of Kristin, in part from our conversation, but mostly from hearing her colleagues talk about her, was captured by Steve Yegge, a programmer at Google, in a post entitled Done, and Get Things Smart:

“At first it’s entirely non-obvious who’s responsible for Google’s culture of engineering discipline: the design docs, audited code reviews, early design reviews, readability reviews, resisting introduction of new languages, unit testing and code coverage, profiling and performance testing, etc. You know. The whole gamut of processes and tools that quality engineering organizations use to ensure that code is open, readable, documented, and generally non-shoddy work.

But if you keep an eye on the emails that go out to Google’s engineering staff, over time a pattern emerges: there’s one superheroic dude who’s keeping us all in line.”²

The trait Kristin shares with Yegge’s coworker at Google is that she excels at understanding how decisions made today—or left unmade today—can impact the future. And she insists on looking at reality as it is and seems likely to be, not as people might wish for it to be. As NCSU’s Associate Director for the Digital Library, one of her major initiatives over the past few years was to lead the group that first introduced faceted browsing to library catalogs, using the Endeca software that was previously used only on commercial websites like Home Depot’s. After rolling out the catalog at NCSU Libraries, she and her colleagues worked with their peers in the Triangle Research Library Network to create an Endeca-powered union catalog (in addition to NCSU, the network comprises the University of North Carolina at Chapel Hill, Duke University, and North Carolina Central University). In an article for the April 2009 issue of College & Research Libraries News she co-authored with TRLN’s Mona Couts, they emphasize the ambiguity inherent in the project:

“TRLN librarians were in agreement that our catalogs were bad, and that what NCSU had in its Endeca catalog was, if not the answer, at least an improvement. The harder challenge is that the very concept of the catalog is in transition. Implementing a “next-generation” catalog doesn’t answer the question, what should a library catalog be anymore?”

When I learned that a group of Assistant/Associate University Librarians and Assistant/Associate Directors (AUL/AD) in academic libraries, known as the Taiga Forum, issued a series of provocative statements on the future of libraries, it was no surprise to me that Kristin Antelman was on the steering committee that helped create the document. And when I read the statements themselves, I was sure I detected some of her ideas.

Over the past few weeks, I had the good fortune to interview Kristin about Taiga, the statements, and the future of libraries. Although during the course of our conversation we chose not to dissect the Taiga Forum members’ creation or discussion of each statement individually,³ we encourage you to use the comments section that follows this article to share your thoughts on the statements themselves as well as the other ideas Kristin shared.

Why did you agree to join the Taiga Steering Committee and to moderate a session? What was it about Taiga that appealed to you?

I got involved with organizing Taiga 4 because I had attended the first three Taigas and found them to be great meetings. They were unlike any professional meetings I had been to; we spent a whole day talking honestly about big and difficult challenges facing academic libraries. At the end of Taiga 3, I felt I wanted to have some input in how the next one was done.

The Taiga meetings were conceived as a venue for people at the Associate University Librarian/Associate Director level in academic libraries to get together and discuss common challenges. We tend to have few peers in our home institutions and, sometimes, in smaller institutions, none at all. The premise of Taiga was that, while directors had venues to talk amongst themselves, there was no such venue for administrators below the level of director to talk frankly about issues across functional lines and with colleagues from other institutions.

The first year saw the development of ten provocative statements. Those statements ended up serving as the basis for lively conversations not only at the first Taiga meeting itself, but in academic libraries across the country for years afterward. I think they struck a chord because they dared to express fears and forebodings about our collective future that many of us were feeling but that we may not have had the courage (at that time anyway) to speak freely about. Taigas 2 and 3, very successfully in my opinion, employed the “open space” approach to participant-defined meetings. You could even say we were ahead of the curve on the “unconference.”

The aspect of Taiga 4 that has received the most attention was its revised “Provocative Statements” document. What was its purpose?

For Taiga 4, which was held this past January before ALA in Denver, the steering group had the idea to revisit which (if any) of the original provocative statements were still valid, and then to add to them. The new statements would be focused around the theme of this year’s meeting, “Organizational Change: Professional Identity and Personal Commitment.” We asked the Taiga community for feedback and took those responses into account when we wrote the new statements. As it happened, we did not carry forward any of the original statements, but incorporated a lot of the same themes in the new ones. The statements were written by a subgroup of the steering committee over several phone calls and wiki work. They were then commented on and edited by the full steering committee, and were distributed to the people who signed up to attend the meeting.

We then asked for volunteers to do “lightning talks” on the statements at the Taiga 4 meeting. Those talks were each followed by 10 or so minutes of discussion, which planted many seeds for conversation for the rest of the day. At the end of the meeting, we reviewed how we felt about the statements. That recap resulted in minor changes, including deleting statement #3 (about the dominance of Google) as not very provocative.

One of the misconceptions about the statements has been that the Taiga meeting participants believe that these things will happen, or, more interestingly, should happen. Actually, their purpose is largely rhetorical. We hoped the statements would inspire conversation—and resistance!—at our meeting. We very intentionally meant to say that we feel that research libraries are facing serious challenges to core areas of what we do and that we want to talk about these challenges without presuming any answers. I would also add (and here I’m speaking for myself and not the group) that I think the statements also explicitly confront superficial optimism about how academic libraries—and librarians—will transition into new roles.

The subtext of many of the statements is the as-yet-unknown impact of a potentially prolonged period of tough budget times, which was just becoming evident when these were written. How libraries build collections and are staffed now is a product of many decades of pretty robust growth. It remains to be seen what path libraries will take when budgets are shrinking, but ideas like realizing we cannot support a hybrid print/electronic model indefinitely, or cannot continue to work around underperforming employees, are a couple responses to these pressures that we explored.

Are the reactions you’ve seen—the ones that respond to the content rather than the context—in any way satisfying, even if their writers appear to be dismissive of the ideas expressed within the statements? Do these librarians’ strong reactions mean the statements are doing what they’re supposed to do?

Any reaction means the statements have had an impact. Response to the statements’ content and their context have been quite intertwined, however. Having made the decision to send the statements out into the world, we made a mistake in distributing them in a static way, with a lack of transparency about their context (who did this? what was the purpose?). We were rightly criticized for that.⁴

Apparently, the Darien Statements might be a response to Taiga, although they don’t claim that.⁵ Aside from being both being list-like and appearing around the same time, I don’t see too many commonalities. Except, that is, in the section called “as librarians, we must…”, where the Darien Statements have quite a bit in common with the spirit of Taiga, including their own expression of some of the points made in the provocative statements.

One aspect of the responses that does concern me is that there seems to be a pervasive, and enthusiastically embraced, gap of trust with administrators. While maybe that’s just something that always has been and always will be, it concerns me because these divisions weaken us. Those of us who are currently AULs or ADs are not MBA-types dropped into libraries; we have spent most of our careers working in various non-administrative librarian jobs. In fact, my impression is that a significant number of AUL/ADs attend Taiga soon after arriving in their positions.

Another criticism I’ve seen is that we’re too negative, that we don’t propose answers. It’s worth noting that, while most of the statements themselves don’t propose answers, the discussion at the meeting did very much address answers. How libraries address the challenges facing us often gets back to organizational culture. Acknowledging the need, and then adjusting what we do and who does it, sometimes in significant ways, is not an easy task for any of us, whether you are a front-line library worker, a manager, or an administrator. A couple colleagues and I have been working on a project to find out more about what future library leaders are thinking. This dovetailed with the Taiga 4 theme, so we prepared a little video of interviews with some of these librarians that we showed at the beginning of the meeting.

Will there be a Taiga 5?

Since Taiga is not a formal organization, we see where it takes us year to year. Thanks to the continued generosity of our sponsors, Innovative Interfaces and R2 Consulting, a Taiga 5 meeting will be possible, but what form it will take remains to be seen.

Time for some non-Taiga questions. What do you think library schools should be emphasizing? Requiring? Or, put another way, what are the abilities you consider most important in potential NCSU Fellows?

Library school programs are becoming increasingly differentiated it seems to me; and they have to in order to survive. Distance education will make it possible for prospective students to find the program that best meets their needs. These are both positive developments. I think that internships are even more critical than ever. Every recent MLS we hire tells us that they learned more in those experiences than they did from their educational program. Separating the Masters coursework from learning library practice would also help address the theory/practice identity crisis characteristic of MLS programs.

In terms of skills, I like to see librarians who have the ability to think through problems in a systematic way, who can learn independently, who are fearless and enthusiastic about technology. It’s critical that they be able to communicate effectively, including in writing, and that they show leadership qualities. They should be focused on the big picture and be pointed toward the future, thinking about what libraries are for, not what we do, because what we do is changing very quickly. I’m very encouraged by the graduates I’ve seen in recent years. The applicants to our Fellows program just seem to get stronger every year.

What are the most useful things ALA can do for us as a profession?

I think ALA is most effective when it works as an advocate for public libraries, promoting the contribution that public libraries make to communities across the country. Our public libraries are a tremendous achievement of this society, really unique in the world, and yet one that we cannot take for granted will always be there, especially as local governments are hard hit economically.

ALA and its divisions also serve as a valuable professional development opportunity, where people can find leadership and other opportunities even if their jobs do not offer them the chance to develop in that way.

Are there any other professional associations or consortia that are more important to you than ALA?

Actually, ALA is pretty important to me. LITA is my primary home in ALA, and I try to stay involved with LITA committees, etc.

Closer to home, the Triangle Research Libraries Network is an important professional connection. TRLN is very active both in developing shared services and sponsoring information sharing and professional development events for staff at the TRLN libraries.

The Digital Library Federation (recently folded into CLIR) and the Coalition for Networked Information have been important associations for me as well. Both organizations hold semi-annual meetings where members can share ongoing work.

What data do you wish you had available to you in figuring out how well the library is meeting its constituency’s needs?

Data about fast-changing areas, such as discovery, would be useful to have. I’m concerned that we understand only in a sketchy way how our different users are finding the information they need, and where and when that leads them to library collections or to library-provided tools.

Two or three years from now, what will be the minimum requirements for a really good library website/catalog? What will its users expect it to be able to do?

I anticipate users will expect to have to interact with the library website or catalog much less, or hardly ever at all—which, I note, is hardly a provocative statement! The library website will continue to lead our users to information about our spaces and services, but our goal should be to make its footprint as minimal as possible in our users’ lives. The resources they can get to by virtue of their institutional affiliation should be seamlessly linkable from course sites and search engines. For this to happen, linking technologies, like OpenURL, will have to work even better than they do now. But we also will have to make this vision a priority—from negotiations with information providers to how we make local investments of our staff time and development resources.

Do you foresee anything changing the dynamic between libraries and information providers?

One frustration for me is that we have not had much success in buying/licensing just data; providers will only offer data in the context of their products, their interfaces. Had libraries been able to buy metadata for scholarly articles, for instance, we could have conceivably developed reasonable metasearch solutions. But that time is passed, now, with Google Scholar. Good data to support reference linking services is still hard to get, and it hurts our services. Quality metadata to drive OpenURL-based services for ebooks is also an area where the information ecosystem has a ways to go. Ebooks themselves have all kinds of platform restrictions that create challenges for libraries. But whether libraries have now, or will ever have, the leverage to get access to more open content is debatable. As the market consolidates around Google and a handful of major publishers, we will likely increasingly be at their mercy, in terms of APIs into their content and services. Even if that’s the case, though, there’s much that can be done with those tools; I think libraries by and large underutilize those opportunities to develop integrated services that are already made available to us.

What could we be doing to better utilize the available tools?

Just looking at the catalog, there’s no reason that any library should be running a last-generation ILS OPAC interface. There are open source and relatively low-cost commercial options that can give your library a current, faceted interface with good relevancy in keyword searching. There are also a range of APIs from Google, OCLC, LibraryThing, etc. that should be employed to make searching the catalog a richer experience, better integrated with the larger information environment.

Is there anything we could do to that would keep us from being at the mercy of Google and the major publishers?

I’m much more concerned about being at the mercy of publishers than Google. Google has advanced access to information worldwide far more than libraries ever could dream of doing; where they encroach on our area they are changing the paradigm for the better (for example, full text-based rather than metadata-based discovery of books).

Scholarly publishers, operating in an increasingly consolidated market, will continue to raise prices beyond inflation and restrict libraries through complex big deal licenses. They do have us at their mercy. Open access may be the eventual solution (and I think it is) but, in the interim, the detrimental impacts of their dominance (smaller market for monographs, for instance) will continue to be significant. One thing libraries can do—and many have done—is never again enter into big deals, where flexibility is traded for cost savings. Another thing libraries can do is to be less fixated on collecting for posterity. Scholarly work is increasingly preserved beyond our walls: a significant percentage of the best articles are already openly available on the web (and this segment is growing), while another significant percentage is made openly available by publishers after an embargo period. Libraries, collectively, will have to be less dogmatic about licensing (and replicating) complete and official versions of the STM (scientific/technical/medical) literature. At risk are two dimensions of our mission that have historically (and justifiably) defined us as research libraries: developing collections of significant breadth to meet the needs of all our constituents and maintaining the capacity to invest in new services.

If a large library had to make big cuts, what are the first expenses that should go? What are the programs/positions, etc. it should absolutely protect?

This is very much a local decision and depends on where the library has already had to cut back and where its strengths lie. While downsizing is an opportunity to be strategic about positioning our organizations for the future, I don’t think we’re yet in a climate where our parent institutions will tolerate unbalanced cuts, i.e., cuts that too disproportionately affect either collections or services. One of the provocative statements (or perhaps two) addresses the need to reduce speculative spending;⁶ I think that will have to come to pass, and sooner rather than later. I also think we’ll have to get out of the local catalog business within a couple years, and that has significant implications for our technical services staff. Digital library development is still starved in most institutions, resulting in the poor discovery tools and websites that we see now. How each library faces these challenges, both the process they take and the outcome, will reveal much about the character of an organization and its leadership. I also see that libraries will have to focus somewhat less on serving the broader library community and community of future scholars, and much more on their mission within their own organization.

For libraries with limited resources, there’s often a tension between serving the broader scholarly community and meeting local needs. How do you see this playing out?

This question gets at what I think is a big challenge for us. Our special collections may be where we are unique, and can make the greatest contribution to the cultural heritage community at large, but they will never be where we will make the greatest contribution locally. And the path forward (digitization) is expensive. So the question arises, why would—or should—our universities fund that work? One of the Taiga statements mentioned that these efforts would be privately funded, and I think that will have to be the case, although this will result in organizational inefficiencies and relatively slow progress overall.

But special collections are not the future for most academic libraries. The future that we all share is becoming much better integrated into campus life, and closer to teaching and learning (there’s a Taiga statement about that, too, the “blended librarian” idea).

Let’s finish on a positive note. What have been some of your most pleasant surprises over the last five years? What’s happened for you professionally, for NCSU Libraries, or for the profession as a whole that’s far exceeded your expectations?

As far as NCSU Libraries goes, the biggest surprise has been that the state legislature funded a new library for NC State University. The $126 million library, the James B. Hunt Jr. Library, is currently completing the design phase and is scheduled to open in 2012. Librarians who have lived through retrofits and add-ons to existing buildings know how constraining that can be in terms of creating new spaces for users. The opportunity to participate in the design of new learning, collaborative, and research spaces, rich in technology and good design, has been a huge thrill for me. If we do this right, it will serve as a model for what an academic library can be going forward.

In terms of the profession as a whole, I would return to the topic of the new graduates that our library schools are producing. I would say that, without question, the graduates of the last five years are more well-rounded, smarter, and better prepared to make immediate contributions than at any time since I’ve been a librarian. These people are, by definition, our future. It’s up to us to give them the tools they need and the latitude to realize their potential within our organizations. If we can do that, libraries will have a bright future.

Thanks to Kristin Antelman for her thoughtful responses and her generosity, and to Stephanie Atkins, Beth Picknally Camden, Claire Stewart, and Hilary Davis for their helpful comments on earlier drafts of this article.

1. The full quote by Alan Kay: “Don’t worry about what anybody else is going to do… The best way to predict the future is to invent it. Really smart people with reasonable funding can do just about anything that doesn’t violate too many of Newton’s Laws!” He said it during an early meeting of PARC members and Xerox planners.
2. Yegge writes later in the essay: “Incidentally, they hired plenty of other brilliant seed engineers who were equally responsible for Google’s great technical infrastructure. I’m just using this one guy as an illustrative example.” I’m doing the same. A lot of people are responsible for making NCSU, in my opinion, the best library in existence. But I got the sense, from my conversations that day, that they credited Kristin with keeping them all in line.
3. As Kristin noted, “I’m uncomfortable speaking for the group in that way, i.e., interpreting the meaning behind the statements or characterizing the discussions of the day (I couldn’t even accurately recall such, even if we didn’t tell people they were confidential).” I think this makes a great deal of sense, especially once you understand how Taiga works and the reason the statements were drafted.
4. Some of the writers who have been involved in the conversation inspired by the 2009 provocative statements include: Steven Bell (ACRLog); John Dupuis (Confessions of a Science Librarian); Meredith Farkas (Information Wants To Be Free); Steve Lawson (See Also…); Dorothea Salo (Caveat Lector); and Roy Tennant (Library Journal Digital Libraries).
5. The official version of the Darien Statements on the Library and Librarians is hosted at John Blyberg’s blyberg.net. For more on the Darien Statements, see posts by Cindi Trainor at Citegeist and Kathryn Greenhill at Librarians Matter.
6. Statements 2 and 7, which read as follows:

   2. In five years collection development as we now know it will cease to exist as selection of library materials will be entirely patron-initiated. Ownership of materials will be limited to what is actively used. The only collection development activities involving librarians will be competition over special collections and archives.

   7. In five years libraries will have abandoned the hybrid model to focus exclusively on electronic collections, with limited investments in managing shared print archives. Local unique collections will be funded only by donor contributions.

Thanks to Flickr user gynti_46 for use of the photo.

Since October something has been weighing on my professional mind: my abusive relationship with Google. I love Google, I don’t ever want to leave my Gmail, my Gchat, my GoogleDocs, my web searches, my Google Reader, but right now I wish I weren’t so dependent on it.

The weight to which I am referring is the proposed Google Book Search Settlement Agreement. Google knows with whom I e-mail and chat, for what I search, what blogs I read, and on and on. With the proposed settlement Google will take a further step in controlling my (and libraries’) information use and seeking behavior. Google will know what books I read, what pages I read, how long I read them, what pages I print, and what passages I copy and paste. (If you don’t know what I’m talking about you should stop reading immediately and read the 2-Page Super Simple Summary on the Google Book Search settlement agreement produced by the ALA Office of Information Technology Policy (OITP). Then, and only then, continue here at ItLWtLP.)

For those of you who aren’t going to go read this document, here’s my simple recap: The American Association of Publishers (AAP) and the Authors Guild filed a class action lawsuit against Google Book Search for copyright infringement. Instead of going to trial, the parties have agreed to settle out-of-court. Google has agreed to fund a rightsholder database called the Book Rights Registry, which will be run by the rightsholders (authors and publishers). Google will sell books to individual consumers, but rightsholders will have financial stakes in the product. Libraries will be able to subscribe to gain full-text access to books via the Google Book Search Project, mimicking the same model as many other library products. The proposed settlement has far-reaching implications for use of digitized materials in libraries, the role of fair use, and the future digital market. Unfortunately, many of the agreement’s facets are antithetical to the mission and purpose of libraries. In fact, some libraries, such as  Harvard, immediately pulled out of participation with the Google Book Search Project.

I won’t provide you with a more in depth analysis of the suit in this blog post. As I mentioned in my first sentence, this abusive relationship has been eating at my brain for many months and it’s just now beginning to solidify. What I do want to share is what I think we in the library community can do about the settlement. The stakes of the settlement are enormous, and neither the rightholders or Google represent libraries in this process. But we, librarians and the library community at large, are an ornery bunch. Aren’t we the community that took to court over the PATRIOT Act? Aren’t we the community that instigated a public outcry when Michael Moore’s publisher pulled Stupid White Men for being too critical of former President Bush? Aren’t we “radical” and “militant?”

Because I don’t want libraries, information advocates, patrons, or anyone else to be trapped in an abusive relationship with Google I would like to offer the following suggestions for what individuals and the professional community can do to protect and salvage what remains of our relationship with “the big G.” (And maybe even make this Google Book Settlement Agreement a bit more reasonable.)

Individuals

Educate yourself.

Knowledge is empowerment. Read through blog posts, documents, and news articles about the proposed settlement agreement. The ALA Washington Office is tracking most everything that’s out there and has made a nice little portal web site for you to use. Particularly useful is also the Guide to the Perplexed: Libraries and the Google Library Project Settlement. This longer document provides a broader view than the 2-page document. You might even consider checking out what Google has to say about the “groundbreaking agreement.”

Because the settlement is so intrinsically tied to copyright law and fair use, this is an ideal time to refresh yourself on the basics. Re-read Kenny Crews’s Copyright Law for Librarians and Educators and Carrie Russell’s Complete Copyright. Subscribe to blogs that deal with copyright such as librarycopyright.net or Karen Coyle’s blog.

Ruminate.

Ask yourself and think about the tough questions. During the Google Book Settlement: What’s in it for Libraries? panel at ALA Midwinter, Karen Coyle posed the following questions: Does the product serve my users? What will the collection be? What is the quality of the product? Panelist Laura Quilter pushed the panel participants and audience to consider the privacy issues presented by the proposed model for accessing digital materials through Google Books. As librarians we have a responsibility to protect our users. Mold and define your personal and professional values for privacy. This will be incredibly useful if you are put in a place to consider purchasing and implementing this subscription product in your library.

Be an advocate in your community.

Let’s face it. There are so many issues to follow in our profession, that chances are many of your colleagues might not know anything about this proposed settlement agreement. Talk with your colleagues and share with them what you have learned. Push your administrators to find out if any pre-emptive discussions regarding this product have occurred. What is the institutional stance on the settlement agreement and Google Books in general? By asking the hard questions of our supervisors and administrators, we are often able to generate institutional discourse.

The Community

Ask and discuss.

ALA has very bright and informed people working to understand the Google Book Settlement Agreement. Librarians who specialize in information policy, entire offices and committees that deal with legislation and lobbying for ALA interests. But this 300+ page legal document that is the agreement is confusing and still not fully understood by the library community. At the aforementioned Midwinter panel discussion, many things came to light that we (or at least I) did not previously know about the settlement. For example, the settlement will not allow for a subscriber library’s users to login via remote access and access their library’s subscription to the Google Books database. Users who are community members of a subscribing institution will only be able to access the resource “on campus.” Another fine example is how Google will serve public libraries with this product. Google will allow public libraries one access station to the product. Only one.

We need more fora in which to engage to find out exactly what the settlement agreement means to us and our users. Professional organizations, ALA, SLA, PLA, ARL and others should consider hosting more web-hosted seminars for their members on the subject. Moreover, hosting other kinds of discussion fora to ask questions and commiserate within the library community such as BBS or wikis or even blogs will be helpful to those of use who struggle to understand the issues inherent with the settlement.

It is also of import to note that during the panel at Midwinter Dan Clancy, Engineering Director for the Google Book Search Project, said he would like to be able to be available to the library community for more discussion. State libraries, consortia, or other large groups should consider contacting Dan and schedule a telecon about concerns.

Educate Google.

I would like to give Google the benefit of the doubt. However, the fact remains that Google is a business and will not implement policy or procedure based upon it being “the right things to do.” Rather, Google will make policy, and change procedure, as it is beneficial to business and the deep Google pocketbook. That being said, I think Google would attempt to take more responsibility for “doing the right thing” if the company were to realize that the proposed settlement model is not one upon which libraries will willingly spend their money. Just because Google will have a monopoly on the digitized books, does not mean that we should lower our standards for offering resources to patrons that are easy to use and ethically implemented. We, as a community, need to share with Google the ethical principles and best practices that we have worked so hard to develop—of particular relevance, the Principles for Digital Content, and the Principles for a Networked World.

Develop position statements, draft and pass resolutions, or take other governmental action.

A unified voice of librarians can be a powerful thing. Moreover, if professional organizations such as ALA, whose membership is purported to be 65,000 (according to the ALA Annual Report), use their position as the good stewards of knowledge and information, we have the ability to put up a good fight that might yield some positive results. Currently the Washington Office is working to gather ALA membership input so that it can issue a position statement or take other action on the settlement. (I don’t even know the proper channels to let ALA where I stand on this issues. To this end, ALA should consider creating a system that enables soliciting and gaining membership comment when warranted.)

ALA Council should also consider passing a resolution regarding the Google Book Search Settlement Agreement. It is not out of the question that this kind of political activity will help the organization to retain its integrity and ethics regarding privacy, information policy, and what best serves libraries and patrons.

ALA and other library organizations should consider future legal action. It seems to me that libraries would have a good case to bring forth their own class action lawsuit. This might be a last case resort, but I do not think we should not sit idly by if a large market-driven product were to threaten the library community’s ability to best serve the public.

Create support materials and documents for libraries to use.

Shortly after the court “okays” the Google Books Settlement agreement, libraries will face a “purchase or not to purchase” question for the Google Books subscription product. Navigating the ins and outs of the legalese in the settlement will be daunting for any library system, consortium, or lone library that chooses to buy the product. Having FAQs handy or even an ALA Toolkit on best implementation practices for Google Books would be a great service.

It doesn’t have to be a waiting game.

If we work now to understand what we can about the proposed settlement, if we start to evaluate the effect purchasing this product will have on our libraries and patrons, if we create a unified voice and foster discourse, then we will better be able to keep fires under control and perhaps keep our brains in our heads. Google is a powerful company, but powerful, too, is the voice of libraries and librarians. I firmly believe that if we continue to put our efforts toward understanding everything encompassed by the Google Book settlement issue, then we will better be able to serve our communities, and perhaps inform positive changes that will let us sit in better peace with our friend and enemy. This is my call to you, colleagues, to engage, think, debate, and defend library values. Take control and save yourself from this abusive relationship. Google can be a reference librarian’s best friend, but right now, with the proposed settlement, it is looking as if we are subject to continued abuse.

Le Mundaneum à Mons (Belgique) © dalbera / CC-BY-NC

Depending on books can feel like relying on snail mail. “Now that I’ve showed you how to find some articles,” I say to people at the reference desk, “I’ll show you how to use our website to find some books you might want to check out. And after that, wouldn’t it make your grandmother’s day if you wrote her a letter?”

For anyone accustomed to the Internet, books can lack the immediacy of articles or websites. Books generally have slower developing narratives, and often have longer paragraphs, sentences, and words, which means they don’t lend themselves to skimming. Compared to digital material, relevant passages can be hard to find, and even finding the right book can be challenging.

Although library websites are improving, keyword searching doesn’t work well at most libraries and faceted browsing—the links down the left side of the page on Amazon—is still a rarity. More importantly, with one notable exception, there is a good chance that nothing on the shelf that is “printed on paper and constructed on the model of the codex” includes the exact information you have in mind.

This is where universal catalogs come into play. If there’s nothing on the shelf that meets your needs, the next step is to figure out if such a book exists. There are five websites that provide relatively complete and easily accessible lists of books: Amazon, Google, LibraryThing, WorldCat, and Open Library. In order to make the best use of these websites, it can be useful to learn how each of them started, what keeps them going, and how their business models and practices affect the data they collect and and how they go about sharing it.

Amazon

It’s tempting to think of Amazon as a technology company. That’s how Werner Vogels sees it, which is understandable: he’s their Chief Technology Officer, and he seems to have done a very good job of it, because Amazon’s technological initiatives have taken a leap forward since Amazon hired him away from Cornell in 2004. Over the last couple of years, Amazon has made its mark as a service supplier, rewriting the rules for online hosting with its Amazon Web Services; it has developed a successful consumer electronics product (the demand for its Kindle e-book reader consistently exceeds supply, and it seems to be extraordinarily popular with publishers as well: they have made almost 200,000 titles available); and it has also made use of its infrastructure with offerings as diverse as its Mechanical Turk and Fulfillment services.

But if you look at its revenue stream, it’s pretty clear that Amazon has very little in common with a traditional technology company, such as Microsoft, its Seattle-area neighbor. Instead, Amazon is probably most like a different neighbor: Costco.

Amazon’s founder, Jeffrey Bezos, seems to have a firm grasp of three important aspects of retailing:

• Look for items that can be sold in near limitless quantities (such as “books, music and videos”);
• Figure out how to sell them profitably but with minimal markup (“He said he would ‘relentlessly slash prices,’ even if it cut into incremental profits, because he was convinced that it was the right thing to do”); and
• Focus your energy on building customer loyalty (“Satisfaction surveys show that Amazon enjoys a golden reputation among most of its 49 million active customers”).

Similarly, Costco’s founders, James Sinegal and Jeffrey Brotman, stock their retail outlets to the rafters, refuse to mark up items more than 15%, and, in their most recent report to shareholders, they note, “This past year we also enjoyed the highest membership renewal rate in our history at 87%, attesting, we believe, to the high level of satisfaction our members have in our products and services.” Think about the things you typically shop for at Amazon: are they more like what you buy from Microsoft or are they more like what you buy from Costco?

Because of Amazon’s size, breadth, and ubiquity, it can be easy to forget that its original business model was pretty basic: it resold books it bought from Ingram and Baker & Taylor. As Tim O’Reilly points out in an apologia on Web 2.0, Amazon purchased a database of book information from R.R. Bowker, put it on the still new World Wide Web, and encouraged its customers to share reviews, bibliographies, and even correct any mistakes or omissions in its data. Two years later, when Amazon went public, it carried more than 2.5 million titles, “including most of the estimated 1.5 million English-language books believed to be in print, more than one million out-of-print titles believed likely to be in circulation and a smaller number of CDs, videotapes and audiotapes.” Out-of-print titles were generally available within two to six months.

Amazon’s original formula hasn’t changed all that drastically. In 2007, books and other media accounted for 62% of its net sales, down from 66% in 2006 and 70% in 2005. The trend may be downward, but media sales are actually improving—it’s just that other sales are improving even faster.

Despite investments in other areas, Amazon knows that it is still primarily a retailer of books and other media, and it continues to invest in complementary initiatives and businesses that fortify its ability to sell these items. Its recent acquisitions, including Audible, Shelfari, and AbeBooks (which brings with it a 40% stake in LibraryThing), join other Amazon businesses, including the Internet Movie Database (IMDB), Alexa, and BookSurge. It also developed its own search subsidiary, A9, it was an important participant in creating ONIX, “the international standard for representing and communicating book industry product information in electronic form,” and it published a hugely successful API (now a part of its Associates program) through which it makes book jackets and summaries available to affiliates (including libraries), and also shares a percentage of sales, inspiring creative programmers to develop websites like BigBookSearch and Zoomii.

Amazon does all this so it can sell more goods and, in general, it seems to be working. Consumers are getting deeper discounts on a broader range of books and other media than ever before, and they have an easy time finding the items they want thanks to Amazon’s faceted browsing interface, its active user community, and its search engine which, in many cases, makes it easy to search within the text of published items.

While Amazon does everything it can to provide you with as much information as possible about the items it has in stock, there’s no motivation for it to share information about items it can’t sell in volume, such as out-of-print material. If the information you’re seeking is likely to be included in new, commercially available books, then Amazon is an excellent resource. If not, you’re best served looking elsewhere.

Google

Amazon is one of two major corporate alternatives to libraries; Google is the other.

Amazon followed one of the two traditional paths for forming a giant corporation: it was founded by an entrepreneur who had a good idea for a company and then hired talented people to build its technological infrastructure. Google followed the other path: its founders, Larry Page and Sergey Brin, created something the world wanted and then hired people to turn their idea into a profitable corporation.

While still graduate students at Stanford, Page and Brin took Eugene Garfield’s work on citation indexing and adapted it for the World Wide Web. Garfield, who marketed information products through his company, the Institute for Scientific Information (now a Thomson Reuters subsidiary), records how often scholarly papers are cited by subsequent scholarly papers, which is useful because citation frequency is a reasonable proxy for importance. Similarly, Google’s PageRank algorithm is primarily a scheme for measuring and weighting links between Web pages: the more links to a page or website, the more likely it is to be important, especially if those links come from other important sites. PageRank is intended to determine which Web pages are likely to be perceived by Google’s users as relevant.

It was soon apparent that Google worked—users found what they were looking for—but no one saw any money in it. Page and Brin tried to sell their technology for $1 million to the big players in the Web market. After everyone turned them down, they decided to start their own company, focusing their attention on attracting as many users as possible.

Where Amazon is a retailer that can be thought of as a virtual Costco, Google is an entertainment company like News Corp or Viacom—it generates 99% of its revenue from advertisements. Just as Amazon is primarily a reseller of products others make, Google is primarily a portal into content others create. Its mission is to “organize the world’s information and make it universally accessible and useful.” Note the absence of the word “Web” in that mission statement: Google’s goal is to organize every bit of information. For instance, Google created its free telephone directory assistance project, GOOG-411 in order to develop speech recognition software. In turning spoken words into text, Google opens up the possibility of searching audio and video files through the same Google search box that is currently used to search websites.

Though the Web has become many people’s primary information source, a great deal of the world’s information is still found in books. In order to harvest that data, in December 2004, Google announced that five libraries—the University of Michigan, Harvard. Stanford, Oxford, and the New York Public Library—had agreed to let Google begin scanning their collections (and several more have since joined the project). Multiple elements of this arrangement remained secret, including the terms of these agreements and the rate at which books were being scanned. It was also unclear how Google would deal with potential copyright issues, especially after the Association of American Publishers and the Authors Guild almost immediately filed a joint lawsuit.

This copyright lawsuit mirrors another: Viacom’s suit against Google acquisition YouTube for copyright infringement. There was some speculation that Google bought YouTube specifically to make sure YouTube didn’t lose its lawsuit, establishing a precedent that Google would have to overcome if it were ever sued for hosting video files. When Google reached a settlement in its book scanning lawsuit this past October, Viacom saw a potential concession in its own suit.

The book-scanning settlement has raised concerns about preservation and access for Google-scanned materials. Harvard has expressed its reservations publicly, and Peter Brantley has been doing an extraordinarily good job of identifying and summarizing the issues involved. How all this will affect people who want to read books online has yet to be determined.

What does seem settled, at least for now, is that Google has archived an unparalleled number of books (and also scholarly articles) whose entire text could be as easy to search as the Web. With the success of Google-411, it seems likely that Google will soon be able to offer text-based searching within audio and video files as well.

What’s not clear is whether advertising will make these ventures profitable or if Google can successfully transition to alternative business models for subsets of its data. Right now, it resells access to scholarly articles and newspaper stories for several publishers, and it appears that it will soon be selling access to the books it has digitally archived. It’s also not clear if Google sees any point in developing an active user community around books. While Google allows users to add reviews at its book website, user-contributed content is not a focus in the same way it is at Amazon or at LibraryThing.

LibraryThing

Founder Tim Spalding’s LibraryThing is a new kind of Internet-enabled organization, the small company that operates on a large scale. This method for doing business has been best documented by programmer, essayist, and venture capitalist Paul Graham, one of Spalding’s inspirations, though LibraryThing probably resembles Craigslist more than it resembles any of the YCombinator companies Graham has helped to shepherd into existence.

Like Craigslist, LibraryThing has an evangelical faith in its users, maintains a simple and easy to understand interface, is satisfied with steady and modest profitability, and competes for attention in a field with significantly larger entities (Craigslist is often cited as a cause of the newspaper industry’s financial difficulties, even though it employs fewer than 30 people).

LibraryThing gets its data from Amazon, from libraries that make their catalogs available through the Z39.50 protocol, and from its users, who supplement the data by providing reviews, cataloging information, adding tags, and disambiguating records. These last two seem to be particularly successful even though they vary from standard library practice.

The tagging concept, popularized by Joshua Shachter’s group bookmarking website, del.icio.us, allows users to catalog items using whatever keyword they wish. This enables works like Bridget Jones’s Diary to be tagged “chicklit” or Neuromancer to be tagged “cyberpunk,” subject terms that differ greatly from Library of Congress designations for these works by Fielding and Gibson.

Disambiguation allows users to clarify records by taking actions such as combining entries for works that are identical but released under different titles, or aggregating work under a single author heading even though that person has released work under multiple names. These can be difficult tasks when a small group of staff members attempt to take this on manually, and it has proved tricky to teach computers to disambiguate records programmatically. For instance, author Cyril Northcote Parkinson’s name is subject to multiple permutations (C.N., Cyril N., C. Northcote, etc.), and his most famous work, Parkinson’s Law (which expands on his belief that “work expands so as to fill the time available for its completion”), has been released with multiple title variations and in numerous editions. Amazon struggles to make it clear which edition of Parkinson’s Law a potential customer might wish to purchase and Google offers a few different options that are not readily distinguishable from one another. LibraryThing, while representing more options than either of the other two, also makes it clear which title its users believe should be considered definitive.

It’s worth noting that Amazon, Google, and LibraryThing are not operating on a different scale when it comes to the number of books they’re cataloging. LibraryThing, which launched on August 29, 2005, has catalog entries for over 32 million books. While open cataloging has its limitations, LibraryThing’s website regularly demonstrates the power of crowdsourcing big tasks to a large, devoted community.

That community is the key to LibraryThing’s success. Just as del.icio.us users socialize around shared bookmarks and tags, LibraryThing users socialize around the books in their collections. Users can add 200 books for free, but to add more they have to pay either $10 per year or spend $25 for a lifetime membership.

That’s one way LibraryThing makes money. The other is LibraryThing for Libraries, a service that allows libraries to integrate LibraryThing’s tag database and, as of September 2008, its user reviews, into participating libraries’ websites. This service is offered on a sliding scale, with the smallest libraries paying $1,000 per year.

While Amazon’s business model does not target libraries in any discernible way (either as customers or competitors), and Google appears to be interested only in the largest libraries as partners, LibraryThing seems to be actively interested in selling its services to pretty much every kind of library—dozens have already signed up for LibraryThing for Libraries—and in digesting Z39.50 feeds (or getting records in other formats) from any library willing to share. In a pinch, it appears that LibraryThing will even take care of your cataloging.

WorldCat

OCLC is a nonprofit consortium that includes almost 70,000 libraries as members. It was founded in 1967 as the Ohio College Library Consortium. In 1977, it began allowing libraries outside Ohio to become members, and in 1981 it changed its name to the Online Computer Library Center. It has made multiple acquisitions as it has grown, including the Dewey Decimal Classification System and its only competitor, the Research Libraries Group, which operated from 1974 until 2006. This sort of activity, and OCLC’s business model, led to its nonprofit status being investigated, but ultimately recognized. Understandably, OCLC uses its tax status to its advantage, just as some nonprofit hospitals take advantage of their status and IKEA makes use of its unusual structure.

OCLC’s most widely visible product is an amazingly good website, WorldCat.org, which provides free access to over 110 million library catalog records, most of which are for books: member libraries provide access to their entire collection, which includes articles, audio, and video. Right now, WorldCat.org is the best free website that lets visitors use keywords to conduct serious research across all media types, a feature which all on its own would make it valuable. On top of that, OCLC has integrated its work on FRBR and xISBN—projects that make it easier to find what you’re looking for—helping to turn WorldCat.org into an invaluable resource.

One of the two major problems with WorldCat.org is what it doesn’t include: the long tail of library records. With 70,000 libraries contributing records, it’s tempting to assume that just about every book is included in the WorldCat.org database, but that’s probably far from true. OCLC’s Karen Calhoun has written about its efforts to position its pricing and services so smaller libraries can participate, and OCLC is making inroads, but it still serves far fewer than half of the smaller libraries in the United States. This won’t affect most of the popular material—big libraries have just about every major work held by a smaller library, so the small libraries’ records are redundant in these instances—but it does mean that more obscure works collected by smaller libraries, representing local authors and regional historical resources, may not be included.

This sort of limitation affects everyone from amateur genealogists to academic researchers. For instance, I have a friend who is writing her doctoral thesis on the history of illness in the counties surrounding Philadelphia. Almost none of the libraries, archives, and historical societies she is relying on have shared their catalogs with OCLC. This means she must make use of each of these collections individually, usually in person, and spend time learning how each collection is organized. This is the research equivalent of using a manual typewriter instead of a MacBook Pro to type her dissertation, and represents a failure to make the best possible use of available technology. These collections’ records should be included in WorldCat.org.

This kind of wasted opportunity to assist researchers is one major disadvantage of WorldCat.org’s omission of smaller libraries’ holdings. The other major problem arises when researchers try to make use of one WorldCat.org’s signature features. When users search for an item in WorldCat.org, they can select a tab labeled “Libraries,” which takes them to a list of local libraries that have that item in their collection. However, only libraries that share their records with OCLC are listed. For example, search for Daemon: a novel by Leinad Zeraus and select the Libraries tab. WorldCat.org displays ten libraries where you can find this book, in descending order of proximity. It would be natural for WorldCat.org visitors to infer that these are the ten closest libraries that have this book. Unfortunately, that’s probably not the case. Instead, WorldCat.org is displaying the ten closest libraries that share their records with WorldCat. Users who believe that WorldCat.org is helping them search their nearby libraries may be led to believe that their local libraries don’t have any books at all—or, at least, none of the books they’re hoping to find.

Of course, it’s possible that some libraries may not want their records included in WorldCat.org. I’m not sure why they would feel that way, aside from the recent hullabaloo over licensing which appears to be getting increasingly heated. However, the library where I work very much wants its records in WorldCat.org so that our neighbors in town can use it as an alternative way of looking for the books that are available in their local library.

OCLC markets WorldCat and other services through a network of regional service providers. The provider for our area is PALINET, so if we want to get our records into WorldCat, we have to go through PALINET. Unfortunately, between OCLC and PALINET, a sort of “if you have to ask, you can’t afford it” pricing structure seems to have emerged for getting records included in WorldCat.org.

I don’t think this is anyone’s fault. Everyone I’ve met at OCLC and PALINET is smart, dedicated, and helpful. My guess is that it’s more like Kate Sheehan’s post office story in which her attempt to pick up a package left her feeling “broken or inept.” That’s certainly how I felt after spending a month exchanging emails with PALINET. At the end I was so confused that it just didn’t seem worth bothering to get an accurate price to take to my board, because the one thing about which I was relatively certain was that we didn’t have enough money to share our records on the WorldCat.org website.

The folks at OCLC seem to be working hard to remedy this situation. I have faith that they’ll get there. But until they do, there will probably be a lot of libraries that would like to share their records in WorldCat.org and either can’t afford it or can’t figure out if they can. That means researchers are going to have to keep working harder than necessary, WorldCat.org users will keep being misled by its Libraries tab, and frustrated libraries may find themselves looking for more accommodating partners.

Open Library

Along with OCLC’s WorldCat.org, Open Library is one of two major nonprofit initiatives centered on creating a universal book catalog: its goal is a page for every book ever published, and to enable those pages to be updated by users, just as LibraryThing or Wikipedia pages are edited by site visitors. Since its founding in July, 2007, it has added over 30 million records to its book database.

For now, Open Library may be best known for its founder, Brewster Kahle, and its technical lead, Aaron Swartz. Both are Internet celebrities and serial entrepreneurs, though both specialize in nonprofit startups. Kahle has sold companies to AOL and Amazon, but he is best known for his work on the Internet Archive, home of the Wayback Machine, which attempts to archive the entire Web. Swartz was a founder of Reddit, which was sold to Condé Nast, and a developer of RSS, which enables websites, most notably blogs, to deliver content directly to readers. Open Library is currently funded by the Internet Archive and the California State Library and is committed to remaining entirely free, right down to the code that runs the site, which it makes available through an open source license.

Unlike our experience with OCLC, sharing our records in Open Library was dead simple: I emailed Aaron Swartz and he replied that receiving our records “was cause for much rejoicing.” (I also emailed Tim Spalding at LibraryThing to see if he might be interested in our records, and I found out he was as well.) Open Library is actively soliciting these contributions from libraries. However, it could, potentially, get these records directly from library websites. The technology involved is pretty simple and fairly well understood.

For example, the library where I work recently introduced a new website that’s powered by Casey Bisson’s fantastic Scriblio project. To import the Collingswood Library’s old records into our new website, we had Scriblio visit the web page for each record in the old catalog and import its data into the Scriblio database, turning blah into beautiful. We also use scrib_availability to show website visitors if the book is on the shelf.

Open Library clearly has the technical knowledge to do something like this and, because just about every library has a web-based catalog, it could easily include every book from pretty much every library in its database, enabling site visitors to learn if their local library has the book they want. For now, Open Library’s book pages, LibraryThing’s book records, and Google’s About this book pages link to WorldCat.org. (Edit: I originally wrote that Google’s About this book pages did not link to WorldCat.org. In the future, I’ll try to remember to disable my Firefox extensions before making such claims.)

The issue isn’t technical; it’s legal and ethical. On behalf of the library where I work, I uploaded our records to archive.org, making it possible for Open Library to use them, and on behalf of my library I uploaded them into our Scriblio-based website. It seems unlikely that libraries will have their records aggregated without their permission, at least in the near future. However, it wouldn’t be surprising if Kahle or Swartz, instead of asking for our records, began asking for our permission: what if they came to us and asked if they could automatically index our catalogs, creating for free a service that costs libraries thousands of dollars through OCLC? Even non-OCLC libraries are used to sharing their records. Why wouldn’t they accept Open Library’s offer to create a universal catalog? For most libraries, there’s no downside, but there’s an enormous upside: a single website where the world could see their records, and a free hub they could use for sharing records with each other.

A Useful Amplification

In his 1992 Redesigning Library Services: A Manifesto, Michael Buckland writes that, “(f)rom an operational perspective the library catalog can be seen as a useful amplification of records that are unavoidably needed anyway. The information in a catalog can be useful in a variety of ways to library staff and library users. The difference between modern library catalogs and those before the late nineteenth century is essentially that the modern catalogs have a much larger bibliographical superstructure added to the locational information than had previously been the case.” In a nutshell, Buckland is saying, libraries decided that, since they had to keep a list of what they owned, they might as well describe each item and make sure they knew exactly where copies of it could be found. “With materials on paper, having copies stored locally is a necessary (though not a sufficient) condition for convenient access. With electronic materials, local storage may be desirable but is no longer necessary…. The answer is to shift from catalogs to union catalogs or linked catalogs…. Arguably the present day catalog… is more a product of the limitations of nineteenth century library technology than of present day opportunities.”

Between Amazon, Google, LibraryThing, WorldCat, and Open Library, we’re getting ever closer to setting aside nineteenth century models and to more fully taking advantage of present day opportunities. There is no technological reason preventing us from building a universal catalog that contains information on every book in existence and locates that book in every library that has a copy available for use.

We’re also closing in on having a digital scan of every book, making full-text searching possible, as well as concurrent, remote use of scarce resources (by which I mean, I can look at the text of a book on my screen while you’re looking at it on yours, a feature not available in a paper-based book, which is limited to being used in a single location and, generally, by a single user). It’s an exciting time to be a booklover, and it gives one hope that, with better resources available, books will begin to seem as accessible and vital as born-digital resources.

I like the alternatives that Amazon, Google, LibraryThing, WorldCat, and Open Library make available. I think each has made the other better, and I like having alternatives in researching books just as I like having FedEx, UPS, DHL, and the United States Postal Service available when I’m trying to send a package. I don’t think researchers are generally lazy, and I don’t think they want fewer options. What they want are a few really good choices, and they have them. It’s exciting for all of us that these good choices seem intent on becoming great ones.

Thanks to Tim Spalding and Aaron Swartz for reading an early draft of this article, and to my ItLwtLP colleague, Hilary Davis, for helping me with its final version.

Photo by Flickr member Bill Gracey

Recently I found out that it could be happening to all of us: Google and the Internet as a medium could indeed be changing the ways our brains function and process information. “As Marshall McLuhan pointed out in the 1960s,” writes Nicholas Carr in The Atlantic, “media are not just passive channels of information. They supply the stuff of thought, but they also shape the process of thought. And what the Net seems to be doing is chipping away at my capacity for concentration and contemplation.” Carr’s article in the July/August issue of The Atlantic, “Is Google Making Us Stupid?,” received some attention for accusing its readers of not being able to accomplish deep, sustained reading in the age of the Internet. According to the article, the Web is reprogramming our brains in a fundamental, biological way. (Note: for a smart, satirical look at the issue, check out Stephen Colbert’s interview with Carr).

The responses to Carr’s article came from both sides of the fence: those who agreed with with him and those who objected to the perceived insult to their intelligence. The Chronicle of Higher Education came out with three articles that expressed concern and agreement: “Your Brain on Google,” a compilation of somewhat ironic quotes from the Web, “On Stupidity,” an extended book review of “a cartload” of recent books on anti-intellectualism, and “On Stupidity, Part 2,” an English professor’s response to the problem. Meanwhile, The New York Times Technology section printed a counterpoint by Damon Darlin, “Technology Doesn’t Dumb Us Down. It Frees Our Minds,” that accused Carr of being a technophobe and insisted that “writing, printing, computing and Googling have only made it easier to think and communicate.”

The irony of the entire argument is encapsulated in the first two lines of the New York Times article: “Everyone has been talking about an article in The Atlantic magazine called ‘Is Google Making Us Stupid?’ Some subset of that group has actually read the 4,175-word article.” Darlin builds the satire by attempting to sum up Carr’s article in a Twitter “tweet” of less than 140 characters, but only skims the surface of the real irony: the likely truth that very few of the people discussing Carr’s article had been able to read the whole thing. There’s something amazing and a bit disturbing about a culture in which everyone’s opinion is equally important and valid, no matter whether or not one has even a basic knowledge of the subject.

As an academic librarian, I’m particularly interested in the implications for libraries of Carr’s article. Hand in hand with Carr’s concern about a growing inability to engage in deep reading is the equal possibility of a growing inability to engage in sustained research. Google leads us to believe that searching for information is easy when library research is complex, often frustrating, and full of twists and turns. So the next question is: does it have to be that way? It’s a given that library systems tend to be overly complicated, even for simple searches. The common refrain is: how can we be more like Google?

The followup question is: do we want to?

These days academic libraries are grasping at every possible product—from federated searching to LibraryThing—that might ease our students’ apparent impatience with the challenges of research. After all, the 2002 Pew Internet & American Life report, “The Internet Goes to College,” made it clear that our students rely on the Web first when they’re doing research, and generally use the library only as a latter resort. If academic libraries don’t make it easier for students to find relevant information for their course projects, they may not come at all. We may as well just hand Google Scholar the keys.

On the other hand, a recent study of the research practices of college students in the humanities and social sciences offered more heartening results. Alison J. Head’s article, “Beyond Google” in First Monday (later written up for September 2008’s College & Research Libraries) found that students are using libraries in greater numbers—and earlier in their searches—than the Pew Research Center would have us believe. Granted this was a study at a single, small, liberal arts college that doesn’t necessarily reflect the situation everywhere. But we can glean some optimism from the study, along with the requisite grain of salt.

On the positive side, academic libraries have the benefit of a captive audience of students whose professors often require the use of library resources. While we may hope that these requirements train students in the ways of deep research, the day-to-day interactions at any academic reference desk would indicate otherwise. Instead, a majority of students reflect a desire to find adequate sources for a given project as soon as possible, even if those sources are not ideal. Is it Google that has raised their expectations for how quickly an information search can be accomplished? A study from the British Library calls this a “truism in the age in which we live” that “crosses all generational boundaries in the digital environment…. The speed of new media has cultivated a lowered tolerance for delay.” The study goes on to say:

There is considerable evidence to support the view that many students do not explore information in any deep or reflective manner. The lack of any evaluative efforts on the part of information users has been documented…. According to Levin and Arafeh (2002) most students stop searching at ‘good enough’ rather than trying to find the best source etc. Some ‘view the Internet as a way to complete their schoolwork as quickly and painlessly as possible, with minimal effort and minimal engagement.’

English professor Thomas H. Benton’s personal observations are nearly identical. In “On Stupidity, Part 2,” he writes:

Essentially I see students having difficulty following or making extended analytical arguments. In particular, they tend to use easily obtained, superficial, and unreliable online sources as a way of satisfying minimal requirements for citations rather than seeking more authoritative sources in the library and online. Without much evidence at their disposal, they tend to fall back on their feelings, which are personal and, they think, beyond questioning.

The echo of Carr’s article in both of these quotes is unmistakable. Whether or not Google is actually changing the biology of our brains it is difficult to say, but it does seem possible that Google could be damaging our students’ ability or inclination to conduct real research.

I’m not blaming our students. It is not the fault of anyone in particular if they are losing the interest and ability to conduct complex research. They are products of their culture, just as we all are. Just as I am.

In fact, those of us currently in our early to mid-thirties are in a unique position to address this issue. You see, I didn’t grow up with computers, but computers and I grew up together. I can remember, back in grade school, Atari and I bumbling our way through Asteroids. In high school, America Online and I had our first heady experiences in online chat rooms. When I went to college my library’s young OPAC was incomplete and I still had to use the card catalog to find certain items. Computers were leaking into my research in college, but their effect was fragmented. Google was founded the year I graduated from college.

I grew up with computers, but I grew up knowing that they were fickle, fallible, and constantly changing. I still have a collection of old floppy disks with files I will never be able to access again. I greatly enjoy technology, but I maintain a certain skepticism about it.

That said, I had to make a conscious effort to read Nicholas Carr’s article all the way through. The first time I linked to it, I skimmed the first few paragraphs and bookmarked it. The second time, I skimmed further into the text. I didn’t actually read the whole thing until I chuckled at Darlin’s observation on how few had read it and realized that I was not one of them.

What happens to our libraries in a culture where sustained reading and deep research are skills that our students and patrons increasingly do not value? There is no easy answer, but the most critical thing we can do is reflect passion for our work and share it with our students. Benton writes, “Effective teaching requires embodying the joy of learning — particularly through lectures and spirited discussions — that made us become professors in the first place. It’s extremely hard, but teachers have been doing it for generations.”

Notice his admission that playing such a role is “extremely hard”; we can all appreciate his honesty there. It is hard to be an intellectual in a culture that values actors over educators. It is hard to face a constant onslaught of superficial research when we know how much richer and more inspiring information can be. But the payoff comes when we open the door and a student steps through, leaving Google aside for the moment, to consider the wealth of research tools at their disposal that they never knew existed.

If only it happened more often.

Many thanks to my ITLWTLP colleagues Derik and Brett, and to Rick Stoddart, Tom Hillard, Ellie Dworak, and Elaine Watson for offering feedback that helped shape this post.