Reference & Instruction Librarian
Extended Services Librarian
Mount Saint Vincent University
Lawrence J. Duggan
Cape Breton University
Mount Saint Vincent University
New and alternative methods of service delivery in libraries must be evaluated in order to offer optimum assistance to patrons. In early 2007, the Novanet1 Live Help committee (the committee responsible for administration and staffing of the consortium's online reference service) decided that the time was right for an evaluation study of alternative online reference platforms. Frustrated with the faulty performance of our licensed Virtual Reference (VR) software, Live Help staff began to look at other tools with which to offer this valued service. Much like the experiences of other online reference services, Live Help did not realize the functionality promised by the VR platform. The co-browse feature had proved so unreliable that we had ceased trying to use it. Defective statistics, dropped sessions and the constant, uncontrollable refreshing of the chat interface were consistent problems. Judging by the literature and our knowledge of other institutions' experience with online reference, fickle VR platforms appeared to be the norm. Pulliam and McMullen of Rhode Island's Higher Education Library Information Network reported that "the software itself proved to be the greatest impediment to offering the service. VR software is cumbersome to use" (63).
The Live Help committee decided to focus our study on freely available Instant Messaging (IM) products; ultimately, we decided that a trial of a multi-protocol IM platform that could simultaneously accept sessions from several major IM services, along with a web-based IM chat widget, would allow us to best evaluate the suitability and potential of IM for the Live Help service. This paper reports on the findings of this study. Throughout this article we will use 'VR software' to refer to the commercial virtual reference platforms, and 'IM' to denote the freely available instant messaging clients widely used by a large proportion of post-secondary students.
The use of IM is ubiquitous among university and community college library users. A 2004 PEW study reported that 42% of internet users use IM and 46% of the Gen Y age group (18-27 years) send instant messages more often than email (Shiu and Lenhart iii). More recently, a 2006 survey of 286 Canadian university students found that 83% of students had been using IM for the past 4 years or more, with 67% using it daily to communicate (Quan Haase 679-680). For evidence of the application of IM for online library services, one has only to look at Library success: A best practices wiki for an ever growing list of libraries using IM for online reference (Farkas "Online Reference"). In an article titled, "Web-based Chat vs. Instant Messaging: Who Wins?" licensed, virtual reference software and IM were evaluated toe-to-toe, with IM coming out the unanimous winner under most criteria (Houghton and Schmidt 5).
Many libraries have performed pilots of IM reference services both individually and integrated with VR software. In some of these cases, such as the Kresge Business Administration Library at the Ross School of Business, IM was the fulcrum upon which the online reference service finally became a success (Doan and Ferry 20). The University of Illinois at Urbana Champaign Library recognized that requiring patrons to use "software communication tools they are unfamiliar with does not necessarily create a formula for reaching the broadest user base possible", and so piloted an IM service in tandem with their VR software-based service in the spring of 2005 (Ward and Kern 428). The result was an overall increase in sessions, with IM receiving the majority of questions. Here in Canada, the University of Guelph reported on the success of their MSN Messenger-based IM service at the 2007 Canadian Library Association Conference (Lupien and Rourke).
By the time Novanet Live Help considered IM for online reference, chat widgets - embeddable, online messaging boxes that increase the usability of IM - had arrived on the scene. Memorial University of Newfoundland Library has had anecdotal success using IM widgets for online reference (Gordon; Pretty), but a literature search prior to this trial found no reports of studies on the use of chat widgets by academic libraries in a consortial environment.
We acknowledged from the start that piloting an IM reference service was not going to be simple. While we anticipated that IM and chat widgets would appeal to more patrons than the VR software, we weren't confident that the service itself could cope with IM's lack of the key VR features we had become accustomed to; in essence, what the Live Help service had been built upon. These features include: automatic statistics gathering, transcript logs and transcript delivery, not to mention multiple log-ins (hitherto considered essential for a consortially-staffed service).
To better understand the workings of the IM trial, a few explanations regarding IM services are required. Popular examples of freely available, commercial IM services include AOL Instant Messaging (AIM), MSN Messenger, Yahoo Messenger and Google Talk. In order to use these IM services, users must create an account. Until recently, the user had to download a program to their computer in order to use most of these services. Now, most services have a web-based IM system available. To communicate with others, the user must add their "buddies" accounts to their buddy or contacts list. These buddies must also use the same IM service.
The lack of uniformity in IM services currently used by library patrons poses a problem: How can you offer an online reference service to only those who use the same IM service that you do? The answer is by using a multi-protocol IM program. Examples include Meebo, Pidgin and Trillian. These services allow you to send and receive messages from several of the commercial IM services simultaneously through one single interface, thereby allowing you to IM with all your buddies in all your IM service accounts.
Having solved the problem of lack of uniformity in IM services, there also remains another a problem: How can you offer an online service to your patrons who do not subscribe to an IM service? The best current answer is to use a chat widget. This is a simple messaging box that can be embedded into any webpage. The widget increases the usability of IM because the user does not require an account, a download, or even prior IM experience. Two popular examples are MeeboMe and Plugoo. Depending on the chat widget service you choose, this widget account may also be added to your multi-protocol IM program.
The Live Help committee tested various free IM platforms, multi-protocol services and chat widgets and evaluated these against a weighted list of desired features. We decided that the combination of a multi-protocol IM service combined with a chat widget was the optimal IM reference trial scenario. We chose Pidgin as the back-end platform due to its many available features. Pidgin (previously known as GAIM) is a free, open source, desktop-based, multi-protocol IM service. It allowed Live Help staff to handle accounts from all of the major IM services (AIM, MSN Messenger, Yahoo Messenger, Google Talk) in a single interface. The trial also added a Meebo account to Pidgin; permitting the use of a MeeboMe chat widget (Figure 1) to offer direct service to users through our webpages.
Figure 1 - MeeboMe chat widget
We designed a study to evaluate IM by offering it concurrently with our VR software for the winter semester (January - April) of 2008. The main objective of the trial was to determine if an IM-based reference service would appeal to more patrons and provide sufficient back-end support for our service framework. VR software had provided us with administrative features we had considered pertinent to the service. The trial would reveal if patron preference warranted a switch to an IM service despite the loss of these features.
We created Live Help accounts with four major IM services: AIM, MSN, Google Talk, and Yahoo Messenger. A Meebo account was arranged and a MeeboMe chat widget created. HTML code for the chat widget was disseminated amongst the Novanet institutions to be embedded in contact pages. In all instances where users were previously offered a link to the existing VR service, they were now also offered a link to the IM trial service, with both links side-by-side in as many cases as possible. For integration within our OPAC, rather than placing the widget directly in the interface, a linkable logo was created that launched the chat widget in its own window. (Placing the widget directly in the OPAC would be technically challenging because the OPAC interface design does not allow users to navigate in the same window without refreshing the widget.)
Service providers volunteered to staff Live Help from both services (VR and IM) simultaneously and additional volunteers were recruited to assist during busier times. Supporting material was placed on the Live Help committee's wiki and training and on-going troubleshooting was provided. Since the IM service did not allow for multiple service-provider log-ins, a step-by-step procedure for logging in and out of the IM service during shift changeovers was agreed upon. We had several IM accounts in operation, however, so this allowed us to have two (or more) service providers online at once, each staffing separate IM accounts.
Data-gathering instruments were deployed to measure the success (or failure) of the IM service. Although the existing VR software had built-in features to collect and store service statistics, the IM service needed to be supplemented with a statistics database into which Live Help service providers were required to enter their session data. Through our investigations of other libraries' trials of IM, we found a freely available online reference statistics database developed by the University of Nevada Las Vegas.2 This open-source solution allowed us to create categories within the database that mirrored the statistical measures provided by the VR service.
The intention was to compare the data gathered from both services with winter 2007 statistics to determine:
If there was an increase in the number of Live Help sessions
Which service (IM or VR) was used the most
Whether one service was preferred for certain types of questions (e.g., circulation, access, technical, quick reference, research assistance, etc.)
Patron surveys were conducted for each service to determine:
Patron type (undergraduate, graduate, faculty, staff, other)
Satisfaction with the service
Preferred method of accessing reference services (in-person, phone, email, IM, VR, other)
If patrons used their individual IM service or the chat widget. If they used their own IM service, did they add the Live Help service to their IM buddy list?
Service provider surveys were conducted to determine:
Satisfaction with the services
Any technical difficulties or additional notes for qualitative consideration in the study
This valuable information allowed us to capture very interesting trial results, which resulted in a concrete recommendation to the Novanet Board of Directors.
We achieved healthy response rates to our surveys. 13% of VR patrons, 20% of IM patrons, and 25% of service providers completed the appropriate survey (Figure 2).
The patron response rates are themselves revealing. The VR service allowed us to add an automatic patron survey prompt. The IM service did not provide this feature, so a direct request from the service provider with directions to the online survey was required. We were concerned that this discrepancy between survey prompts may have resulted in a higher response rate from the VR patrons. It is interesting to see that the opposite occurred. We surmise that this is due to the direct and "live" request from the service provider.
Figure 2 - Survey response rates
22% more questions were received by the IM service than the VR service.
Total sessions Winter 2008 VR = 251 45% total questions
Total sessions Winter 2008 IM = 305 55% total questions
The IM figure is more impressive when one considers that the IM service was new to our patrons and to our service providers. As with any new online service, technical difficulties and "growing pains" were abundant during the first few weeks of the IM service, resulting in some dropped sessions, offline periods and perhaps some disappointed patrons. As mentioned earlier, the IM service did not permit multiple log-ins and service providers needed to follow strict logging-in procedures. If a second service provider logged in to the IM service, the original service provider would lose connection and any sessions in progress. Service providers eventually grew accustomed to the careful logging-in procedures and lost sessions were minimized.
The Live Help service experienced an impressive 85% increase in number of sessions compared to the 2007 winter term, but part of this increase was a result of our newly extended service hours. To better measure a service increase attributable to the trial of IM, we excluded the sessions accepted during shifts not offered in both years. This adjusted our totals to:
Total sessions Winter 2007 VR (adjusted) = 295
Total sessions Winter 2008 VR & IM (adjusted) = 425
These totals reflect a 44% increase in online reference sessions. Live Help is not yet a formal service and has yet to be formally promoted and marketed to Novanet patrons. Since promotional and marketing efforts have been minimal to date and consistent from winter 2007 to winter 2008, we feel this 44% increase in online reference sessions could be attributed to the trial of IM.
Patron satisfaction was measured by the first question in both the VR and IM patron surveys (Figure 3). We acknowledge that responses represent a blend of impressions of the service itself and the mechanism for receiving the service. Nevertheless, respondents reported higher satisfaction with IM (with both services receiving a high satisfaction rate). 92% of IM patrons were very satisfied or satisfied with the service; 79% of VR patrons were very satisfied or satisfied.
Figure 3 - Patron satisfaction with IM and VR
We next asked patrons about their preferred method of acquiring reference assistance. We acknowledge that since we surveyed online patrons, the response to this question would reflect some bias toward online assistance. Given Cummings, Cummings and Frederiksen's conclusion that online patrons use IM for personal communication rather than academic or business purposes (94), the preferences of online patrons are very relevant to this study. We must also recognize that many library patrons are not aware that the Live Help service exists and, consequently, are not included in this study.
Combining responses from both the IM and VR survey respondents, and identifying the responses by patron type, we were able to gain some insight into patron preferences (Figure 4). Response rates from "faculty" and "other" were significantly low (5 faculty responses, 13 responses by others) so we couldn't make any conclusions about their preference and eliminated them from our results.
Figure 4 - Patron preferences (Undergraduate vs. Graduate)
The IM widget was the preferred method of acquiring reference assistance by the majority of survey respondents, both graduate and undergraduate. This was followed by face-to-face reference help. Several students commented on the option of online versus in-person reference services and this is discussed in the Patron Comments section (see below). In regards to the other methods of chat reference, more undergraduates preferred using their own IM account to contact the library over the VR platform; however, the opposite was the case for graduate students. Service by phone was not a popular choice.
Perhaps the most interesting observation here -- and a very clear statement from our students -- undergraduates do not want to use email to acquire reference help. 0% of our undergraduate respondents stated a preference for email. Graduate students, on the other hand, indicated no preference between email and using their own IM accounts. This finding is in contrast to those of Lancaster et al., who concluded that while students may use IM extensively for personal communication, e-mail is preferred for academic purposes. (19)
In order to observe preferences by service, we created charts for users of both services (Figure 5). As expected, respondents stated a preference for the service they chose to use; however, the preferences reported by the VR service respondents are spread out across the choices more than those reported by the IM respondents. IM respondents appear to be more unified in their preference for IM, with face-to-face service being the second choice. We also noted the preference rate for the opposing services: there is a higher IM preference amongst the VR service respondents than VR service preference amongst the IM respondents.
Figure 5 - Patron preferences by service chosen
We included an option for "other" methods of acquiring help, expecting that some patrons would suggest text messaging or other methods in the free-form text response box provided. The few who selected "other", used it as an opportunity to restate that they preferred a combination of in-person and online services. No one mentioned text-messaging.
Survey respondents had the opportunity to comment at the end of the survey, and many did (Figure 6). 75% of IM survey respondents contributed comments, compared to 55% of the VR service respondents. In keeping with the higher satisfaction rate reported by the IM respondents, many of their comments were positive about the service. More IM respondents than VR respondents suggested improvements to the service, mostly concerning placement and promotion.
Figure 6 -- Percentage of patron comments by service
Following are some examples of patron comments. Generally speaking, patrons from both services expressed an appreciation for in-person service while simultaneously valuing the online option for when they are off-campus. This indicates that face-to-face and IM reference are complementary rather than competing services. It is clear from these comments that patrons value the reference help they receive in the online environment:
We analyzed our service statistics to see if there was a connection between the type of reference questions asked and the chat platform used. After careful analysis we concluded that the IM service experienced only slightly higher numbers of instruction and research-assistance sessions. It was not an impressive difference, but does reflect what other IM pilot studies have observed: patrons seem comfortable using IM for longer questions - an outcome that has surprised others during their studies. Doan and Ferry reported receiving "more complex and research-oriented questions than the ones we tended to get in person" (21). Ward and Kern observed a tendency among IM patrons to fetch some information, leave the session to process it, and then return for more help. They posit: "this behavior perhaps mimics users' other IM conversations with friends and family" (426). We suggest that the IM tool simply lends itself well to this "coming and going" behaviour, sometimes with little or no farewells from patrons.
The fourth question on the IM patron survey asked if the patron added Live Help to his/her IM buddy list. This question was directed only to those who had contacted the service via their own IM account. Our intention here was two-pronged: 1) to determine what percentage of our IM patrons used their own IM service rather than the chat widget, and 2) to find out how many of these patrons added us to their buddy lists. The study conducted by Doan and Ferry, which did not use a chat widget, reported that many patrons had added the service to their buddy lists (21).
We feel that our wording of this particular survey question was flawed as the response rate of 66% (that is to suggest: 66% of the IM patrons used their own IM service rather than the chat widget) was not supported by our actual experience staffing the service. Based on this experience and preferences stated in the surveys, we instead conclude that the majority of users used the chat widget, and that our respondents did not understand that only patrons who used their own IM account to contact the service were required to answer this question. Of the 66% who responded, only 7.5% stated that they had added us to their buddy list. We feel this percentage is more indicative of the proportion of respondents who used their own IM service rather than the chat widget.
Based on examples of IM-speak in presentations by Lupien and Rourke and by Etches-Johnson, we anticipated receiving a great deal of IM-speak during our online sessions. It is interesting to note that upon review of the pilot, and in consultation with all service providers, we can state that we received very little IM-speak from our patrons, and instead tended to use it amongst ourselves. Upon further review of the literature this experience is not without precedent. In a study of IM communication among students, Baron found that abbreviations and contractions made up less than 1% of total words used in IM communication ("See you Online" 411). In addition, it would appear that students are adapting their online communication to different contexts and audiences (Lewis and Fabos 495; Baron "Instant Messaging" 31; Maness 34). Our chat widget was predominantly used during our pilot, and with a title on the widget of "IM a Librarian", we assume that patrons felt the need to be more formal during their session, reserving their IM-speak for casual conversations in their own IM services.
In order to capture user satisfaction of all our stakeholders, we chose to survey ourselves, the service providers (Figure 7). Service providers were asked to complete the survey after each shift. In the survey, they were asked to rate their satisfaction with the service tool used and to provide any comments. As with the patrons' responses, service providers were somewhat more satisfied with the IM service.
Figure 7 - Service provider satisfaction
32% of service provider respondents were very satisfied with IM, whereas 22% were very satisfied with the VR service. 14% were dissatisfied with VR whereas only 3% were dissatisfied with IM. Closer analysis of the satisfaction levels reported over time showed an increasing satisfaction with the IM service over the course of the trial. It must be remembered that the first few weeks of the service were not without some frustrating technical glitches and oversights as well as learning curves for service providers. Service provider comments support the rising levels of satisfaction:
Based on service statistics, the high levels of patron satisfaction and preference for the IM service (particularly the IM widget), as well as service provider feedback, the Live Help committee unanimously supported a switch to IM as our online reference platform. The Live Help service providers opted not to retain the existing VR service as there was no clear indication that it was preferred by any of the stakeholders. The universality of the chat widget obviates any need for additional chat software to serve patrons who do not use IM. The Live Help committee put forward the recommendation and the Novanet Board of Directors approved the switch to IM for the fall of 2008.
This trial of IM also provided insight into how a trial should be run. Based on our experience we recommend:
Thorough testing before trial launch to uncover potential glitches
Refresher training for service providers just before service launch particularly if the launch date occurs immediately following a break/holiday period
Thorough pre-testing of the survey tool to ensure that questions are clearly understood by the target audience
Although a switch to an IM-supported online reference service has been approved, the Live Help committee recognizes some limitations with the IM software that is currently available. These confines were acknowledged at the outset of the trial and include the lack of multiple log-in permissions, which had been considered essential by the Live Help committee for the operation of this consortial service. An IM platform that allows more than one service provider to staff the service at one time is desired and will be sought in the future. In addition, the committee acknowledges that by switching to IM, it loses the automatic reporting procedures provided by licensed virtual reference software products, and a system of self-reporting procedures is required to collect and keep valuable service statistics.
Before finally deciding upon a multi-protocol IM service and chat widget to be used for the ongoing service, the committee will revisit the evaluation matrix prepared at the outset of this investigation to re-assign weights to desired features in light of our trial experience, and re-evaluate all available tools. The Live Help committee must stay abreast of new communication technologies, including licensed, free, and open-source products, in order to provide what is best for our patrons. As Houghton and Schmidt concluded, libraries will "realize that other alternatives, of which IM is only the first, offer a much better reference environment for their users."(30) The Novanet Live Help committee expects that evaluation and trials of new services and delivery platforms will be an ongoing process in the coming years.
This article is the result of a study performed across several post-secondary institutions in Nova Scotia that would have been impossible without the enthusiastic, dedicated library staff who supported it. The authors wish to acknowledge and thank all the Novanet Live Help service providers, the Novanet office staff, and the technical support staff who assisted us.
Baron, Naomi S. "Instant Messaging and the Future of Language." Communications of the ACM 48.7 (2005): 29-31. Business Source Premier. EBSCOhost. Mount Saint Vincent University Library, Halifax, NS. 15 July 2008 <http://web.ebscohost.com/>.
---. "See You Online: Gender Issues in College Student use of Instant Messaging." Journal of Language and Social Psychology 23 (2004): 397-423. 14 July 2008 <http://jls.sagepub.com/cgi/content/abstract/23/4/397>.
Cummings, Joel, Lara Cummings, and Linda Frederiksen. "User Preferences in Reference Services: Virtual Reference and Academic Libraries." portal: Libraries and the Academy 7 (2007): 81-96. 15 July 2008 <http://muse.jhu.edu/journals/portal_libraries_and_the_academy/v007/7.1cummings.html>.
Doan, Tomalee, and Kristine Ferry. "Instant Messaging (IM): Providing Services and Enhancing Communication." Journal of Business & Finance Librarianship 12.2 (2006): 17-22 Education Research Complete. EBSCOhost. Mount Saint Vincent University Library, Halifax, NS. 15 July 2008 <http://search.ebscohost.com>.
Etches-Johnson, Amanda. "IM Reference @ Your Library." Education Institute. (2007) 21 July 2008 <http://blogwithoutalibrary.net/talk/ei2007/im_fall.pdf>.
Farkas, Meredith. (Creator). "Online Reference" Library success: A best practices wiki. 18 July 2008 <http://www.libsuccess.org/index.php?title=Online_Reference>.
Gordon, Shannon. "RE: Questions re Reference IM." Email to Linda Bedwell. 05 June 2007.
Houghton, Sarah, and Aaron Schmidt. "Web-Based Chat VS. Instant Messaging: Who Wins?" Online 29.4 (2005): 26-30. Research Library. ProQuest. Mount Saint Vincent University, Halifax, NS. 15 July 2008 <http://www.proquest.com/>.
Lancaster, Sean, et al. "The Selection of Instant Messaging or e-Mail: College Students' Perspective for Computer Communication." Information Management & Computer Security 15 (2007): 5-22 15 July 2008 <http://www.emeraldinsight.com/10.1108/09685220710738750>.
Lewis, Cynthia, and Bettina Fabos. "Instant Messaging, Literacies, and Social Identities." Reading Research Quarterly 40 (2005): 470-501. Research Library. ProQuest. Mount Saint Vincent University, Halifax, NS. 13 July 2008 <http://www.proquest.com/>.
Lupien, Pascal, and Lorna Rourke. "Learning from Chatting: Keeping up or leading the way?" Canadian Library Association (CLA) National Conference, St. John's Newfoundland, 23-26 May 2007.
Maness, Jack M. "A Linguistic Analysis of Chat Reference Conversations with 18-24 Year-Old College Students." Journal of Academic Librarianship 34 (2008): 31-8. ScienceDirect. Mount Saint Vincent University, Halifax, NS. 15 July 2008 <http://www.sciencedirect.com/>.
Pretty, Heather. "RE: Questions re Reference IM." Email to Linda Bedwell. 04 June 2007.
Pulliam, Beatrice R., and Susan McMullen. "RYT: Are You there? A Consortium's Switch to IM Reference." College & Undergraduate Libraries 13.4 (2006): 55-73
Quan-Haase, Anabel. "University Students' Local and Distant Social Ties: Using and Integrating Modes of Communication on Campus." Information, Communication & Society 10 (2007): 671-693. 14 July 2008 <http://www.informaworld.com/10.1080/13691180701658020>.
Shiu, Eulynn, and Amanda Lenhart. "How Americans use instant messaging." Pew Internet and American Life Project. 2004. 15 July 2008 <http://www.pewinternet.org/pdfs/PIP_Instantmessage_Report.pdf>.
Ward, David, and M. Kathleen Kern. "Combining IM and Vendor-Based Chat: A Report from the Frontlines of an Integrated Service." portal: Libraries and the Academy 6 (2006): 417-429. Research Library. ProQuest. Mount Saint Vincent University, Halifax, NS. 15 July 2008 <http://www.proquest.com/>.
1 Novanet (http://novanet.ns.ca) is a consortium consisting of ten University and Community College library systems located in Nova Scotia, Canada.