Towards a User-Centred Approach to Digital Libraries
The National Library of Finland led the organisation of this conference to bring together librarians and researchers from around the world to discuss progress with digital libraries. The aims were to explore how users were responding to digital services and to examine how services could be made more 'user-centred'. The conference was attended by 200 delegates from 23 countries. The Powerpoint presentations of speakers have been placed on the finelib Web site [1] and some of the papers have been published in the electronic journal Information Research [2].
Many of the papers reported evaluations of user behaviour and it proved a good forum to judge the value of different evaluation methods and to assess the progress that is being made around the world to make digital libraries a reality. Since the focus was on libraries, the conference also provided some insights into how librarians currently feel about the great change that digital technology is bringing to their profession.
The papers were organised into five themes:
- Methodologies of user studies
- Usefulness and validity of digital resources in research
- Impact of digital libraries on student learning
- Evaluation of digital library programmes and services
- Visions of the Future
Methodologies of user studies
Linda Banwell from the JISC-funded Jubilee Project at Northumbria University gave the keynote address for this theme. The Jubilee project is developing a range of evaluation methods for librarians to use and Linda was keen to encourage libraries to use a mixture of quantitative and qualitative methods in order to understand both what users were doing and what was important to them. Philip Pothen, Communications Manager for JISC, gave an overview of UK projects that were evaluating user behaviour. I attended the conference to report our evaluation work on the Zetoc Service and Lesley Huxley reported work on SOSIG, both also funded by JISC. There was some envy from delegates of other countries that in the UK we recognise the importance of user evaluation and have a wide-ranging programme for funding evaluation work.
Linda Banwell's call for different types of evaluation was echoed in a number of papers that compared the results of different types of evaluation. Ellen Hoffmann reported a study of ECO (Early Canadian Online) which used a range of methods including user surveys, server logs, focus groups and interface assessments. Mark Notess from Indiana, similarly used a mixture of methods including session logging, a questionnaire and contextual inquiry, (including observation, interviews and focus groups), in a study of Variations, a music teaching system. They reported that the more quantitative methods allowed usage trends to be followed but it was the qualitative methods that enabled explanatory models of users to be developed which could guide future service development.
Usefulness and validity of digital resources in research
Carol Tenopir from Tennessee gave the keynote address for this theme and focussed on evidence that scholars are reading more because of the availability of digital resources. She was careful, however, to highlight the differences between disciplines. Sanna Talja from Tampere reported a qualitative study of scholars from four disciplines that looked at why there were differences in take-up of electronic journals between the disciplines. The study concluded that it was factors such as domain size, degree of scatter and relevance criteria rather than familiarity with the technology. Other papers in this theme gave a broadly optimistic view of the usefulness of digital resources and reported that scholars were beginning to exploit these resources. But they repeatedly came back to one point; scholars may be able to access a great quantity of information but what they really wanted to do was assess quality and this was much harder to do in the anarchic world of Web resources.
Impact of digital libraries on student learning
Sakari Karjalainen, from the Finnish Ministry of Education, led this theme by describing the national strategy in Finland to develop an information society, part of which includes a national plan for a digital library. Delegates from other countries were impressed by the long-term, integrated approach being taken in Finland and decried the fragmented approach in their own countries. Many had been able to get funding to launch new services but not to sustain or develop them further. The importance of evaluating user responses to a service was widely acknowledged but it carried the implication that there would be resources to modify and develop the service to meet emergent requirements. Service providers often ran into difficulties because there was no further funding for iterative development.
A number of papers considered student responses to digital libraries. John Colvin and Judith Keene from University College, Worcester, reported on a study in which e-journal access was embedded in teaching materials for business studies students. This proved a valuable service but the quality depended on good co-operation between academics and library staff. A common concern about student use of electronic resources was a fear of the dominance of Google. Many students now come to university knowing that you get information by searching the Internet. They tend to use Google and the other Internet search engines rather than use the specially developed services of the digital library - and they get results. The worry (again) is that they get a mixture of good and bad quality results and modern students have to be taught how to establish the quality of what they find on the Internet. Lesley Huxley and Angela Joyce from Bristol University, reporting user behaviour with SOSIG (Social Sciences Information Gateway), were also concerned about Google. Any subject portal has to keep pace with the facilities offered by general Internet services and, whilst SOSIG can offer quality services to social scientists, it was more difficult to sustain the iterative development costs to keep pace with the services offered by commercial services.
Evaluation of digital library programmes and services
In this theme there were examples of different kinds of evaluation and evaluations of different kinds of services. A number of macro-studies of users were reported, some over a period of years, which gave some insights into the progress of adoption of digital services. Grace Saw, for example, reported the changing needs of the user community of the University of Queensland. She drew attention to the continuous need for the iterative and evolutionary development of library services and gave examples of the wide range of evaluation methods they used to assess their services. Kristiina Hormia-Poutanen, head of the National Digital Library of Finland, gave a paper about the use of the national service from 1998 to 2002. The results show a gradual increase in usage of the service, especially amongst scientists, and gradual increases in levels of user satisfaction. However, the percentage of those willing to cancel print subscriptions when an e-journal is available has only increased in this period from 49% to 58%. Brinley Franklin from Connecticut reported a review of Web-based services at four health science libraries and two large campuses from 1998 to 2003. By 2003 the ratio of 'remote' users to in-house users in the health sciences libraries was 4:1 and in the main campuses 1.3:1, although few of the on-campus users were now visiting the library. This change in the location of the users was reflected in a study by Kirsti Nilsen from Ontario. He reported the challenges for the library reference desk when it no longer met users in the library but at a 'virtual reference desk' because the users were remote from the library. The overall changes were reflected in a paper by Waldomiro Vergueiro reporting a survey of users in San Paulo. In general users, especially from the hard sciences, were pleased with the electronic resources available to them, but still made use of and continued to value the print stocks in the library. Users liked the digital library but a hybrid rather than a wholly virtual library remained essential. The authors of the survey in particular noted that most users had a very limited understanding of the electronic services available and were rather naïve in their use of them.
These broader surveys were complemented by more focused studies of particular services which used a mixture of quantitative and qualitative methods to explore usage trends, user needs and motivations. The paper I presented on the zetoc evaluation [3] was in this category. zetoc provides alert and search facilities for British Library holdings to users across UK Universities. Our evaluation used log statistics, two questionnaire surveys and an interview programme to develop a broad understanding of usage trends and an in-depth understanding of user strategies and motives. The full evaluation report is available on the zetoc Web site [4] and it shows that zetoc is a very successful service, well regarded by its users as a comprehensive current awareness service. Most of the discussion of the paper was about our finding that there are a minority of active users but a majority of more passive users. Others agreed this was a common pattern and said it was important not to base evaluation (and service changes) just on the views and behaviour of activists. It was also important to understand the passive users who may have different needs and problems.
Others also drew attention to the 'innovators' and the 'laggards' amongst the user population. Paola Garigiulo from Rome described a major study of 25 universities using a collection of 3,566 e-journals. Like the earlier SuperJournal study [5], a cluster analysis was performed on the results of a questionnaire survey of 1,305 users. It showed that, whilst 28% could be described as 'innovators without hesitation', (mainly engineers), the majority had reservations, used e-journals sparingly and still valued the print tradition.
A number of studies looked at efforts by libraries to create personalised services for their users, for example, Jette Hyldegaard from Copenhagen and Ursula Jutzi from Zurich. In general they found that users valued them for convenience but their over-riding need was for the service to stay relevant to their needs without the need for them to spend their own time managing the resources.
A number of studies examined the usability to digital services. While agreeing the importance of usability, several authors warned of the danger that studies could give misleading results. Kai Oorni from the University of Oulu reported that usability studies had low predictive validity and recommended that studies should be contextually defined. Teal Anderson from John Hopkins University, Baltimore, reported the use of a range of methods on a variety of services. He made a number of specific recommendations to ensure the results were accurate; that, for example, task scripts reflected the diversity of user groups and the system was tested for its usability when the user had unexpected as well as standard tasks.
Visions of the future
Lars Bjornshague from Lund University gave the keynote address for the final theme by drawing attention to the proliferation of services that now existed to make the digital library a reality. These services were poorly integrated which meant the digital library was far from 'user-friendly'. The big technical challenge for the future was to achieve a seamless integration of all these services for the benefit of users. (There was little discussion at the conference of integrating mechanisms such as open urls but this was not a gathering of technical specialists). Lars also felt that these developments carried with them big organisational challenges for libraries, changing their roles in relation to their users and in relation to the providers of resources and services. There was some pessimism voiced about the power of individual libraries in these times of great change. It was felt that 'Digital Darwinism' might prevail and only the 'fittest' services would survive. It seemed unlikely that under-funded services developed by libraries would survive against the commercial forces of the publishers and others. There were calls for libraries to co-operate but no mechanisms were proposed. One of the final papers was by Bo-Christer Björk from Helsinki who reviewed the progress of the open access movement and asked whether the future would be free access to scholarly information. The audience were clearly in favour of this approach and hopeful that it would develop as much momentum as the open source and free software movement. However, many were fearful that the commercial interests of the big publishers would in fact determine the future.
Conclusions
My overall impression was of a community that is excited by the digital library but expected, and perhaps hoped, that libraries would remain 'hybrid'. There was a desire for libraries to continue to be the local centre of services to their user community. However, they saw major organisational and business changes coming in the wake of technical developments and many were fearful that libraries would lose their role as the interface to local communities of users if they were unable to make their voices heard internationally in a concerted way. They also had to understand the needs of their user community so that they could act as a 'value-added' virtual gateway to complex services for their users. If they cannot provide a 'value-added service' some aspects of their traditional role of local service might be usurped by powerful commercial services. Evaluation is obviously a key to understanding user requirements and there was much to commend in the many evaluations reported. However, evaluation research is a complex field; developing a sophisticated level of competence amongst librarians in the undertaking of evaluation studies may be quite a challenge.
The delegates obviously found Digilib a useful forum for sharing these important issues with colleagues and there were hopes of holding another such conference in two years time if a volunteer came forward to act as organiser and host.
References
- FinELib, the National Electronics Library of Finland http://www.lib.helsinki.fi/finelib/digilib/
- Information Research: an international electronic journal, Vol.9, No.2 (2)
http://informationr.net/ir/9-2/infres92.html - Eason, Ken, MacIntyre, Ross, Apps, Ann and Ashby, Martin. (2003). 'Early Integrators and the Passive Majority: An evaluation study of a large web-based bibliographic reference database', Proceedings of the 2003 Conference on Users in the Electronic Information Environments. UEIE2003 - Towards a User-Centred Approach to Digital Libraries) http://epub.mimas.ac.uk/papers/ueie2003.html
- zetoc - British Library Electronic Table of Contents at MIMAS http://zetoc.mimas.ac.uk
- Eason K.D., Richardson, S, Yu, L., 'Patterns of use of Electronic Journals', Journal of Documentation, Vol 56(5) pp 477-504.