A flipped classroom approach to teaching search techniques for systematic reviews to encourage active learning

This project report describes the rationale for moving a one-shot library teaching session on advanced searching for systematic reviews to a flipped classroom approach (e-learning ahead of face-to-face teaching) and the process this took. It examines the e-learning and active learning elements designed to support learners engage with challenging threshold concepts including subject headings. Learner feedback during, immediately at the end of each session, and in response to a follow-up impact survey is considered. Overall, learner feedback on the flipped classroom was very positive and teachers reported improved learner outcomes (formative in-class informal assessment). Areas identified for development are presented. The report extends the body of research on the use of the flipped classroom in information literacy and provides evidence that active learning techniques can be successful in increasing learner engagement and achievement even in a one-shot setting.


Introduction
This project report describes the rationale for moving the teaching on advanced searching for systematic reviews to a flipped classroom approach (e-learning ahead of a 3-hour face-to-face teaching session) and the process this took. It examines the e-learning, the active learning elements and assessment which was included in the new lesson plan, alongside learner feedback, both immediate at the end of each session, and in response to an impact survey.

Background
Librarians supporting the health faculties at King's College London have been involved in developing and teaching regular one-shot face-to-face group sessions on advanced searching for systematic reviews since 2011. Demand has been continually high with staff, PhD students, MSc and even BSc students requesting support and keen to attend teaching sessions. The face-to-face teaching session has moved through several different iterations (Blackstock & Lipczynska, 2015); demand has continuously outstripped supply.
Although satisfaction from learners who completed the end-of-session questionnaire was generally high, the experience was less positive for librarians teaching the sessions. The main issue identified in teacher reflection was the difficulty of trying to teach advanced searching skills suitable for completing a high quality systematic review to learners who had little to no experience of undertaking a search on a structured database such as Embase or APA PsycInfo, nor of using database subject headings (thesaurus terms). The learning journey that attendees were asked to take in just one three-hour session was extreme. This often led to a lack of time to cover all planned elements of the lesson, particularly the section on searching grey literature. During the individual practice database search activities where librarians provided feedback it was apparent that learners were struggling to apply what they had just been taught and needed more time to try to understand this.
In 2019 it was decided to teach this as a flipped classroom (e-learning ahead of a face-to-face session). E-learning, which was developed in-house, supporting systematic reviewers seeking to develop searching skills had been in place since 2018. However, when checked in 2020 only 12% of the attendees on the previous versions of the workshop had ever engaged actively with the e-learning at any time.
In July 2019 e-learning was made compulsory ahead of attending the sign-up search techniques for systematic reviews course. The lesson plan for the face-to-face teaching session was completely overhauled and new activities introduced. It aimed to give more time to learners to work on challenging elements such as subject headings, which can be considered threshold concepts (Meyer et al., 2010), and to be supported in class if these were proving troublesome. Active learning methods within the face-to-face teaching were used.

Flipped Classroom
The flipped classroom approach which has been used successfully in one-shot information literacy teaching (Brooks, 2018) requires learning to be done ahead of a session, with class time used for discussion and checking learning. It repurposes the classroom, frontloading the content so that more passive learning, for example an introduction to a concept, happens ahead of class, and can allow the time with the teacher to be more active and interactive. Benefits can include a positive impact on learning motivation and/or engagement along with academic performance (Zainuddin et al., 2019). Importantly, Weightman et al. (2017, p.21) conclude in their systematic review that there is "compelling evidence that information literacy training is effective and well received across a range of delivery formats". This finding has been confirmed in an update search and appraisal of the literature (Morris, 2020, p.19) which concludes that "experiences are comparable and student preference is generally neutral in relation to delivery format."

Active Learning
Active learning covers a wide range of learning and teaching methods and activities which aim to place the learner in a position to think about and apply what they are learning in a real-world meaningful context. This should encourage motivation to learn and learner engagement. Brame (2016) discusses the link between undertaking actions and successful learning, emphasising the importance of: "…activities that students do to construct knowledge and understanding. The activities vary but require students to do higher order thinking. Although not always explicitly noted, metacognition-students' thinking about their own learning-is an important element, providing the link between activity and learning." Research specifically within the information literacy field has demonstrated results in-line with the wider research on active learning. Holderied (2011) discusses the positive impact that active learning activities (e.g. interactive technologies) can have on student engagement and learning of information literacy concepts.

Flipped classroom: e-learning
E-learning supporting information literacy was developed by the teaching librarians at King's College London to address issues around equity of teaching, to support academic engagement, and to scaffold learning through a learner's time at the university.
In the new flipped version learners were asked to complete the library's standard Advanced Health/Systematic Review e-learning pathway (see Figure 1) hosted on the library's Moodlebased Virtual Learning Environment (VLE) module.
Self-diagnostic quiz (1 attempt allowed) E-learning objects: 1. Starting your research and exploring frameworks 2. Exploring databases 3. Ways of searching 4. Searching in action 5. Combining your searches with OR and AND 6. Using limits in your searching 7. Accessing an article's full text 8. (Systematic Reviewers) Using methodological filters 9. The e-learning objects comprised a mix of textual presentation, some interactive content, and recorded demonstration of databases etc (see Figure 2). If a learner failed a quiz question, they received auto-feedback after the quiz was submitted to guide them to the appropriate learning object. It was estimated that if a learner were required to complete all the e-learning objects it would take approximately one hour. The learners were then required to pass a final Moodle quiz (if they hadn't passed the self-diagnostic quiz) to check knowledge.
A new section in the e-learning module was created to describe the process for booking onto the group workshop. Learners were required to pass a quiz prior to the booking link for the workshop becoming visible (pass score set at 10 out of 14). Self-selection of completion (where learners tick a box to show they have completed an activity) was not used as there was concern that some learners might select this without having completed the e-learning.

Face-to-face teaching
When setting the learning objectives (see Figure 3) both the e-learning and the face-to-face session were considered as a whole.
Whilst time was allocated within the face-to-face session to briefly recap the e-learning and to identify what the learners still needed to know, the aim was to use the majority of the time to scaffold the learning to move learners progressively towards a deeper understanding and to give them the chance to put their new knowledge into practice in a supported environment. Learning objectives and the lesson plan were deliberately focussed on the challenging threshold concepts, which had been highlighted by librarians as the points at which students experienced difficulty and anxiety. When grasped they lead to a transformed and deeper way of understanding the subject and without this happening the learner cannot progress (Meyer et al., 2010).
The elements highlighted included: understanding the structure of a database (specifically subject headings); the ability to translate knowledge of advanced searching to assess the quality of a search strategy; and the importance of using grey literature. Whilst, as Hosier (2017, p.3) discusses, "threshold concepts theory frames learning as a highly individual journey" which can clash with the expectation in learning objectives that students will be able to achieve the same learning goals in the same time frame, it was felt important that these were kept at the forefront when planning the session so that all learners could progress to some degree in each of these aspects. It was very likely that even though all attendees had passed the quiz that there would still be a lot of variation in ability.
By the end of the intervention learners will be able to: 1. Create a focussed search question and identify key concepts 2. Construct a search strategy using advanced searching techniques demonstrating an understanding of the value of keywords and subject heading searches 3. Appraise and evaluate their own work and the strategies of others 4. Identify types of grey literature used in published systematic reviews, and communicate this to their peers. 5. Demonstrate an understanding of the importance of matching an information need with an appropriate product by selecting an appropriate grey literature source for their systematic review topic, and undertaking a search 6. Engage with a community of practice; post on the discussion forum and propose answers to questions from their peers 7. Know where to go for further information and support When designing active learning activities it was essential to keep in mind the aim for them to be meaningful and for learners to clearly see the link between them and how they would use the knowledge and apply this to a real-world situation. The Active Learning at King's site (King's College London, 2020a) proved a useful tool for considering the principles of active learning and possible activities alongside evaluation approaches.
Activities were designed to allow informal in-class formative assessment, giving learners the opportunity to demonstrate understanding. For example, by articulating why they chose to use a particular grey literature source for a specific purpose.

Delivery
The newly designed flipped classroom approach launched in July 2019 with the face-to-face three-hour one-shot session element running 17 times, approximately every two weeks until March 2020 (when all face-to-face teaching was paused due to coronavirus). 123 learners in total attended the new version of the session. The session was open to MSc and PhD students as well as staff, and a few undergraduate students also attended. Each session was co-delivered by two teaching librarians.

Activities in the classroom
Aspects of active learning included collaborative and cooperative learning, pair/share and peer feedback activities; problem-based learning with learners creating and evaluating their own search strategy; and authentic formative assessment, with teachers able to address common elements of troublesome knowledge.
Demonstrations of database searching were kept at a minimum as this was covered in the elearning. Instead pair/share activities were used, including an activity where learners were provided with four handouts of article database records and asked to decide in their pairs whether each article would have been retrieved by a specific search line. This encouraged further understanding of database structure and why both subject heading and keyword searching was necessary.
The session was designed to provide learners with plenty of opportunities to work on their own research question and search strategy. As well as time working on a database of their choosing there were activities around searching on PROSPERO, locating a published search strategy that could be useful in developing one of their own concepts, considering what published filters might be appropriate for them to use, and finally undertaking a search on a relevant grey literature resource.
Time was allocated in the session for learners to explore not only what grey literature is but also to consider its strengths and weaknesses and discuss any concerns. They were then given the opportunity to search on their own topic. The AACODS checklist (Tyndall, 2010) was briefly introduced as a tool to enable evaluation and critical appraisal of grey literature.

Two activities utilised the Peer Review of Electronic Search Strategies (PRESS) 2015
Evidence-Based Checklist (McGowan et al., 2016). In the first, a pair/share activity, learners assessed a search strategy (Ovid Medline) containing specific errors. The second activity focussed on learners' own search strategy and encouraged learners to give feedback to each other in their pairs using the PRESS checklist.
Peer feedback was also encouraged in the activity looking at planning a search with learners making suggestions on their partner's Concepts and Frameworks Planning Template.
The teachers circulated during the activities and provided individual feedback as required to learners, particularly around the individual planning and searching activities.
One of the aims of the session was to encourage engagement with a community of practice. As well as encouraging this via the use of small group work in the session, learners were also asked as a final activity to engage with a specific Moodle discussion forum, a space that staff and students at King's College London can post a search strategy for comments.
Throughout the session the Searching for Systematic Reviews LibGuide (King's College London, 2020b) was used as a launchpad for resources and when answering questions in order to encourage long-term engagement with this (see Figure 4).

Feedback
We have collected a variety of feedback and conducted research into the impact from our flipped classroom workshop.

In-session feedback
An in-session connect activity using a Padlet provided feedback on what learners found most useful/memorable from the e-learning but also on what they still needed to know (see Figure 5). Padlet is an online virtual "bulletin" board (Padlet, 2020), where students and teachers can collaborate, reflect, share links, add comments and 'like' another post. Posts can be anonymous if learners wish.
Common and frequently 'liked' themes in the responses to the question 'What did you learn?' from the e-learning included: combining (using AND/OR); database subject headings and keyword searching; search frameworks e.g. PICO and PEO and templates; and search techniques such as truncation and phrase searching. Less common were responses around grey literature and the use of search filters and limits.
Common and frequently 'liked' themes in the responses to the question 'What do you need to know more about?' included the need for more support around: the use of database subject headings and focus and explode options; use of advanced techniques e.g. proximity/adjacency searching; development of their own search strategy (putting the e-learning into practice); grey literature; limits and filters. Often responses focussed on how a learner could apply these aspects to their search e.g. a learner showed they had an awareness of frameworks (PICO or PEO) but wasn't quite sure what best fit their own research question.
Overall, responses to the Padlet indicated that the e-learning element of the flipped classroom was working as hoped, introducing the concepts and delivering the passive learning element. Many learners indicated there was still a need to understand certain aspects further, but the lesson plan was designed to develop this initial learning and put the new knowledge into practice in a supportive environment. The Padlet was completed at the start of the workshop and revisited at the end to ensure that all posts had been covered.
The Padlet was used in each session to act as a check that each specific group of learners' needs were being met but has also been utilised to inform further development of the session in general as well as identifying further developments for our e-learning. The Padlet was cleared after each session but the responses were recorded. Developments as a result of reviewing Padlet posts have included extending our teaching to cover an element on searching for scoping reviews and developing further e-learning on adapting a search strategy to different databases.

Critical Incident Questionnaire
At the end of the session learners used a Critical Incident Questionnaire (CIQ) via Microsoft Forms to reflect on what they had learnt and when they were most and least engaged in the session. There were 87 responses from the 123 attendees, a response rate of 71%. The learner reflection both at this point and with the earlier Padlet encourage meta-cognition.
The CIQ used was one already in use for all other information literacy sessions. It has been expanded from the standard 5 questions (Brookfield, 1995) to also include a question to gauge how learners might change their practice as a result of what they have learnt, for example what they perceive the immediate impact might be.
All respondents said they would recommend this session to a friend (our main quantitative teaching performance indicator). Common feedback themes included an increase in confidence, a better understanding of subject headings and keyword searches and other advanced searching techniques, and of grey literature. There was also appreciation of the time available in the session to work on their search strategy and receive feedback. Learners frequently mentioned that they welcomed the opportunity to put into practice in the face-to-face session what they had learnt during the e-learning.
Learners regularly highlighted the times in the session when they were trying to apply the learning to their own search strategy and develop their search terms as the point at which they were most engaged. Many also referenced the interactive practical activities in general and the group work as points when they were highly engaged (although there were a few responses which felt there were too many activities).
In terms of being least engaged several responses mentioned the grey literature element, often because as MSc students they had already been guided by their supervisor to not include grey literature. Others felt that the recap of the e-learning was too repetitive and as a direct result of these comments the lesson plan timing was altered to ensure that this recap element was minimal.
Respondents valued the personalised feedback from teachers during the individual work on their search strategy and that there were lots of opportunities to ask questions. For a few respondents, the pace of the session was too quick whilst others didn't feel enough new content was included. This is unsurprising given the variation in levels of study and experience but is something to be conscious of when trying to appropriately differentiate learning. With respect of the threshold concepts identified it was pleasing that there were very few comments which identified either subject headings/keyword searching, appraising a search strategy, or grey literature as an area where they were puzzled or confused during the session.
A common feedback theme was around increased confidence to undertake and develop a highquality search strategy. The majority were actively working on a systematic review, often with relatively short deadlines, and many responses stated an expectation of using their learning for that purpose. In addition, many respondents noted that they would also use what they learnt to support their practice or for further assignments, or to support students they might be supervising.
These reflections have fed into further developments of the e-learning and session including more coverage of adapting a search strategy to different databases.

Impact questionnaire
An impact questionnaire was sent via Microsoft Forms in February and March 2020 to all 123 attendees on the flipped classroom version of the session since July 2019. Some attendees were therefore reflecting on the experience six months after attending the session and others only weeks or days after. 30 responses to this questionnaire were received, a 24% response rate.
Responses were from attendees across 12 of the workshops, with dates attended ranging from July 2019 to February 2020. There was a range across the health faculties and a variety in the type of respondents (with the highest number being PhD students).
Many respondents provided clear statements about the workshop as a whole meeting their learning or research needs. Comments were received about the workshop being helpful, informative, and invaluable. 28 out of 30 would recommend this session to a friend (one negative, one recommending for non-expert searchers).
The one negative response was from a session where all the CIQ responses had been positive. Those willing to complete the reflection at the end of the session may well be those who were most engaged and therefore had a more positive experience. Ensuring a follow up impact questionnaire as standard when learners have had a chance to reflect may help balance this to some degree and potentially identify any problem areas.

Short-and long-term impact
Over 56% of respondents reported that the e-learning and workshop contributed an immediate impact ('Definite immediate contribution') on three of the aspects identified: 'Improved quality of research'; 'Contributed to personal or professional development'; and 'More informed decision making' (see Figure 6).
'Improved quality of research' also had the most responses for 'Definite long-term contribution' at 33%. Overall, the number of responses selecting 'No contribution' or 'Unsure' was low.
The aspect of 'Publication of research' was the one where there was less certainty from respondents about definite impact with 47% selecting 'Probable future contribution'. This is unsurprising as the systematic review process can take months to complete and be published and for some MSc attendees publication was never the goal. This aspect also saw the most 'No contribution' responses (10%).

Impact on practice
Respondents reported using the learning, and returning to the materials, when working on their systematic review. There was mention of using the VLE forum to receive feedback on a search strategy after the session. Respondents reported having been able to develop a higher quality search strategy, with one already having presented the findings from their systematic review at a conference.

Reflections on flipped classroom and active learning
The flipped classroom was referenced with one respondent mentioning the comprehensive nature of the combined e-learning and workshop, and another that it was good to have the planning template to work on ahead of the session. Some responses suggested more content could be flipped allowing more time for learners to be supported in undertaking their own searches and to ensure that all attendees were at a more standardised advanced level. Several responses valued the practical time in the session to undertake their own search with teacher support, and to receive tailored advice. Many suggested that there be more time allocated to this.
In line with the CIQ responses the variation in respondents' experience of research/systematic reviews was clear. This had an impact on how much they felt the workshop was offering new learning versus consolidating the flipped e-learning content/what they already knew.
Some suggestions for improvement from learners validated further developments already made to the workshop in late 2019 in response to feedback in the CIQs and teacher reflection: flipping the completion of the planning template and showing how to repeat searches on different databases.

Impact interviews
To date three interviews have been carried out using a semi-structured approach and starting with a cognitive mapping exercise. Further interviews are anticipated, and analysis will then be performed.

Some initial aspects of immediate interest have been identified.
One interviewee spoke about the barrier caused by the compulsory e-learning. This individual attempted and failed the quiz multiple times before eventually passing.
The need to ensure that learners are given an opportunity to put the e-learning into practice, check their knowledge and be guided to further support as necessary has also been surfaced. The aim in flipping the learning was to use the e-learning to deliver the introductory content but if some students are struggling with these concepts then this is not meeting the objective fully.
This ability to apply their knowledge and consolidate their learning is supported in the workshop but the lesson plan deliberately avoids covering too much of the same ground and of course not all learners will be able to attend a workshop, so there is a need to develop this aspect for those undertaking solely the e-learning.

Threshold concepts
Overall, the flipped classroom approach has worked well in several aspects. With the three threshold concepts identified there has been a mix of success.
Librarians teaching the session reported that generally they were seeing a more consistent use of subject headings by learners and a better understanding of the reasons that both keyword and subject headings needed to be used for each concept. This is also reflected in the CIQ responses from learners themselves.
Learners were actively engaged during the activities using the PRESS checklist and this activity worked well providing an opportunity to translate their knowledge of advanced searching to assess the quality of a search strategy (their own and published ones). Feedback and guidance could be offered to the small groups as the teaching librarians circulated and the responses the learners put forward indicated that they were able to use the checklist appropriately.
The final threshold concept concerned the importance of using grey literature. Whilst some CIQ responses mentioned grey literature as something they learnt in the session, there were many more responses that identified grey literature as something that was not relevant for them. This was often because they were MSc students undertaking a dissertation project which whilst called a systematic review was very time limited and therefore supervisors were recommending that grey literature not be included.

Active Learning
It should be noted that within the confines of one three-hour session there are obvious limitations to how far many of the active learning activities can be developed but the aim was to base activities on the principles and make some impact on student motivation and engagement and this appears to have been successful. Frequent mention of the positive aspects of the activities in the CIQs and impact survey support this.
As observed by Wiggins and McTighe (2005, p.8) "Individual lessons are simply too short to allow for in-depth development of big ideas, exploration of essential questions, and authentic applications". It would be interesting to explore further how in-depth aspects of active learning, for example ongoing cooperative learning and problem-based learning, could be incorporated into a more embedded programme working over time with a specific cohort.

Timing
Timing was still an issue within the flipped session. Frequently, and as expected, the learners wanted to ask a lot of questions during the activity where they worked on their own search strategy but care needed to be taken not to let this overrun at the expense of other content. It was decided not long after the new session format started that an additional activity should be flipped to free up more time in class and so learners were asked to complete the concepts sheet (using PICO, or other suitable framework) as introduced in the e-learning for their own topic ahead of coming to the session.

Differentiation
It was also a challenge where some learners had already developed a provisional search strategy and they were paired with someone who hadn't yet started searching at all ahead of the session. The peer feedback activity looking at each others' search strategy was often imbalanced and this was highlighted in the impact questionnaire responses as an area which could be improved. On some occasions the activity was deliberately dropped in favour of more individual search development if the teaching librarians felt that most appropriate for that specific group.

Community of Practice
Further work needs to be undertaken to encourage a community of practice around systematic reviews. Ideally, staff and students would feel happy in responding to questions on the forum. They would feel engaged to support their fellow reviewers even though they may have moved past the searching stage. It may also be possible to provide a space where researchers can support each other in the aspects of systematic reviewing beyond the library's scope, for example assessing bias, use of software, and the meta-analysis process.

Access barriers
One impact study interview indicating that some learners struggled to pass the e-learning quiz has raised some immediate concerns about how many learners were not able to access the teaching session at all. In response work to improve automatic interventions (after multiple quiz failures) and redirect to individual support (e.g. forum, 1-2-1, live chat) is underway. It will also be important to consider alternative routes to the webinar, perhaps attendance at an introductory session.
The literature is clear that a major challenge to the flipped classroom approach is learner motivation to complete learning ahead of a session (Zainuddin et al., 2019). This may affect the session more if it moves to a timetabled slot for some cohorts as opposed to learners choosing to attend and therefore being more motivated to engage with the process.

Next Steps
A five-week asynchronous online course hosted on the Moodle VLE which uses many of the same active learning techniques and peer support has been developed. This was piloted with library staff and ran once in earnest in April/May 2020. Feedback from this will be analysed, the course developed further, and will likely run again in January 2021.
Face-to-face on-campus teaching is unlikely to resume in the first semester of 2020/21 and the demand for support with searching for systematic reviews will be heightened with faculties offering student systematic review style projects to replace laboratory and clinical projects.
Finding a scalable solution to support students across the faculties is under discussion and it is likely that embedding the existing e-learning into module VLE spaces and timetabling this eleaning will be a first step. Teaching librarians are ensuring that conversations with academics are covering the aspect of what the scope of the student project is, and whether it is truly a systematic review or instead a critical review or systematic literature review in order to ensure that the appropriate e-learning is embedded.
Work is underway to translate the sign-up three-hour face-to-face teaching to a shorter webinar version. Increasing the flipped content is being considered whilst aiming to retain some of the peer-feedback and collaborative working elements.
Further potential developments include a course to support social science systematic reviewers and to translate the course materials to support NHS colleagues.
Finally, work is continuing around the impact survey and interviews and further detailed analysis and reflection will be generated by this. The aim will be to continue to gather impact responses as teaching recommences.