Muhammad Mustehsan Bashir  ( Department of Plastic Surgery, King Edward Medical University / University of Lahore, Lahore, Pakistan. )
Rehan Ahmad Khan  ( Department of Surgery, Islamic International Medical College, Riphah International University, Islamabad, Pakistan. )

Objective: To compare the effect of individual versus team competition gamification on trainees' engagement in journal club.

Methods: The quasi-experimental study was conducted at two tertiary care hospitals in Lahore, Pakistan, from December, 2019 to May, 2020, and comprised all plastic surgery residents in training years 3-5 who had attended non-gamified journal club sessions. They were divided into two groups. Initially, two journal club sessions were run in a gamified manner. Group 1had an individual competition whereas group 2 had a team competition. In the second phase, the group orientation was reversed. Engagement scores were measured using the student engagement survey tool at baseline, at the completion of the first part, and at end of the intervention. Data was analysed using SPSS 22.

Results: Of the 44 participants, 23(52.3%) were in group 1 and 21(47.7%) in group 2. Overall, there were 27(61%) females and 17(39%) males. The mean age of the sample was 31±2 years. Gamification resulted in significant increase in the level of residents' engagement in journal club activities compared to the baseline scores(p<0.05). There was no significant inter-group difference in the post-intervention engagement scores (p>0.05).

Conclusion: Gamification designs used in the journal club activity increased student engagement level compared to the pre-intervention score, but gamification design was not a significant factor in this regard.

 

Keywords: Journal club, Gamification, Game designs, Student engagement. (JPMA 72: 643; 2022)

 

DOI: https://doi.org/10.47391/JPMA.1261

 

Introduction

 

One of the commonly used teaching-learning activities in medical education is the oral presentation of a published research article, commonly referred to as a "journal club" which serves to keep the participants abreast of the latest literature and provides an arena for the critical re-evaluation of literature.¹ The substantive value of the club is evident by the fact that 70-95% post-graduate residency programmes have integrated journal clubs in their curricula. Despite the incorporation of best practices for journal club in residency education, the mean attendance is typically 60% for many programmes.²

Efforts are underway to introduce active learning strategies in health professional education curricula. Student engagement with academic work is defined as "constructive, enthusiastic, willing, emotionally-positive, and cognitively-focussed participation in learning activities".³ Unfortunately, conventional models of journal club disengage residents. Residents' disengagement compromises the inculcation of critical thinking skills.⁴

The challenge to broaden the range of instructional strategies over and above the conventional pedagogies is colossal.  Gamification as a student-centred pedagogical strategy has been introduced to confront the challenge. Gamification is defined as "application of principles, philosophy, emotions and appearances originating from games in non-game contexts in order to engage and motivate people to help them learn and solve problems".⁵ Gamification involves selecting specific design elements (DEs) and applying them in a non-game situation. The important DEs include points, badges, levels, leaderboards, progress bars/performance graphs, game fiction/meaningful stories, human interaction (teammates and competitors) and avatars.⁶

Studies of the effects of gamification on student engagement in higher education are scant, and this is more so in medical education. The current body of literature on the effect of gamification on students' engagement supports the use of gamification to increase engagement.⁷ The majority of the studies done to date have not tried to examine the effect of different gamification designs on student engagement. Questions about the effectiveness of different gamification designs in enhancing student engagement remain unanswered. This necessitates further research to answer this question.⁸

The current study was planned to explore the effect of gamification on residents' engagement in journal club learning activity and to compare individual versus team competition design.

 

Subjects and Methods

 

The quasi-experimental, counter-balanced study was conducted at two tertiary care hospitals in Lahore, Pakistan, from December, 2019 to May, 2020. The counter-balancing design was used as it was not possible logistically and ethically to conduct a randomised controlled trial (RCT). This design ensured that all participants were subjected to both the treatment arms in a systematic manner to control threat to internal validity. This helped to overcome selection bias and assignment bias. Moreover, confounding effect of temporal variation of both treatments was also nulified.⁹

After approval from the Advance Study Research Board and Ethical Committee of The University of Lahore, the sample size was calculated using the World Health Organisation (WHO) calculator¹⁰ at 0.05 level of significance, 80% power of test, 0.52 standard deviation, 2.62 mean value of pre-intervention engagement score (M1), and 3.0 mean value of post-intervention engagement score (M2).¹¹

The sample was raised using purposive sampling from among plastic surgery residents of either gender in training years 3-5. Residents who had not attended non-gamified traditional journal clubs in the preceding three months were excluded. The residents at one hospital were labelled group 1 and those at other were labelled group 2. The articles for presentation in the journal club were selected by plastic surgeons who had at least 5 years of teaching experience. Residents were required to read the article based on the instructions provided to them about how to critically appraise and prepare the article for the journal club.¹² Each resident was asked to develop at least 10multiple-choice or true/false questions from the article to assess the validity, results, and applicability of the selected article to clinical practice. The questions generated by the residents were checked by the faculty members for correctness and relevance. Residents developing 10 acceptable questions were given as many points.  Residents developing >10 questions got extra points. A quiz based on these questions was developed on Kahoot to incorporate the elements of gamification.¹³ The score of the participants was shown by leaderboard in the individual competition, and by team score in the team competition. The individual competition and team competition gamification designs were labelled design 1 and design 2 (Figure).

 

 

Individual competition amongst the residents was conducted through Kahoot using classic mode (Player versus Player 1:1 devices). For the team competition, residents in each group were randomly allocated into two teams. Differences in the level of experience of residents were balanced across the teams. The team competition was also conducted through Kahoot using team mode (Team versus Team; shared devices). Before the intervention, the student engagement score was measured in the two groups using the student engagement survey (SES) tool.¹⁴ The survey measured cooperative learning, cognitive level and personal skills of the residents as components of the student engagement construct. Cooperative learning measured the enthusiasm and wilful interest shown by the residents to teach and learn from others during journal club activity. The cognitive level of engagement measured mindful commitment and amount of attention deliberated to the tasks assigned to the residents. Personal skills development measured the importance of journal club as the source of development of personal skills.

The original survey used a 4-point Likert scale, with 4 = very often, 3 = often, 2 = occasionally, and 1 =never, for 14 questions related to the constructs of student engagement.

The instrument was previously validated in various studies and across multiple course levels.^14,15 Alpha reliability of the original 14-item instrument is 0.84.¹⁴ However, the survey was re-validated by expert validation, cognitive interviews and pilot-testing. After incorporating the changes suggested by the experts, the final survey comprised 16 items.

In the first part of the study, two journal club sessions, 15 days apart, were run for two articles. Group 1 had gamification design 1 and group 2 had gamification design 2. At the completion of the first part of the study, the student engagement scores were measured in the two groups.

In the second part, two journal club sessions, 15 days apart, were run for the remaining two articles.  Group 1 had gamification design 2 and group 2 had gamification design 1. Residents were asked to fill the SES after the completion of the second part. The SES provided quantitative value, like resident engagement score, for each participant in the domains of cooperative learning, cognitive level, and personal skills development.

Data was analysed using SPSS 22. Categorical variable gender was presented as frequency and percentage. Continuous variables engagement score and age were presented as mean ± standard deviation. The normality of data was checked by the Kolmogorov-Smirnov test. The pre- and post-intervention scores in the two groups were not normally distributed. The mean pre-intervention trainee engagement scores in each domain for the two groups were compared with mean post-intervention scores in each domain for the two groups by Wilcoxon Signed Ranks test. The mean post-intervention score for the two groups for intervention 1 was compared with the mean post-intervention score for the two groups for intervention 2 by Mann-Whitney U test. P<0.05 was considered significant.

 

Results

 

Of the 44 participants, 23(52.3%) were in group 1 and 21(47.7%) in group 2. Overall, there were 27(61%) females and 17(39%)males. The mean age of the sample was 31±2 years.

Gamification resulted in significant increase in the level of residents' engagement in journal club activities compared to the baseline scores(p<0.05) both in individual (Table-1) and team formats (Table-2).

 

 

 

 

There was no significant inter-group difference in the post-intervention engagement scores (p>0.05) (Table-3).

 

 

 

Discussion

 

The trend of incorporating gamification in business and educational contexts is rising. This rising trend is attributable to the studies showing considerable effect on increasing learners' engagement and knowledge attainment through gamification.^16,17 However, the successful outcome related to the introduction of gamification in improving learning is still debatable and findings of a few studies did not show evidence that gamified learning enhanced the students' engagement.^18,19 However, the current study found an increase in student engagement scores with the introduction of gamification.

Instructional content has a direct and pronounced effect on learning behaviour and learning outcome.²⁰ Critical to the success of any gamification effort in graduate medical education is the introduction of instructional content that is centred on a real-life problem, relevant to the learners' needs and likely to benefit them practically. If the instructional content does not engage students, gamification of that content cannot by itself engage the students. Moreover, for gamification to be successful, the behaviour or attitude that is targeted by gamification must itself influence learning. For example, gamification protocols that provide rewards for making high-quality notes are likely to improve engagement. Thus, variation in results found in the studies on the effects of gamification on students' engagement can be attributed to inconsistencies in instructional strategies.

The varied effectiveness of gamification in enhancing student engagement could be explained per se by the variable combinations of design elements found in various gamification protocols.²¹ A study examined the role of game elements as means for increasing engagement in a learning activity, and used Fogg's behaviour model to explain how game elements can be tied together to foster and enhance engagement.²² This model has high applicability in case of human interaction with a gamified environment.²³ The model states that three essential elements — motivation, ability and triggers — when acting in unison can initiate an intended behaviour. For a learning activity to engage students, an intended behaviour must reach an activation threshold. When the behavioural activation threshold is reached, students are ready to engage in reading, learning and solving problems. This readiness to engage in a learning activity further transforms into "concentration": a state when the student interacts most efficiently with the assigned task.²⁴

Introduction of reward for developing high-quality questions after reading the article (motivation), individual/team competition based on the quiz comprising questions contributed by residents (ability), real-time leaderboard update during the competition (trigger) were the salient game design elements that helped to enhance the residents' engagement in the current study.

The effects of the leaderboard on students' engagement can be explained by Alderfer's theory which is an extension of Maslow's hierarchy of needs.²⁵ Alderfer's theory puts Maslow's five needs into three main categories, which are existence needs, relatedness needs and growth needs. Leaderboards helped to engage students by fulfilling relatedness needs as players related their personal scores with others. Existence and growth needs were also fulfilled by the leaderboard as players, in the first instance tried to stay in the ranking and then tried to grow and move up the ranking.

The current study did not find any difference in the engagement score between the two gamification designs.  Engagement in the context of gamification can be divided into two types: first, the engagement between players achieved through the utilisation of game design elements; second, the engagement in the form of interaction, cooperation and altruism between the players.⁷ The reason for the similarity in engagement score in the two gamification designs can be attributed to the similarity in instructions given to the residents to complete the assignment. Task design for individual competition and team competition was similar and provided autonomy to the residents to complete the tasks at their own pace. Every member of the team worked independently to read the article and develop questions. During the competition, team members used their collective wisdom to answer the questions. Thus, the social interaction, cooperation and altruism between members of each team was limited and occurred only during the quiz. This limited interaction between the residents could explain the reason for the similarity of engagement score between two intervention groups. In order to find the true effect of individual competition versus team competition, instructional strategy for completion of an assignment should also be different in the two intervention groups. Thus, the importance of appropriate instructional strategy should be taken into consideration in designing future gamification trials.

Student engagement is a complex construct and can be measured in a number of ways including self-reported surveys, teacher rating of students, direct observations, amount of time participants spent on the online programme, 'volume' related metrics, such as the total number of questions contributed or completed on an online quiz, and the total number of views per participant), interviews and focussed case studies.²⁶ The current study used SES, which is a subset of the National Student Engagement Survey (NSES). The survey provided a means of measuring non-observable, perceptual or subjective indicators of engagement. One of the limitations of self-reported surveys is honesty and/or accuracy of responses. Interviews and focussed case studies can elicit a more detailed, individualised, contextualised understanding of student engagement. Thus, one of the limitations of the current study is the lack of qualitative data about student engagement.

The use of customized app providing the feature of online reading of articles, real-time data about login details of residents, time spent on the interface, and uploading of questions directly in the app could have been more engaging for the residents. Thus, another limitation of the study was not using the customized gamification apps. However, the premium version of Kahoot used in the current study provided the feature of real-time updates of the leaderboard and an opportunity to run an individual competition and team competition.

Results of the current study were based on four sessions of journal club activity conducted through gamification on a small sample size of plastic surgery residents. Thus, the results cannot be generalised. It is possible that a larger sample size of participants and/or a longer period of data collection could affect results. However, instructional strategy based on principles of adult learning and counterbalanced design adds to the internal validity of the study. Moreover, the current study introduced a practical instructional strategy combined with gamified learning that can be used in other residency programmes to engage residents in journal club activity.

 

Conclusion

 

Individual and team competition gamification designs increased student engagement scores compared to the pre-intervention scores. However, there was no difference in post-intervention engagement score in the two gamification designs.

Disclaimer: The text is based on a thesis done for Masters of Medical Education.

Conflict of Interest: None.

Source of Funding: None.

 

References

 

1.       Veresoglou SD, Rillig MC. Research experience modifies how participants profit from journal clubs in academia. J Biol Educ 2019;53:327-32. Doi: 10.1080/00219266.2018.1469541.

2.       Williams AD, Mann BD, Lipman J. Understanding the Modern Surgical Journal Club. J Surg Educ 2019;76:637-43. doi: 10.1016/j.jsurg.2018.09.012.

3.       Skinner EA. Engagement and disaffection as central to processes of motivational resilience and development. In: Wentzel KR, Miele DB, eds. Handbook of motivation at school. 2nd ed. New York, NY: Roultedge, 2016; pp 145-68.

4.       Rosenthal J, Rosenthal KS. Interactive Journal Club: Teaching an Old Dog New Tricks. J Med Educ Curric Dev 2017;4:e2382120517719710. doi: 10.1177/2382120517719710.

5.       Sardi L, Idri A, Fernández-Alemán JL. A systematic review of gamification in e-Health. J Biomed Inform 2017;71:31-48. doi: 10.1016/j.jbi.2017.05.011.

6.       Bai S, Hew KF, Huang B. Does gamification improve student learning outcome? Evidence from a meta-analysis and synthesis of qualitative data in educational contexts. Educ Res Rev 2020;30:100322. Doi: 10.1016/j.edurev.2020.100322

7.       Bouchrika I, Harrati N, Wanick V, Wills G. Exploring the impact of gamification on student engagement and involvement with e-learning systems, Interactive Learning Environments. Interact Learn Environ 2019;7:1-13. Doi: 10.1080/10494820.2019.1623267

8.       Looyestyn J, Kernot J, Boshoff K, Ryan J, Edney S, Maher C. Does gamification increase engagement with online programs? A systematic review. PLoS One 2017;12:e0173403. doi: 10.1371/journal.pone.0173403.

9.       Polgar S, Thomas SA. Introduction to Research in the Health Sciences, 7th ed. Sydney, Australia: Elsevier Ltd; 2019.

10.     Lemeshow S, Hosmer DW Jr, Klar J, Lwanga SK. Adequacy of Sample Size in Health Studies, 1st ed. Massachusetts, USA: John Wiley & Sons; 1990

11.     Andriamiarisoa R. Impact of gamification on student engagement in graduate medical studies. [Online] 2018 [Cited 2022 February 09]. Available from URL: https://scholarworks.waldenu.edu/cgi/viewcontent.cgi?article=6581&context=dissertations

12.     Al Achkar M. Redesigning journal club in residency. Adv Med Educ Pract 2016;7:317-20. doi: 10.2147/AMEP.S107807.

13.     3 Rs 4 Teachers. UPDATED: 2 tools are better than one: comparing Socrative and Kahoot. [Online] 2015 [Cited 2020 April 28]. Avaialble from URL: https://3rs4teachers.wordpress.com/2014/04/19/2_tools_are_better_than_one/

14.     Maskell EC, Collins L. Measuring student engagement in UK higher education: do surveys deliver? J Appl Res High Educ 2017;9:226-41. Doi: 10.1108/JARHE-11-2015-0082

15.     Hopper MK, Kaiser AN. Engagement and higher order skill proficiency of students completing a medical physiology course in three diverse learning environments. Adv Physiol Educ 2018;42:429-38. doi: 10.1152/advan.00128.2017.

16.     Dichev C, Dicheva D. Gamifying education: what is known, what is believed and what remains uncertain: a critical review. Int J Educ Technol High Educ 2017;14:1-36. Doi: 10.1186/s41239-017-0042-5

17.     Geelan B, de Salas K, Lewis I, King C, Edwards D, O'Mara A. Improving learning experiences through gamification: A case study. Aust Educ Comput 2015;30:1-21.

18.     Hew KF, Huang B, Chu KWS, Chiu DKW. Engaging Asian students through game mechanics: Findings from two experiment studies. Comput Educ 2016;92:221-36. Doi: 10.1016/j.compedu.2015.10.010

19.     Leaning M. A study of the use of games and gamification to enhance student engagement, experience and achievement on a theory-based course of an undergraduate media degree. J Media Pract 2015;16:155-70. Doi: 10.1080/14682753.2015.1041807.

20.     Dignath C, Büttner G. Teachers’ direct and indirect promotion of self-regulated learning in primary and secondary school mathematics classes–insights from video-based classroom observations and teacher interviews. Metacogn Learn 2018;13:127-57. Doi: 10.1007/s11409-018-9181-x.

21.     Maheu-Cadotte MA, Cossette S, Dubé V, Fontaine G, Mailhot T, Lavoie P, et al. Effectiveness of serious games and impact of design elements on engagement and educational outcomes in healthcare professionals and students: a systematic review and meta-analysis protocol. BMJ Open 2018;8:e019871. doi: 10.1136/bmjopen-2017-019871

22.     Zimmerling E, Höllig CE, Sandner PG, Welpe IM. Exploring the influence of common game elements on ideation output and motivation. J Bus Res 2019;94:302-12. doi: 10.1016/j.jbusres.2018.02.030

23.     Dicheva D, Dichev C, Agre G, Angelova G. Gamification in Education: A Systematic Mapping Study. J Educ Techno Soc 2015;18:75-88.

24.     Poondej C, Lerdpornkulrat T. The development of gamified learning activities to increase student engagement in learning. Aust Educ Comput 2016;31:1-16.

25.     Nicholson S. A RECIPE for Meaningful Gamification. In: Reiners T, Wood LC, eds. Gamification in Education and Business. Cham, Switzerland: Springer International Publishing, 2015; pp 1-20. DOI: 10.1007/978-3-319-10208-5

26.     Pugh KJ, Bergstrom CM, Heddy BC, Krob KE. Supporting Deep Engagement: The Teaching for Transformative Experiences in Science (TTES) Model. J Exp Educ 2017;85:629-57. Doi: 10.1080/00220973.2016.1277333