Muhammad Shahid Shamim  ( King Abdul Aziz University, Rabigh, Jeddah, Kingdom of Saudi Arabia. )
Bushra Shirazi  ( Department of Surgery, Ziauddin University, Karachi. )
Aamir Omair  ( King Saud bin Abdulaziz University for Health Sciences, Riyadh, Kingdom of Saudi Arabia. )

Objective: To perform an outcome-based evaluation of the diploma programme initiated in 2006 at the Centre of Biomedical Ethics and Culture..
Methods: The broad based evaluation was done at the Centre of Biomedical Ethics and Culture, Sindh Institute of Urology and Transplantation, Karachi, from July 2011 to June 2012 and comprised pass out batches from 2006 to 2010. Outcome logic model was applied through a questionnaire-based approach. Emails were sent to the graduates, containing a mix of closed and open-ended questions. Quantitative feedback was analysed for frequencies and percentages. Content analysis was conducted for open-ended responses. SPSS 19 was used for statistical analysis.
Results: Four batches had graduated in the five-year study period. A total of 50 students had been enrolled, while 41(82%) graduated. Of them, 31(76%) graduates responded. Overall, 10(24%) graduates completed their Masters in bioethics, and 16(39%) were involved in institutional ethics committee. All (100%) believed the diploma had increased their knowledge of ethics and they were involved in disseminating the acquired knowledge through presentations, seminars/conferences 29(93.5%), teaching and awareness activities 22(71%).  Besides, 28(90%) respondents believed their behaviour had changed and 27(87%)were improving the quality of work and environment, while 18(58%) had published related articles after their diploma.
Conclusion: The programme was found to have achieved its objective during the first five years of its inception.
Keywords: Ethics, Bioethics, Diploma, Evaluation. (JPMA 65: 397; 2015)


Introduction

During the past few decades, there has been a significant change, throughout the world, with regards to teaching of Ethics in medical curriculum. The process of change started with "The Pond Report" in 1987. This was subsequently augmented by the General Medical Council\'s (GMC) recommendations for "Tomorrow\'s Doctor". By the beginning of the new millennium, most of the medical colleges of Europe and North America had Ethics as part of their formal curriculum.^1,2
Similar tides of change trickled down to the developing countries like Pakistan, where the Pakistan Medical and Dental Council (PMDC) included Ethics in the syllabus of undergraduate medical education in 2001.³ The challenge that followed this change was "who will teach ethics?" Traditionally Ethics was considered a part of the hidden curriculum which the students may (or may not) learn from their clinical supervisors during clinical rotations and apprenticeship. However, evidence has mostly disproven this unobserved and unmonitored method of learning, without a structured approach due to multiple reasons. The much argued reason is the conflict that develops within the students\' minds when they observe practices in their clinical work that contradict with what they learn from text and untrained Ethics teachers in the classroom.^4,5 This scenario led to the initiation of bioethics diploma programme by the Centre of Biomedical Ethics and Culture to develop teachers who can teach Ethics to medical students and doctors.
The CBEC at the Sindh Institute of Urology and Transplantation (SIUT), Karachi, launched a year-long Postgraduate Diploma (PGD) in Biomedical Ethics in 2006 for healthcare related professionals in Pakistan. The PGD is free of cost for the students and financial support is provided by the SIUT.⁶ The aim of the PGD programme is to develop a community of healthcare related professionals with basic knowledge and skills in biomedical ethics. The programme also encourages graduates to provide help and support in initiating and sustaining ethics related activities in education, clinical practice, and research in their institutions.
The one-year diploma programme is specifically designed for healthcare related professionals, working in different teaching institutions. The programme consists of three 8 to 14 full-day teaching modules held at the Centre approximately every three months.⁶ Between the modules, the students are given tasks and assignments as part of distant learning part of the course. The tasks include different modalities like critical analysis of published work, reflective write-ups on contemporary ethical issues, dilemmas and day-to-day experiences. A fourth and last module includes written examination and presentation of an "Ethics project" which is a plan of activity that each student is supposed to conduct in their workplace during the year after diploma.
Since the inception of the programme in 2006, the current study was the first formally planned evaluation conducted by a non-faculty member. The evaluation was planned to be outcome-based (to assess whether the programme is achieving its intended outcomes),⁷ and formative (for the improvement of the programme). Thereby, the objective of this study was to perform an outcome-based evaluation of CBEC\'s bioethics diploma programme for the achievement of the programme\'s intent.


Subjects and Methods

The broad-based outcome-oriented evaluation with a mix of quantitative and qualitative analysis was conducted at the CBEC-SIUT, Karachi, from July 2011 to June 2012.
The study was approved by the institutional review committees of SIUT and the Dow University of Health Sciences (DUHS), Karachi, where it forms part of Master in Health Professionals Education (MHPE) project.
The model was derived from the outcomes-logic model,⁸ and the process of evaluation was planned to be ethical and transparent to the key stakeholders, faculty and students.⁹ Thequestionnaire contained a mix of closed and open-ended questions to obtain respondents\' feedback (Table-1).

The combination of these two approaches was considered more effective for evaluation in the backdrop of Kirkpatrick\'s hierarchy (Figure),

and enhances the possibility of evaluating programme\'s impact on the participants\' outcome¹⁰ (Table-2).

The model was chosen for its simplicity, economical value and known capability to document proficiency, process and learners\' perception regarding a given programme. The design also facilitated suggestions for improvement which is vital for any formative evaluation.¹¹
Programme graduates were sent an email explaining the study targets, evaluation process and assuring them of anonymity in the process. A questionnaire was sent to each graduate attached with the email and the graduates were asked to send the filled questionnaire on the same email address within seven days. Graduates whose questionnaires were not received after seven days were sent reminder emails. Those who still did not respond were sent mobile text messages, then called through phone and asked to complete and send back the questionnaire at their earliest. Non-responders to telephone calls were contacted again through telephone and were asked the questions directly. The participants of the evaluation who were not contactable after all the efforts were looked for through their available information from CBEC. Those who were not in contact with the Centre and did not respond to mails were labelled as "lost" and were excluded from the analysis. The data pertaining to direct output of the programme was retrieved from CBEC database, faculty and website, and recorded on a separate proforma.
This evaluation examined the efficacy of the programme and was not intended to assess individuals\' performance. The data was categorised as immediate, intermediate and impact variables. These variables were analysed for frequencies and percentages. SPSS 19 was used for statistical analysis.
Qualitative data from open-ended questions in the questionnaire were analysed manually for strengths and weaknesses of the programme, and self-observed change in the behaviour of the participants. A content analysis methodology was used to develop themes and categories from responses and drawing interpretations and inferences.¹² Since, the respondents emailed their filled questionnaire, therefore, there was a potential threat to their anonymity in the evaluation process. However, precautions were taken to keep the confidentiality of respondents by shifting data to separate Excel sheet without disclosing the respondents\' identity. Anonymous forms were also used for those who completed the questionnaire on telephone.


Results

In the five-year period from 2006 to 2010, four sessions of the one-year diploma programme were conducted as no session could be conducted in 2009. A total of 50 students had enrolled in the programme, out of which 41(82%) graduated. All the 41-18(44%) males and 23(56%) females- were sent the questionnaire via emails. Seventeen (41.46%) of them responded on the first email, 10(24%) on reminder emails, and 4(9.7%) after prompting on telephone calls and mobile text messages. Overall, 31(76%) questionnaires were received for analysis. The remaining 10(24%) students included those who had migrated out of country and/or "lost" to follow-up.
The direct output data for all 41 graduates was obtained from the CBEC database regarding their academic progress after completing the diploma programme at CBEC (Table-3).

Ten (24%) of them had proceeded for Masters in Bioethics after completing their diplomas. Five (12%) had Masters from abroad (USA and Canada) while the others received their degrees from institutions within Pakistan. Sixteen (39%) were actively involved in their institutional ethics review committees (ERC) or institutional review boards (IRB). Two (12.5%) of them were chairing the committees, and 2(12.5%) were secretaries of their respective ERC/IRB. One (6.25%) graduate was the chair of a Bioethics Unit at a university in Saudi Arabia. Two (12.5%) graduates were editors for bioethics related medical journals.
The immediate outcomes were obtained for 31graduates who responded to the questionnaires. All agreed that their knowledge about the subject of bioethics had significantly increased during the diploma programme; 28(90%) respondents stated observing a change in their behaviour and way of working after the diploma. None thought otherwise (Table-4).


Besides, 29(94%) respondents initiated their bioethics projects in their workplace or institutes. Of them, 15(52%) were continuing after 2 to 5 years of graduation (not a requirement of the programme). Further, 22(71%) respondents also initiated other bioethics related educational activities in their surroundings and, according to their beliefs, more than half of these activities had been sustained over a "significant" period of time.
The impact of the programme was determined through publications or presentations made by the participants after completing the diploma programme. Before the PGD, 26(84%) respondents had not published anything on bioethics. After acquiring the diploma, however, 18(58%) had published papers in local and/or international journals. Besides, 29(94%) had attended bioethics conferences or seminars, while all the respondents had made presentations and were involved in disseminating knowledge on the subject.  Regarding the quality improvement at work-place, 10(33%) believed that they had brought major changes in their work environment, 17(55%) thought they had managed some change towards betterment but not as much as they wanted.
As for the perceptions about the programme, 28(90%) respondents answered the open-ended questions. Regarding the strengths of the PGD programme, identified themes included "educational environment", "process of learning", "curriculum" and "teaching methodologies". Use of different teaching methods and mediums like artwork from known artists (depicting social issues) and movies which touch upon moral and ethical concerns in different walks of life along with distant learning experience and effective feedback were mentioned most commonly as the strengths of the programme. The graduates also considered diversity of topics discussed in small group interactive discussions as an important strength of the PGD (Table-5).

Themes for the question on weaknesses included "time constraints" and "teaching methodology". Most of the issues with time constraints were due to other professional commitments of students during the course. Some of the students found sessions on philosophy and law as "not up to the mark". Overall, the responses were in favour of the strengths with very limited weaknesses.
Elaborated answers were received for the question regarding change in behaviour after PGD. The changes that were mentioned were classified into two themes: "internal changes" and "external changes". The respondents identified a number of internal and external changes within themselves. Internal changes like increase in motivation to acquire more knowledge about bioethics, sense of responsibility towards the patients and community, and increased understanding of other people\'s perspective were frequently mentioned. Improved skills in teaching, communicating with patients and conflicts resolving in ethical issues were considered external changes in behaviour by the respondents (Table-6).

Suggestions to improve the programme mostly addressed the same issues that were identified in the answer for weaknesses of the programme.


Discussion

Evaluation of a programme is one of the essential components of the educational process.¹³ It is broadly defined as "the process of obtaining information about a course or programme of teaching for subsequent judgment and decision-making".¹⁴ This process comprises different methods, like questionnaires, interviews, focus groups, site visits, etc,¹⁵ and  involves a systematic assessment of the efficiency of the programme and identifies whether programme objectives have been achieved or not.¹⁶ The Accreditation Council of Graduate Medical Education (ACGME) defines "Outcomes" evaluation as "evidence showing the degree to which programme purposes and objectives are or are not being attained".¹⁷ The PGD in Bioethics programme at CBEC-SIUT is the first in public-sector medical institutes of Pakistan. It has never been formally evaluated by external evaluators since its inception. The current study evaluated the programme\'s success in achieving its aims, including the impact that the programme has on the medical institutes and bioethics related educational and research activities overall. The programme was evaluated keeping in mind the cardinal questions in any evaluation "whose opinions matter?" and "what would really be meaningful to them?"¹⁸
The deliberation in the wake of initiation of any such programme is to develop "torch-bearers", teachers and role models in the subject. Those who can further disseminate the information and knowledge in their respective workplace which is considered a need of medical institutes of the country at this time.¹⁹ Therefore, the meaningful impact of programme was observed through identifying the work that the graduates of PGD have commenced since graduation. Another consideration was the "need for enhanced stakeholder participation in evaluation".²⁰ During the entire process of evaluation in this study, from inception to analysis, the faculty of CBEC was involved at all levels without jeopardising the confidentiality of study participants and integrity of results.
The CBEC\'s PGD is a distinctive programme to start with. After doing an extensive literature search, it can be said that there is no example of such a programme where healthcare professionals are given free-of-cost training and hands-on skills in bioethics, for a period of one year, leading to diploma in the subject. Therefore, it becomes difficult to evaluate the programme, as standards as such have not been set. Overall, the responses have shown the programme to be successful in not only disseminating knowledge through graduates (making presentations and attending seminars), but also improving the profession overall through trained and qualified graduates taking up positions in ERCs, initiating activities to increase awareness of patients, students and co-workers, and publishing research papers on the subject of Ethics.
The qualitative part of evaluation provided deeper insight into how the graduates thought about the programme. The issues with qualitative approach are concerned with credibility of data and reliability of the process of analysis.²¹ For credibility of data, only those interpretations of keywords were considered for themes that multiple graduates mentioned in their responses. To ensure reliability of results, the responses were examined for themes and categories by two examiners and similarities between their interpretations were recorded for drawing inferences.
The responses that were received as qualitative data were rich and many facets of graduates\' personal experiences and thoughts emerged spontaneously in the statements. Most of the feedback of graduates validated the programme with appreciation for the contact sessions, process and convenience of distant learning, diversity of course content and use of different teaching methodologies and mediums. Feedback reinforced respondents\' preferences for small group teaching, discourse in discussions on sensitive issues, provision of multiple facilitators for varied subjects within bioethics and contextual part of curriculum rather than theoretical. As a study done in Glasgow has reported similar findings while evaluating an Ethics curriculum in the first year of medical education.²² At times emotional touch was also evident from the text of the responses. Two examples are given here:
"Bioethics Diploma has changed my horizon and has provided me with a new vision to address usual problems. It has provided me with tools to tackle dilemmas in clinical medicine and I feel myself to be better equipped to handle ethically and morally critical situations"
"It not only introduced the subject in its real sense but also provided a strong guideline for furthering the education of bioethics. More importantly, it generated in me a strong sense of responsibility, not only to enhance my own ethical behaviour towards personal and professional life, but also developed a sense of responsibility to improve surrounding environment."
Graduates frequently used words like "changed mindset" and "paradigm shift". An evaluation of continuing medical education (CME) programme on Ethics in Boston, USA, also reported use of emotionally charged statements and words in response to their open-ended questions. The language reflects respondents\' enthusiasm in learning, involvement with the concept and internal changes due to newly acquired knowledge of ethics.²³
Some graduates used the negative notion in statements for strengths of the programme, for example, "life has become miserable after doing the diploma", "it is very difficult to work now" and "I question myself on everything now", referring to the norms in clinical practice that now appear to them as unethical behaviour, causing discomfort in accepting and doing the same. This shows that the diploma programme has brought real change in the thought process and behaviour rather than just providing knowledge.
The programme uses a number of proven teaching tools like small group discussions, movies and role plays, to develop skills of identifying dilemmas and moral reasoning. Almost all the graduates have supported and praised the effective use of a variety of teaching methods. Similar preferences have also been reported by a study from the United Kingdom.²⁴ The bulk of the feedback was given by respondents for answering questions regarding strengths of the programme and changes in their behaviour after the diploma. Very little was written in the weakness part of the questionnaire. Although anonymity and confidentiality was ensured at multiple levels, this may have contributed in the restricted response, which is not uncommon during qualitative analysis.²¹
Similarly, suggestions for improvement of programme given in feedback were less than expected. They were mostly based on individual graduate\'s personal experiences, corresponding to weaknesses mentioned in their respective questionnaires. Another limitation of this evaluation was due to the fact that graduates of CBEC belonged to different parts of country and it was not financially feasible to interview each graduate in person, although this would have been more fruitful for the strength of qualitative data. Questionnaires were therefore used as the next best alternative. The graduates who are not in contact with CBEC and/or did not respond to the emails were excluded from analysis. This further reduced the already small sample size and may have influenced the results.


Conclusion

The evaluation process demonstrated evidence that the PGD programme of CBEC has achieved its objectives-driven outcome with tangible impact in terms of changed behaviour of its graduates.


Acknowledgement

We are grateful to Dr. Syed Moin Ali, Director DME, Taif University, KSA, for his valuable guidance, and to Professor Farhat Moazam and Dr. Aamir Jafarey of the CBEC for their guidance and cooperation. We also thank all the participants.

1. Fulford KWM, Yates A, Hope T. Ethics and the GMC core curriculum: A survey of resources in UK medical schools. J Med Ethics 1997; 23: 82-7.
2. General Medical Council. Tomorrow\'s Doctors. London: General Medical Council; 1993. [online] [cited 2013 May 22]; Available from URL:http://www.gmc-uk.org/Tomorrows_Doctors_1993.pdf_ 25397206.pdf on 16-5-2012.
3. Shamim MS, Shamim MS. Medical Ethics: A slow but sustained revolution in Pakistan\'s healthcare. J Pak Med Assoc 2010; 60: 706-7.
4. Hafferty FW, Franks R. The hidden curriculum, ethics, and the structure of medical education. Acad Med 1994; 69: 861-71.
5. Campbell AV, Chin J, ChuanVoo T. How can we know that ethics education produces ethical doctors? Med Teach 2007; 29: 431-6.
6. Center of Bioethics and Culture, Sindh Institute of Urology and Transplantation, Pakistan. [online] [cited 2013 May 22]; Available from URL: http://www.siut.org/bioethics/.
7. Musick DW. A conceptual model for program evaluation in graduate medical education. Acad Med 2006; 81: 759-65.
8. Armstrong EG. An outcome approach to evaluate professional development programmes for medical educators. Ann Acad Med Singapore 2007; 36: 619-21.
9. American Evaluation Association. Guiding principles for evaluators. Am J Eval 2008; 29: 397.
10. Siddiqui ZS. Evaluation paradigms in medical education. J Coll Physicians Surg Pak 2006; 16: 291-3.
11. Ziaee V, Ahmedinejad Z, Morravedji AR. An evaluation on medical students\' satisfaction with clinical evaluation and its effective factors. Tehran University of Medical Sciences. Med Educ Online [serial online] 2004; 9: 8. Available from URL: www.med-ed-online.org/f0000082.htm on 10-3-2011.
12. Hsieh HF, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res 2005; 15: 1277-88.
13. Morrison J. ABC of learning and teaching in medicine: Evaluation. BMJ 2003; 326: 385-7.
14. Goldie FJ. AMEE Guide No. 29: Evaluating educational programmes. Med Teach 2006; 28: 210-24.
15. Wall D. Evaluation: Improving practice. In: Swanwick T, editor. Understanding medical education: Evidence, theory and practice. Chichester, West Sussex: John Wiley & Sons; 2010; pp336.
16. Musal B, Taskiran C, Gursel Y, Ozan S, Timbil S, Velipasaoglu S. An example of program evaluation project in undergraduate medical education. Educ Health 2008; 21: 1-7.
17. Accreditation Council of Graduate Medical Education, Outcomes Project, Frequently Asked Questions, #3. [online] [cited 2011 Sep 16]; Available from URL: http://www.acgme.org/outcome/ about/faq.asp#a3
18. Cook D. Twelve tips for evaluating educational programs. Med Teach 2010; 32: 296-301.
19. Saeed N. An innovative method of self-education through a bioethics group: the Ziauddin experience, Karachi. J Med Ethics Hist Med 2009; 2: 17.
20. Layne J. An artful approach to program evaluation. Optimum, JPublic Sector Manage 2008; 29(2/3): 1-9.
21. Wahlqvist M, Scott A, Bjorklund C, Dahlgren G, Lonka K, Mattson B. Impact of medical students\' descriptive evaluation on long-term course development. BMC Med Educ 2006, 6:24.
22. Goldie J, Schwartz L, Morrison J. A process evaluation of medical ethics education in the first year of a new medical curriculum. Med Educ 2000; 34: 468-73.
23. Levine MD, Scott L, Curran WJ. Ethics rounds in a children medical center: Evaluation of a hospital-based program for continuing education in medical ethics. Pediatrics 1977; 60: 202-8.
24. Mattick K, Bligh J. Teaching and assessing medical ethics: Where are we now? J Med Ethics 2006; 32: 181-5.