Family Medicine Diploma Program in Jordan: Components for Success and Sustainability

Jordan currently has residency positions available for approximately 20% of graduates from Jordanian medical schools. This percentage decreases when factoring in Jordanians who complete medical school in other countries and desire to return to Jordan to work. ¹ Approximately 30 of the 400 residency positions are in family medicine. Most graduates never complete a residency program and instead work as outpatient general practitioners (GPs), often in rural settings without readily available mentors. ² Due to limited primary care training, GPs make excessive referrals to specialists, overburdening the specialty clinics and escalating costs. For example, referring any patient with chest pain to cardiology is common practice. ³

Because most graduates end up practicing in a purely outpatient setting, a strategic focus on outpatient training is key. The process to set up and manage a residency program is long, and the current need for more well-trained GPs is overwhelming. To address this gap, past projects developed shorter, diploma level training programs directed toward GPs unable to enroll in a full residency program. The World Health Organization (WHO), in association with the Jordanian Ministry of Health (MOH), launched a pilot diploma program for 1 year in 2020. ^{4, 5} The program was mainly online except for a few face-to-face group training sessions. The training had mixed results with some trainees indicating that the training was useful, but the doctors who managed the project reported difficulty evaluating whether the trainees were indeed applying the training to their practice. During each of the last 5 years, the United Nations Relief and Works Agency (UNRWA), in association with Rila University in the United Kingdom, provided a 1-year diploma program to their GPs. Each participant from UNRWA engaged in a mixture of weekly synchronous and asynchronous online learning and also was assigned a board-certified family medicine mentor to visit them for approximately 1 hour every 2 weeks to observe their work and provide feedback in their respective clinics. A study of that program revealed that the program increased the knowledge of GPs immediately after the training. However, the inability to assess the ongoing application of the training in each UNRWA GP’s practice was a program limitation. ³ Another difficulty noted from the UNRWA program was that the GPs were overwhelmed by their daily workload of 80 to 100 patients and did not have the time needed to focus on learning from their patient encounters during their mentor’s visits. ³

Currently, the WHO Regional Office for the Eastern Mediterranean has launched a 2-year family medicine diploma in several countries in the region. ⁶ This program relies on asynchronous online learning so that GPs can study at their own pace but does not involve on-the-job training. No outcome data on this effort were available.

ERASMUS+, a European Union program supporting education and training, provided the money and technical expertise of European family medicine faculty to create a diploma program that enhanced prior efforts. Specific additions to this program were online interactive (instead of asynchronous) lectures, regular assignments and quizzes that were reviewed with the trainees, dramatically increased amounts of on-the-job supervision, faculty development for Jordanian family medicine university faculty, and the development of a train-the-trainer program for MOH family medicine mentors with the goal of creating a sustainable program. ⁷ The diploma program curriculum can be found in Appendix 1.

In this paper, we discuss the additional steps taken to ensure sustainability of the diploma program and show the evaluation data gathered for three levels of Kirkpatrick’s four-level model.

A series of surveys, sites visits, personal interviews, and bilateral meetings were conducted by the project consortium, which consisted of members from each of the three European and three Jordanian universities.⁸ Eight individual meetings and three discussion groups were conducted with MOH GPs, MOH family physicians, and Jordanian family medicine faculty from three different medical schools. A needs assessment questionnaire was designed during these sessions for future diploma trainees and family medicine trainers.⁹ Items identified critical for program sustainability included buy-in from local stakeholders, improved on-the-job training, incentives for the program trainees, an adequate pool of well-trained mentors, differentiating diploma trainees from family medicine residents, and a sustainable financial model.

The Kirkpatrick evaluation model includes four levels: Level 1 examines the relevance of the program to the participants; Level 2 measures the degree to which participants acquire the intended knowledge, skills, attitude, confidence, and commitment based on their participation; Level 3 measures the degree to which participants apply what they learned during the program or initiative when they are back in their environment; and Level 4 measures the degree to which targeted organizational outcomes occur as a result of the initiative.¹⁰

Level 1 data were collected from the first cohort through a survey consisting of six open-ended questions that explored how the training impacted clinical knowledge and practice. Answers were read and organized into themes. ¹¹ See Appendix 2 for the questions. Focus groups also were held at the end of the first cohort with the GPs (participants) and the family medicine mentors, and results are published elsewhere.

Level 2 data were collected from pre- and postknowledge tests to assess the basic primary care knowledge of each GP during the first month of the program and upon completion of the program for both trainee cohorts.

Level 3 data were collected from a survey given to the second cohort of trainees 6 months after the completion of the program to evaluate the impact of the program on behavioral change. ¹² Level 4 data are planned but not yet collected.

The program was developed in the context of Capacity Building in Higher Education (2021–2024). All data were collected as part of the program’s evaluation and served as quality indicators for reporting to the contractor. Consent for participation was obtained at the beginning of the project and was associated with formal registration in the program. All data were processed and analyzed anonymously.

The first cohort included 14 GPs, and the second group had 30. The mean age for the 44 GP diploma trainees was 39.6 years (range 35–49 years), and they practiced as GPs for an average of 10.2 years (range 5–17 years). Males comprised 93% of the total participants. Participants were located across Jordan with 68% in rural areas. Criteria for diploma admission included having at least 5 years in clinical practice and being 35 years of age or older, making them ineligible to enroll in a local residency program (Table 1).

Level 1: The open-ended questions survey results have not been previously published. Nine out of 14 (64%) of first cohort trainees completed the open-ended questions survey. Responses were divided into the following themes: enhanced understanding of primary care, impact on patient management, ethics and values, and strengthened communication skills.

• Understanding primary care. Trainees reported adapting a more organized approach to the patient encounter and integrating history-taking with the exam.

• Patient management impact. Trainees altered their use of antibiotics and recognized that sometimes psychological counseling was sufficient.

• Ethics and values. Trainees tried to give more time and autonomy to the patient as well as placed greater focus on confidentiality, honesty, and empathy.

• Strengthened communication skills. Trainees noted improved listening skills as well as increased ease in integrating family members into the consultation with shared decision-making and in breaking bad news.

These findings demonstrated that the first cohort was satisfied and engaged with the training program (Table 2).

Ten GPs and five mentors engaged in focus groups in the first cohort. In general, GP trainees were happy with the program. Trainees noted a need for uniform access to EBM (evidence-based medicine) databases to complete the assignments. They wanted more face-to-face and practice opportunities. Trainers suggested enhancing practical skills training and increasing Arabic materials. English lectures needed to be delivered more slowly, with an Arabic trainer present for translating concepts. More practical skills were needed, such as interpreting spirometry and other test results. Feedback from both the survey and focus groups was used to make curriculum adjustments for the second cohort of GPs (Table 3).

Level 2: Pre- (n=25) and posttest (n=35) knowledge assessments showed statistically significant increases in scores in both cohorts (respective t test and P value, .06 and .001; Table 4).

Level 3: The second diploma cohort completed a survey 6 months after graduation and program completion to evaluate for improved behaviors. Of the 30 GPs, 21 completed the survey. Seventy-one percent (15/21 GPs) expressed improved confidence when working with patients. Nearly all (20/21 GPs) reported a decrease in the amount of writing antibiotic prescriptions; 86% (18 GPs) reported a decrease in the amount of referrals to secondary care; and 81% (17) needed less advice from family medicine consultants. Regarding managing chronic conditions in the community, 62% (13) felt more equipped. ¹²

Buy-in. To address the need for local buy-in, multiple universities and the Jordanian MOH were involved in the development and implementation of the project at its earliest stages. Previous projects were delivered directly from MOH or from outside nongovernmental organizations. From the beginning of our project, MOH agreed to allow for the training of their GPs, use of their family physicians as mentors, and use of their clinic spaces. Also, faculty members from each university agreed to take part in the training of GPs. During the formation process, meetings to identify sustainable funding mechanisms were held with all stakeholders, including MOH, Ministry of Higher Education, and local university administrations.

Competition with family medicine residency. A barrier to the implementation of previous programs was the fear of diploma programs eliminating the need for family physicians. To distinguish our project as something entirely different from a family medicine residency program, the name Primary Care Diploma program was used rather than Family Medicine Diploma program.

Incentives. To incentivize trainees in the program, the Jordanian MOH preapproved pay increases for graduates of the program. Another incentive was the reopening of the opportunity for the top graduate to enroll in the MOH Family Medicine Residency Program, usually reserved only for people under the age of 35 years. ERASMUS+ funded the diploma program for 3 years, the maximum time period allowed. The program was paused after the completion of the funding until a formal accreditation of the program by the Ministry of Higher Education was secured. The accreditation was recently received, and the program will restart in fall 2025.

Robust group of family medicine mentors. Previous projects that offered face-to-face on-the-job training relied heavily on two to three mentors. This workload was unrealistic, so our project engaged 10 board certified family physician mentors from MOH to provide more individualized training to the GPs and to ensure greater sustainability of the program. The mentors modeled patient interactions in front of the GPs, supervised the GPs seeing patients on their own, and provided immediate feedback. To ensure local mentors were well-trained, a train-the-trainer program was developed. Family medicine faculty from the three European universities delivered training to the mentors. Training occurred within Jordan as well as at European university sites. Radboud University led training on adult learning principles, giving feedback, and mentoring learners through difficulties. KU Leuven delivered training on evidence-based medicine, and Queen Mary University of London focused on coaching and mentoring training. Mentors were encouraged to discuss any struggles they were having with their GPs with Jordanian family medicine faculty at any time to receive outside support.

Creating postgraduate medical programs in low- and middle-income countries that result in practice change is incredibly difficult to achieve. For a program to truly be sustainable, many factors need to coalesce. As mentioned earlier, these factors include buy-in from local stakeholders, an adequate amount of on-the-job supervision in a setting conducive to learning, incentives for the program trainees, an adequate number of well-trained mentors, a method of differentiating diploma graduates from physicians who completed a family medicine residency so that both pathways are sustainable, and a sustainable financial model.

Low- and middle-income countries like Jordan often rely on outside support, and in this project ERASMUS+ funding allowed the consortium to improve on previous diploma endeavors. With the help of all six partners, three levels of evaluation evidence in the Kirkpatrick Model have been demonstrated. Level 3 evidence of self-reported behavior change, including a change in antibiotic prescription practices and decreased specialist referrals, was reported 6 months after graduation from the program. The primary care diploma program plans to welcome the third cohort of trainees in fall 2025.

As noted in previous projects, a successful program must ensure that the information delivered is not just understood but is also implemented into the clinical practice of the learners. Many online courses and training programs offer degrees and certificates, but educational meetings alone are not likely to change complex behaviors.¹³ Absolutely key is a mentor to help guide the trainee through the process of diagnosing and treating patients. ¹⁴, ¹⁵ Trainees need to have practices modeled for them: how to comfort a patient in distress, how to break bad news, how to encourage a patient to lose weight or quit smoking, and many other types of encounters that are routine in primary care. Once they have observed a mentor doing these things, trainees themselves need to be observed leading these encounters and then to receive feedback to help them improve the next time they manage the same situation. Furthermore, enough time for trainees to be able to reflect upon and learn from their patient encounters with their mentors is needed for trainees to develop.

Our train-the-trainer program played a key component in developing qualified mentors for the diploma program. A recent ERASMUS+ grant procured by a consortium of all Jordanian medical schools, aimed at developing well-trained medical preceptors, will further increase the amount and skills of mentors in the medical setting available for future diploma programs.

Previously, on-the-job supervision had been difficult to sustain. Developers of the model adapted with the ERASMUS+ program decided that placing trainees at the mentor’s clinic allowed for focused teaching and avoided the distractions of an excessively large panel of patients. This approach also addressed the problem of mentors needing to travel long distances to rural settings multiple times. While not training in the trainee’s own setting is not ideal, it is only for the duration of the program, and early observational evidence suggests that the mentoring continues after graduation through platforms like WhatsApp and phone calls.

During the first year of the program, faculty from European universities led the sessions with the support of faculty members from Jordanian universities. Shifting the delivery of the material from European faculty to Jordanian faculty during the second cohort increased participant understanding and allowed local faculty to speak to the local context. Modifying the curriculum for the second cohort also allowed us to focus on material most relevant to the trainees.

Even with all these adaptations and additions to this program, sustainability of the program remained the key challenge to the project. Specifically, a long-term sustainable financial model proved to be the largest obstacle. Recently, the program received a higher level of certification at the national level, which provides learners with an increased incentive to participate in the program. The hope is that MOH will agree to pay for the training of their GPs and that non-MOH GPs will enroll themselves due to the higher pay they could receive upon completion of the program.

Limitations to the current research included only short-term measures of the program’s success 6 months after graduation and small numbers of participants to date with some missing data. Despite the limitations, we demonstrated evidence of three levels of Kirkpatrick’s evaluation model. We identified a feasible way to accomplish on-the-job training for trainess and to expand the pool of trained family medicine mentors. The conflict between family medicine residency training and the family medicine certificate program was resolved, and we are continuing to develop a robust and stable group of family medicine mentors. Future work will include assessments of the trainer program and long-term measurements to assess the learners’ implementation and actual change in their practice as described in Kirkpatrick’s Level 4, which includes the program’s impact on patient outcomes such as diabetes measures, cost savings, and improved health care team performance.

Diploma programs fill an important void in the expertise of GPs and can benefit the quality of patient care by providing relevant, outpatient-focused training and on-the-job mentorship for GPs who are not able to enroll in full residency programs. Building on prior diploma program efforts, our program added critical elements to build a successful and sustainable program. Evaluation of the long-term impact of the program (Kirkpatrick Level 4) is needed to assess whether graduates are continuing to apply the new knowledge in their practice. Moreover, greater attention needs to be given to a sustainable financial model.