The Critical Thinking Skills of Practicing Family Physicians: A Population-Based Cross-Sectional Study

Cognitive errors due to cognitive and affective biases, faulty or inaccurate reasoning, and misunderstanding probabilities can lead to diagnostic errors and patient harm, and are some of the most complex issues surrounding patient safety.1 To begin to ameliorate some of these biases and errors, it has been suggested that education, beginning in medical school, teach physicians analytical and rational thinking.2,3 While the processes of clinical reasoning (CR) are debated,4 it is generally accepted that the analytical and rational aspects of CR consist of the ability to identify a problem, select and evaluate pertinent information, recognize assumptions, formulate appropriate hypotheses, and draw valid conclusions and critical inferences.5-8 These aspects of CR are proximal to scientific thinking and the cognitive and metacognitive processes and dispositions to make meaning of information.9 Scientific thinking is a domain of the broader construct of critical thinking (CT),10 with CT being essential for CR. CT consists of six skills: interpretation, analysis, evaluation, inference, explanation, and self-regulation.11,12

CT skills serve as a good predictor of academic success in health care,13-16 yet it is unknown what the level of CT skills of practicing physicians are and how continued development of CT skills can be supported. Critical thinking is essential to the safe and effective practice of family medicine, as family physicians (FP) must diagnose and manage numerous undifferentiated conditions17 in complex systems18 that require a broader general knowledge base than those in a specialist field.19,20 As the initial point of contact with a patient, it is important for a FP to be able to engage CR bolstered by skillful CT to make an initial diagnosis and pursue an appropriate management plan.17,21 Deficits in CT skills can contribute to the inability to make effective clinical decisions when faced with clinical uncertainty, a situation with potentially devastating consequences.22

Previously, CT skills have been examined in residents, showing no change between entering and completing residency.23 The next step is to examine CT skills in practicing family physicians. Currently, there is limited knowledge regarding the level of CT skills of practicing physicians, and FPs specifically. It will be important to examine if change in CT occurs in practice when compared to residency and understand what potential variables may be predictive of CT skills in practicing FPs.

To better understand the CT skills of practicing FPs, and potential changes in CT, we designed a population-based, cross-sectional observational study. Three aims for the present research were developed: (1) examine the level of CT skills of practicing family physicians using the California Critical Thinking Skills Test (CCTST), (2) compare the CT skills of practicing family physicians with the CT skills of family medicine residents using a cross-sectional design to determine group differences, and (3) identify which individual variables and practice characteristics are predictive of CT skills.

We recruited family physicians from Alberta, British Columbia, Ontario, and Quebec through professional list serves, at professional conferences, and by site leads for primary care networks. Family physicians were recruited between May and December 2019. An initial contact email was distributed; interested participants were sent a consent form and a link to the CCTST. Participants completed the test on personal devices. Completion of the test ended participants’ involvement in the study. Data collected from residents in June 2013 at the end of a residency program at a single institution (previously presented by Ross et al23) were used for comparison purposes. Research Ethics Board 2 at the University of Alberta granted ethics approval for this study. We used STROBE cross-sectional reporting guidelines.24

We used the CCTST test as a measure of critical thinking skills. This 34-item test demonstrates strong validity evidence, is predictive of academic success in health professions, and has been used extensively in research in health professions.15,16,25,26

The CCTST is comprised of an overall score and five subscales, each measuring a different aspect of CT.11,12 The overall score describes strength in using reasoning to form reflective judgments about what to believe or what to do (scored: 0-35). Analytical skills are used to identify assumptions, reasons, themes, and the evidence used in making arguments or offering explanations (scored: 0-8). Inference skills are the ability to draw conclusions from reasons, evidence, observations, experiences, or values and beliefs (scored: 0-15). Evaluative skills are used to assess the credibility of the claims people make and to assess the quality of the reasoning people display when they make arguments or give explanations (scored: 0-10). Inductive reasoning relies on estimating likely outcomes and decision making in contexts of uncertainty (scored: 0-15). Deductive skills are used to determine the precise logical consequences of a given set of rules, conditions, beliefs, values, policies, principles, procedures, or terminology (scored: 0-15).27 The CCTST requires approximately 45 minutes to complete.

Data were analyzed using SPSS Version 25.28 We used descriptive statistics to examine practicing FP CCTST scores and demographic characteristics, and we conducted univariate analysis on categorical variables. The primary analysis consisted of two steps. First, we conducted a one-way multivariate analysis of variance (MANOVA) to determine differences between practicing FPs and residents on the CCTST and its subscales. Second, we conducted a hierarchical multiple linear regression with the practicing physicians’ data to determine predictor variable importance for CT skills in practicing physicians.

Assumption Checks. Assumptions for MANOVA were checked. Examination of Q-Q plots and Shapiro-Wilks test indicated multivariate normality. Residual plots and Q-Q plots identified a borderline outlier. One resident was an outlier on percentile rank and the induction subscale, though not on other scores; we decided to remove the participant. We identified high multicollinearity among the variables, subscale relationships were examined, and we removed the analysis subscale to carry the test forward.29 Box’s test indicated homogeneity of covariance matrices, χ² (١٠), P=.460. We checked assumptions for regression; residual plots and Cooks Distance indicated there were no outliers. We identified no violations for the Durbin-Watson test for autocorrelation, variance inflation factor for collinearity, and Q-Q plots and measures of skewness for normality. We deemed the sample size adequate for the number of participants per predictor included in the model.29

We recruited 62 practicing family physicians. The sample was composed primarily of participants of Caucasian (41, 66%) and Asian (14, 23%) backgrounds. Thirty-six of the participants were female (58%). The majority of the sample worked in full service/broad scope family practice (33, 53%), followed by solely office-based family practice (11, 18%). Sixty percent (38) of the sample worked in population centers with >500K people; 10% (6) worked in centers of 50K-100K people, 20% (12) worked in centers with 10K-50K people, and 10% (6) worked in centers of <10K people. There were 53 Canadian medical graduates, eight international medical graduates, and one Canadian who had studied abroad. Twenty-seven percent of the sample had a graduate degree; two PhDs, and 15 master’s degrees. Participants engaged in a wide range of continuing medical education (CME) with the majority indicating their primary form of CME was conferences (28) and self-study (16). For further sample characteristics see Table 1.

Secondary data from 60 residents was available from the Ross study.23 Thirty-five (56%) of the residents were female with a mean (SD) age of 27.9 (5.1) years.

Family medicine residents and practicing FPs exhibited a wide distribution of scores on overall CCTST score and the subscales. Residents tended to perform better than practicing physicians on all measures, except for the analysis subscale (Table 2) and scored higher on mean percentile rank (Table 2). Percentile rank is a norm referenced score based on an aggregated sample of test takers.30 We conducted univariate analysis on the categorical demographic variables, and found no significant differences for overall CCTST score or the subscales on any categorical variables. The second highest scoring participant, scoring in the 97th percentile, was 64 years old; only one resident scored higher, scoring in the 98th percentile.

We ran a one-way MANOVA (independent variable: practicing physicians vs residents and dependent variables: inference, evaluation, induction and deduction). Results indicated there was a significant difference between residents and practicing family physicians on the subscales of the CCTST (F [4,116]=4.75; Wilk’s ƛ=.86; P<.001).

Follow-up univariate tests identified significant differences on the evaluation subscale (F [1,119]=14.56; P<.001), and induction subscale (F [1,119]=9.92; P=.002; Table 2). There were no significant differences on the analysis subscale (F [1,119]=.20, P=.659), but there were differences on the overall CCTST score (F [1,119]=5.02, P<.027); and percentile rank (F[1,119]=4.97, P<.028). Subscale effect sizes indicate small to large effects for the subscales and overall score, except for analysis.

We used only data from practicing FPs for a series of hierarchical multiple regressions with overall CCTST score and each subscale as dependent variables. We used age, clinical days per week, patients per day, research hours per week, and hours of teaching per week as predictor (independent) variables. Overall model fit was significant for overall score, evaluation, and induction. Age was found to be a significant predictor of lower CT scores for overall score and for each subscale, except for analysis. Research hours per week and clinical days per week were also predictive of lower scores on evaluation and induction, respectively (Table 3).

As age was the most common and powerful predictor of CT skills, we conducted a follow-up analysis to better understand the relationship between age and CT. For comparability across subscales, all scores were scaled (z score, mean=0, SD=1). We used polynomial regression to test nonlinear relationships, quadratic, cubic, and quartic. Results indicated that the data were best fit by a quadratic model for all scores, except for analysis. Analysis displayed no significant linear or nonlinear relationships (Table 4). No other higher order polynomials were significant. The parameters of the quadratic regression were used to predict scores on the overall and subscale scores at 5-year age intervals (Figure 1). The follow-up analysis indicates a negative quadratic relationship between age and CT skills.

We found that family medicine residents and practicing FPs differ in CT skills. Practicing physicians generally exhibited lower scores on the CCTST compared to residents, with age as the strongest negative predictor of CCTST scores. Group differences are likely due to the homogeneity of age for the resident sample compared with the heterogeneity of age amongst the practicing physicians. CT skills appear to decline at the greatest rate after 60 years of age. It appears the cognitive demands of the medical profession do not protect against normal age-related decline in the aspects of cognition related to CT skills. The results of this present study converge with other findings on aging and cognition showing a normal pattern of senescence, especially when considering the quadratic relationship.31-33 Deficits in the cognitive skills related to CT are most strongly related to declines in working memory (WM).32 Though the sample used was restricted to family physicians, it can be hypothesized that the results are generalizable to other specialties in medicine.

Practicing physicians performed similarly to residents on the analysis subscale. Analysis measures the ability to identify assumptions, reasons, themes, and the evidence used in making arguments or offering explanations. These skills are concordant with the use of pattern recognition and illness scripts in the diagnostic process and are more reliant on knowledge.34 As humans age and fluid intelligence declines, crystallized intelligence remains fairly stable until approximately the eighth decade.33 The amount of knowledge that a person has, and can develop, does not decline at the same rate as WM and so the aspects of CT that do not rely on WM remain more stable.

Though individual variance is important when considering aging and CT skills, as shown in Figure 1, CT skills are not as strong after age 60. Previously, a small but significant negative relationship has been found between diagnostic accuracy and age.35,36 Lower diagnostic accuracy and consideration of the need for CT skills in CR,1 and the need to engage System II processing for CT,37 indicate that it is important to assess CT skills longitudinally; stability in pattern recognition may not be protective for optimal diagnostic accuracy. Declining CT skills could be a harbinger of reduced clinical ability.

Three unexpected findings were identified in the present study. The first two are the negative relationship between research hours and evaluation, and clinical days worked and induction. It could be expected the two variables would be positively related with CT skills as engaging in research and spending more time in the clinic may hone CT skills. However, patients seen per day was not correlated with clinical days worked, so spending more time in the clinic may not increase the opportunity to practice CT skills. It is possible more time spent in the clinic is related to less time engaged in reading or other activities that would support induction skills, and may be concurrently associated with greater fatigue, which can negatively impact cognitive abilities.38-40 The negative relationship between research hours and evaluation scores is also counterintuitive. More investigation is required to understand how conducting research in conjunction with practicing medicine influences CT skills. Finally, no relationship was found between population size served and patients seen per day and no differences were found for CT skills based on population size served; therefore, based on the current data, it does not appear these variables interact. Further investigation of the influence of clinical days worked, research hours, and populations served is necessary. A more granular approach along with purposeful sampling is likely required to target these issues.

It is important to assess CT skills longitudinally across physician populations due to the importance of CT skills in the diagnostic process and the potential negative implications of declining CT skills for diagnostic accuracy. Large sample frames and longitudinal data collection could employ data mining techniques to better identify the contributing factors to CT and changes in CT skills. Consistent assessment of CT skills could be used to inform physicians of changes in their cognitive skill set and identify physicians that may need continued professional development and support to remain in practice, or ultimately direction away from clinical practice. A measure of CT alone would not be appropriate for this but would function as one aspect of a broader evaluation of a physician’s practice.

The present study sampled family physicians exclusively. It is reasonable to expect that since the pattern of decline is comparable to samples from the general population, similar results would be observed across medical specialties. Further research on a wider range of specialties is necessary to address this hypothesis. Finally, direct comparisons to the general population or among other professions should provide further insights for physician CT skills.

This study has four main limitations. First, there was a relatively small sample size. Nevertheless, there were robust effect sizes for the differences between residents and practicing physicians indicating adequate statistical power for these analyses. Secondly, we assumed that the entire sample of family physicians was healthy with no neurodegenerative disorders. At present it is not possible to know if any of the participants had any age-related or other pathologies that may have influenced scores. Thirdly, the environmental testing conditions were not identical between residents and FPs. Residents completed the test during protected time while FPs completed the test at their own discretion. However, test forms, instructions, and allotted time to complete the test were identical. Finally, all residents were from a single institution. At present it is not possible to know if the CT skills of residents between institutions is uniform, limiting generalizability.

CT skills are an essential consideration in clinical competency. Changes in CT skills in practicing family physicians reflect normal human aging indicating CT assessment could be a diagnostic method for identifying potential declines in physician performance. CT skills should not be the only consideration, or the ultimate determinant, as individuals will vary in clinical performance. It may also be possible to develop external supports and cues to help older doctors remain effective in practice when it is expected CT skills may be declining. Further research on CT skills employing other physician specialties, other professions, and the population in general may be highly instructive of life span changes in cognitive skills.