Validation of the Use of a Large Language Model for Detecting Sentiment in Student Course Evaluation

The use of large language models (LLMs) and natural language processing in medical education has expanded rapidly in recent years.^1,2 As artificial intelligence (AI) has become a more common and a more powerful tool, health professions educators find more applications for it, such as creating or editing content, personalization of learning, and data analysis.^3,4 A 2023 guide from AMEE (International Association for Health Professions Education) identified several possible applications for artificial intelligence in health professions education (HPE) research, including literature review, data analysis, and qualitative text analysis.^5-7

Available tools, such as ChatGPT, are easy for an inexperienced user to engage with, but their methods are opaque.^5,8,9 This means that an HPE researcher can input data into an online tool and get an answer, but without knowing how the AI tool arrived at that answer. A lower level of expertise in a clinical content area has been associated with a higher level of trust in AI output, suggesting that a user with less knowledge is more likely to accept an AI response with lower face validity.^8,10

AI output is not always reliable, and some errors, particularly hallucinations, can be convincing and difficult to detect.^11,12 Fact-checking and triangulating data can validate the output of text generation.¹³ However, when using AI tools to perform data analysis, the methods and output must be validated if the intention is to use the results to make decisions.¹⁴

Validation of an LLM requires both internal validation and verification of the AI tool’s performance, as well as assessment of the validity of the tool’s output. The step of internal validation of the LLM results in an understanding of how accurately the LLM performs after training on a dataset and testing on a second dataset; internal validation measures whether the LLM produces output as expected.⁹ The assessment of validity for application is often reported as a sensitivity, specificity, or other measure of comparison; this step often uses an accepted gold standard. Various gold standards are used, but consensus exists in medicine that human-based validation is required due to the high stakes of the application of the output.^9,15

HPE applications are analogous. The AMEE guide highlighted the importance of validating the AI model and tool being used for HPE research because of the risk of errors and bias.^6,8,16-18 One place AI is used in HPE is student course and clerkship evaluations.^1,19 Evaluation data are used to make decisions such as faculty promotion and course continuation, which hold substantial stakes for students, faculty, and institutions.

This study presents the validation of a natural language processor (NLP) to identify the presence and patterns of sentiment in end-of-course evaluations from M3 core clerkships at multiple institutions. This study also presents the gold standard analysis: the interrater reliability between human identification of sentiment and LLM identification of sentiment.

An overview of the steps in the validation of the LLM and its output appears in Figure 1.

The Messick and Kane models are validation frameworks used in HPE that can be applied to assess the accuracy and reliability of LLM output for text analysis.²⁰ A 2024 article offered a series of questions authors, editors, and readers should consider when approaching a study that uses an AI tool for data analysis.⁹ These frameworks describe validation of AI tools in the context of HPE (Table 1).

The research team was composed of members from nine US medical schools accredited by the Liaison Committee on Medical Education or the Commission on Osteopathic College Accreditation; each school contributed de-identified course evaluations from core M3 clerkships for the 2022–2023 academic year. The validation dataset included de-identified written comments from four medical schools. Research team members followed institution-specific requirements for obtaining data without identifiable student information. Institutional review board approval or a nonhuman subjects waiver was obtained from each site.

Based on a review of the literature, the research team determined three potential frameworks to identify sentiment in written text: the Feelings Wheel, Plutchik’s Wheel of Emotions, and the Emotoscope Feelings Chart.^21-23 These were each assessed for completeness and accuracy of emotions in a sample of written comments from each school using a separate validation process not reported here. Based on this assessment, the Feelings Wheel was identified as the preferred framework.

A sample of written comments from each of the four pilot schools was included in the validation sample. Teams of two human coders completed two rounds of coding using the Feelings Wheel framework. Each team of two coders independently coded comments from two schools. Each round included comments from 20 respondents from each school. Different schools used different prompts, and students wrote responses of varying length and content. Thus, 20 written responses could result in multiple instances of sentiment, resulting in uneven numbers of comments identified from the responses. For example, the prompt “What was the best aspect of this clerkship?” generated a single response: “Resident clinic was a valuable learning experience, and community preceptors demonstrate a passion for teaching students.” This would be analyzed as two comments: Resident clinic was a valuable learning experience. And, separately, community preceptors demonstrate a passion for teaching students.

The coders independently identified a primary sentiment: positive, negative, neutral, or no sentiment, and, for comments with a positive or negative primary sentiment, a secondary sentiment (Table 2). The coders then discussed discrepancies with a goal of improving future consistency. During this process, coders also identified the specific words from the written comments that signaled sentiments. The intention with two human codings was twofold: to create a gold standard for the definition of the sentiment terms as applied to the dataset, and to provide specific information for training the LLM. Interrater reliability metrics were calculated between human coders using Fleiss’ κ.

We used different generations and varying sizes of LLMs. These included the bidirectional encoder representations from transformers (BERT) family of models, which are primarily trained using the masked language modeling paradigm, as well as generative language models trained auto-regressively. The generative language models showed more capabilities in zero-shot and few-shot in-context learning, which made them a strong basis for further modeling and domain-specific improvements. A traditional BERT model was selected because it is a pretrained model and lighter version. It can be run locally without using too many computer resources, while other models, such as RoBERTa, are reserved for large datasets and have longer training time. We opted against a generative AI model such as ChatGPT or Gemini in order to use a model compliant with the Family Educational Rights and Privacy Act (FERPA).

The results of the human-identified gold standard were then used to train the LLM to identify primary and secondary sentiments. We trained BERT for sentiment prediction with the following steps: (1) preparing labeled data, (2) tokenizing with BERT’s tokenizer, (3) fine-tuning a pretrained BERT model with a classification head, and (4) evaluating with accuracy/F1 metrics.

After two human coders established the gold standard for the definition of the sentiment, the LLM was trained on the gold standard. The LLM then independently coded a separate validation sample of 20 responses per school. Two teams of two human coders each also coded the same validation dataset (20 responses per school, with each team responsible for two schools). Interrater reliability between the LLM and the human coders was quantified using Fleiss’ κ to evaluate the model’s performance relative to human raters.

A total of 364 discrete comments were evaluated. Significant variations existed in the total number of comments (range 40–162) per school and the number and percentage of sentiments identified by human coders. The range of positive (30.6%–61.0%), negative (4.9%–39.5%), neutral (9.8%–19.0%), and mixed (1.7%–27.5%) sentiments varied by school (Table 3). Human-human and human-AI interrater reliability also varied by school (range of Fleiss’s k: 0.658–1). In aggregate, human-human and human-AI agreement were comparable (range of Fleiss’s k: 0.640–0.945 (Table 4).

We found that a trained LLM could identify sentiment in written comments from student end-of-clerkship evaluations with a similar interrater reliability to human coders. We observed considerable variability across institutions in sentiment distribution and interrater reliability, but the summary data showed acceptable validity. This study provides an example of the validation process for an AI tool from both the artificial intelligence and the human application perspectives.

Our approach started with determining human gold standard of sentiment identification through multiple rounds of coding before introducing an AI component. This initial step served as a benchmark to compare against the AI output, rather than assuming AI accuracy from the start. This approach to validation can be used as a framework for other educational researchers who plan to use AI tools.

One identified flaw of the use of LLMs is that the patterns of errors and biases in any given tool and dataset are unknown.²⁴ When interpreting data created or analyzed using AI, the AI tool must be trained and validated on the dataset being analyzed. This step ensures that the inputs and analysis provide reproducible, predictable, and transparent results. Previous studies have shown heterogeneous evidence of validation prior to the use of LLMs in medical research.²⁵

AI tools are particularly appealing when the volume of data is large and when existing methods for handling the data would be cumbersome.²⁶ This savings is reasonable only if the output is also valid: if the results afford meaningful contributions to decision-making.²⁷ The correct AI tool must also be selected. Our choice of LLM was based on factors including its ability to perform the analysis, use of computing resources, and FERPA compliance. FERPA compliance was a strong factor in the selection of BERT; we used an internal model that did not collect or share data. At the time of the analysis, this was not a universal feature on most generative LLMs, and for privacy and safety reasons, we considered it essential.

Course evaluation data often includes both numeric and written comment data, which is often copious in volume. Thus, these data are a tempting target for LLM application. Student course evaluations are used for faculty promotion and impact curriculum decisions.^28,29 Given the stakes inherent in the process of course evaluation, ensuring reliable output is critical.

The goal of validation is high-quality decision-making, which requires a reliable tool and useful, accurate output.^8,30 Our study demonstrated the use of several best practices for using LLMs in medical education research. Many purported uses of AI in medical education train the model but omit the step of testing the output, employing an empirical trust of the AI training process.⁹ The potential consequences of forgoing validation include a threat to the validity of both the method of analysis and the outcomes themselves.^31,32 Human-based validation is essential in establishing the LLM’s reliability and generalizability when applied to high-stakes decisions.

Interprofessional collaboration is expected in HPE. This study required experience and expertise in a range of areas. HPE faculty have the background in theory and provide the conceptual foundation; clinical faculty support the practical interpretation and implementation; and statistics experts provide methodological and analytic direction. The research team included members with backgrounds in HPE, clinical medicine, and statistics, including expertise in LLMs. As AI becomes more mainstream and is integrated thoughtfully and rigorously into HPE research and practice, high-quality research will require more collaboration of this kind.³³ Similarly, HPE researchers will need to be familiar with AI tools and techniques as they become more common in the literature and in practice.

Our evaluation has several significant limitations. This study coded for the researchers’ interpretation of student comments. Because we were using a de-identified data set, we were not able to ask students directly what their emotions were at the time they completed the clerkship evaluations. That would be the true gold standard.

While our pilot study demonstrated the feasibility of using LLMs for emotional analysis in end-of-course evaluations, it also highlighted the need for more research on this topic. With the results of this pilot study, we intend to train the NLP models with human annotations to more precisely and accurately identify emotional language relevant to medical education, our dataset, and the chosen frameworks.

With training, AI tools detect sentiment in medical student course evaluations with an interrater reliability similar to human coders.

This work was presented at the Association of American Medical Colleges meeting, October 2024, Atlanta, Georgia.