Could LLMs Capture What Conventional Pain Scores Miss?

Large language models (LLMs) may help quantify the sensory, emotional, and psychological dimensions of chronic pain by analyzing patient-provided text, according to a study published October 6 in npj Digital Medicine.

“Current pain assessments, which generally ask patients to rate their pain on a scale of 0 to 10, fail to capture critical information such as whether the pain interferes with other functional activities, coexisting issues such as depression, and the extent to which treatments are helping,” said Ira Hofer, MD, an anesthesiologist and clinical informaticist at the Icahn School of Medicine at Mount Sinai, New York City, who was not involved with this research. “Additionally, these scores are consistently subjective and vary from person to person,” he noted.

“In contrast, an LLM-based [assessment] would enable the consistent analysis and categorization of a much larger set of information, reliably capturing not just severity of pain but its overall effects on a patient’s life,” Dr. Hofer said.

LLM-Based Analysis of Patient Pain Narratives

In this study, LLMs were used to analyze transcripts of semi-structured patient interviews and generate scores across nine domains:

• physical pain

• emotional pain

• quality of life

• catastrophizing

• depression

• anxiety

• rumination

• agency deficit

• narrative fragmentation

The cross-sectional observational study included 131 participants, 67 with chronic low back pain, 33 with major depressive disorder (MDD), and 31 pain-free healthy controls. Participants were older than 40 years, and data were collected between December 2022 and June 2025. Semi-structured interviews were conducted via Zoom using open-ended questions to elicit patient-centered pain narratives.

OpenAI’s Whisper base model was used to transcribe the interview recordings. The resulting transcripts were analyzed using Meta’s Llama-3-405B-Instruct model. They were then accessed through IBM Watson’s application programming interface and run on IBM servers. The model was prompted to act as a clinical evaluator, assigning scores from 0 to 10, with higher scores indicating worse outcomes.

To assess whether these scores reflected established clinical measures, researchers compared them with patient-reported scores from numeric rating scales (NRS), visual analog scales (VAS), and validated questionnaires.

LLM Scores Identify Distinct Clinical Patterns

Patients with chronic low back pain reported the highest physical pain scores, while those with MDD had higher scores for impaired quality of life, emotional pain, depression, and agency deficit. The LLM-derived measures were also compared with interview word count. The LLM scores clearly separated the three groups, while word counting showed significant overlap.

Spearman’s rho correlations were calculated between LLM-generated metrics and patient-reported clinical scores to assess construct validity. Overall, 85 of 117 correlations (73%) were reported to be statistically significant after correction.

To determine whether LLM-derived assessments could distinguish between chronic low back pain, MDD, and healthy controls, comparison of four machine learning algorithms (Random Forest, Logistic Regression, Support Vector Machine, and Naive Bayes) took place. Model performance was assessed using accuracy, balanced accuracy, and macro-averaged F1 scores. (The F1 score measures the difference between precision and recall, with higher values indicating better classification performance.)

A logistic regression model combining agency deficit and rumination number had an F1 score of 0.851 (very high discrimination). A model combining agency deficit and narrative fragmentation had an F1 score of 0.71 (high discrimination). These results suggest that additional psychological constructs derived from patient narratives can help distinguish groups even when they have overlapping symptoms.

Finally, analysis of Reddit posts also found similar clinical patterns: Depression-related posts had higher psychological scores, while chronic pain posts had higher physical pain scores. However, scores were higher in online communities, suggesting that writing style, context, and unverified diagnoses influence LLM-generated assessments.

Considerations for Using LLMs to Guide Pain Care

“Large language models do well when trained on a large volume of well-labeled text,” said Dr. Hofer. “Unfortunately, these do not exist today. Few large, diverse datasets contain narrative descriptions of a patient’s pain, and how a patient describes their pain likely varies based on factors such as their primary language, level of education, and emotional state. Thus, while in theory an LLM can create a much more robust and standardized way of measuring a patient’s pain, the underlying data for training are currently missing,” he told MedCentral.

“If an LLM can reliably flag patterns suggesting catastrophizing, social isolation, or poor perceived control, it could help triage patients toward the right components of a multimodal plan,” said Rohan Jotwani, MD, anesthesiologist and pain management specialist at Och Spine at New York-Presbyterian, New York City. “But I worry about a subtler problem as patients may begin unconsciously reshaping how they understand their pain to match what they sense the system expects.”

Dr. Jotwani, who was not part of this study, also added that if clinicians routinely defer to LLM narrative summaries, “we risk eroding the deep listening and interpretive skills that create what is known as the ‘therapeutic alliance.’ We know from the literature that physician empathy directly influences chronic pain outcomes, and that relationship cannot be outsourced without cost. If used to guide treatment decisions, these tools function as clinical decision support and should be evaluated with the same rigor we would demand of any intervention we bring into the room.”