Context Matching is not Reasoning: Assessing Generalized Evaluation of Generative Language Models in Clinical Settings.

Current discussion surrounding the clinical capabilities of generative language models (GLMs) predominantly center around multiple-choice question-answer (MCQA) benchmarks derived from clinical licensing examinations. While accepted for human examinees, characteristics unique to GLMs bring into question the validity of such benchmarks. Here, we validate four benchmarks using eight GLMs, ablating for parameter size and reasoning capabilities, validating via prompt permutation three key assumptions that underpin the generalizability of MCQA-based assessments: that knowledge is applied, not memorized, that semantic consistency will lead to consistent answers, and that situations with no answers can be recognized.

Gated recurrent unit with decay has real-time capability for postoperative ileus surveillance and offers cross-hospital transferability.

Background: Ileus, a postoperative complication after colorectal surgery, increases morbidity, costs, and hospital stays. Assessing risk of ileus is crucial, especially with the trend towards early discharge. Prior studies assessed risk of ileus with regression models, the role of deep learning remains unexplored.

Association of Covert Cerebrovascular Disease With Falls Requiring Medical Attention.

Background: The impact of covert cerebrovascular disease on falls in the general population is not well-known. Here, we determine the time to a first fall following incidentally detected covert cerebrovascular disease during a clinical neuroimaging episode.

Methods: This longitudinal cohort study assessed computed tomography (CT) and magnetic resonance imaging from 2009 to 2019 of patients aged >50 years registered with Kaiser Permanente Southern California which is a healthcare organization combining health plan coverage with coordinated medical services, excluding those with before stroke/dementia.

Investigating the Impact of Social Determinants of Health on Diagnostic Delays and Access to Antifibrotic Treatment in Idiopathic Pulmonary Fibrosis.

Idiopathic pulmonary fibrosis (IPF) is a rare disease that is challenging to diagnose. Patients with IPF often spend years awaiting a diagnosis after the onset of initial respiratory symptoms, and only a small percentage receive antifibrotic treatment. In this study, we examine the associations between social determinants of health (SDoH) and two critical factors: time to IPF diagnosis following the onset of initial respiratory symptoms, and whether the patient receives antifibrotic treatment.

The Use of Large Language Models to Accelerate Literature Review Towards Digital Health Equity and Inclusiveness.

Digital health technologies (DHTs) have revolutionized clinical trials, offering unprecedented opportunities to streamline processes, enhance patient engagement, and improve data quality. Growing technology device and broadband access are contributing to the increasing number of DHT-enabled trials. Ideally, DHTs have the potential to make clinical research more inclusive and diverse.

Social determinants of health extraction from clinical notes across institutions using large language models.

Detailed social determinants of health (SDoH) is often buried within clinical text in EHRs. Most current NLP efforts for SDoH have limitations, investigating limited factors, deriving data from a single institution, using specific patient cohorts/note types, with reduced focus on generalizability. We aim to address these issues by creating cross-institutional corpora and developing and evaluating the generalizability of classification models, including large language models (LLMs), for detecting SDoH factors using data from four institutions.

Application of Claims-based Frailty Index to a Structured Electronic Health Record Data.

Background: The Claims-based Frailty Index (CFI) has been developed and validated using Medicare claims data. However, whether CFI can be applied to structured electronic health record data has not been studied.

Methods: We applied the CFI to a structured electronic health record dataset (Explorys dataset) and a Medicare fee-for-service 5% sample data and compared the prevalence of frailty from each dataset, using the cohort of older adults.

A scoping review of OMOP CDM adoption for cancer research using real world data.

The Observational Medical Outcomes Partnership (OMOP) common data model (CDM) supports large-scale research by enabling distributed network analyses. However, the breadth of its adoption in cancer research is not well understood. We conducted a scoping review to describe the adoption of the OMOP CDM in cancer research.

Development and Validation of Natural Language Processing Algorithms in the ENACT National Electronic Health Record Research Network.

Electronic health record (EHR) data are a rich and invaluable source of real-world clinical information, enabling detailed insights into patient populations, treatment outcomes, and healthcare practices. The availability of large volumes of EHR data are critical for advancing translational research and developing innovative technologies such as artificial intelligence. The Evolve to Next-Gen Accrual to Clinical Trials (ENACT) network, established in 2015 with funding from the National Center for Advancing Translational Sciences (NCATS), aims to accelerate translational research by democratizing access to EHR data for all Clinical and Translational Science Awards (CTSA) hub investigators.

A Comparison of Patient Self-Reported Fatigue in the FRAIL Scale With a Validated Fatigue Measure.

Rationale: The FRAIL scale is a self-administered tool used to screen for frailty in clinical, community and long-term nursing settings. Patient's self-reporting of fatigue in the FRAIL scale may raise concerns of subjectivity and accuracy in frailty assessment.

Objective: To assess the performance of the patient-reported fatigue measure in the FRAIL scale in comparison to a validated fatigue measure, the Fatigue Severity Scale (FSS).

The ENACT network is acting on housing instability and the unhoused using the open health natural language processing toolkit.

Housing is an environmental social determinant of health that is linked to mortality and clinical outcomes. We developed a lexicon of housing-related concepts and rule-based natural language processing methods for identifying these housing-related concepts within clinical text. We piloted our methods on several test cohorts: a synthetic cohort generated by ChatGPT for initial infrastructure testing, a cohort with substance use disorders (SUD), and a cohort diagnosed with problems related to housing and economic circumstances (HEC).

Characterizing the progression from mild cognitive impairment to dementia: a network analysis of longitudinal clinical visits.

Background: With the recent surge in the utilization of electronic health records for cognitive decline, the research community has turned its attention to conducting fine-grained analyses of dementia onset using advanced techniques. Previous works have mostly focused on machine learning-based prediction of dementia, lacking the analysis of dementia progression and its associations with risk factors over time. The black box nature of machine learning models has also raised concerns regarding their uncertainty and safety in decision making, particularly in sensitive domains like healthcare.

A framework for human evaluation of large language models in healthcare derived from literature review.

With generative artificial intelligence (GenAI), particularly large language models (LLMs), continuing to make inroads in healthcare, assessing LLMs with human evaluations is essential to assuring safety and effectiveness. This study reviews existing literature on human evaluation methodologies for LLMs in healthcare across various medical specialties and addresses factors such as evaluation dimensions, sample types and sizes, selection, and recruitment of evaluators, frameworks and metrics, evaluation process, and statistical analysis type. Our literature review of 142 studies shows gaps in reliability, generalizability, and applicability of current human evaluation practices.

A Case Demonstration of the Open Health Natural Language Processing Toolkit From the National COVID-19 Cohort Collaborative and the Researching COVID to Enhance Recovery Programs for a Natural Language Processing System for COVID-19 or Postacute Sequelae of SARS CoV-2 Infection: Algorithm Development and Validation.

Background: A wealth of clinically relevant information is only obtainable within unstructured clinical narratives, leading to great interest in clinical natural language processing (NLP). While a multitude of approaches to NLP exist, current algorithm development approaches have limitations that can slow the development process. These limitations are exacerbated when the task is emergent, as is the case currently for NLP extraction of signs and symptoms of COVID-19 and postacute sequelae of SARS-CoV-2 infection (PASC).

Adoption of the OMOP CDM for Cancer Research using Real-world Data: Current Status and Opportunities.

Background: The Observational Medical Outcomes Partnership (OMOP) common data model (CDM) that is developed and maintained by the Observational Health Data Sciences and Informatics (OHDSI) community supports large scale cancer research by enabling distributed network analysis. As the number of studies using the OMOP CDM for cancer research increases, there is a growing need for an overview of the scope of cancer research that relies on the OMOP CDM ecosystem.

Objectives: In this study, we present a comprehensive review of the adoption of the OMOP CDM for cancer research and offer some insights on opportunities in leveraging the OMOP CDM ecosystem for advancing cancer research.

Considerations for Quality Control Monitoring of Machine Learning Models in Clinical Practice.

Integrating machine learning (ML) models into clinical practice presents a challenge of maintaining their efficacy over time. While existing literature offers valuable strategies for detecting declining model performance, there is a need to document the broader challenges and solutions associated with the real-world development and integration of model monitoring solutions. This work details the development and use of a platform for monitoring the performance of a production-level ML model operating in Mayo Clinic.

Stratifying heart failure patients with graph neural network and transformer using Electronic Health Records to optimize drug response prediction.

Objectives: Heart failure (HF) impacts millions of patients worldwide, yet the variability in treatment responses remains a major challenge for healthcare professionals. The current treatment strategies, largely derived from population based evidence, often fail to consider the unique characteristics of individual patients, resulting in suboptimal outcomes. This study aims to develop computational models that are patient-specific in predicting treatment outcomes, by utilizing a large Electronic Health Records (EHR) database.

Large Language Models for Social Determinants of Health Information Extraction from Clinical Notes - A Generalizable Approach across Institutions.

The consistent and persuasive evidence illustrating the influence of social determinants on health has prompted a growing realization throughout the health care sector that enhancing health and health equity will likely depend, at least to some extent, on addressing detrimental social determinants. However, detailed social determinants of health (SDoH) information is often buried within clinical narrative text in electronic health records (EHRs), necessitating natural language processing (NLP) methods to automatically extract these details. Most current NLP efforts for SDoH extraction have been limited, investigating on limited types of SDoH elements, deriving data from a single institution, focusing on specific patient cohorts or note types, with reduced focus on generalizability.

A taxonomy for advancing systematic error analysis in multi-site electronic health record-based clinical concept extraction.

Background: Error analysis plays a crucial role in clinical concept extraction, a fundamental subtask within clinical natural language processing (NLP). The process typically involves a manual review of error types, such as contextual and linguistic factors contributing to their occurrence, and the identification of underlying causes to refine the NLP model and improve its performance. Conducting error analysis can be complex, requiring a combination of NLP expertise and domain-specific knowledge.

Revolutionizing Postoperative Ileus Monitoring: Exploring GRU-D's Real-Time Capabilities and Cross-Hospital Transferability.

Background: Postoperative ileus (POI) after colorectal surgery leads to increased morbidity, costs, and hospital stays. Identifying POI risk for early intervention is important for improving surgical outcomes especially given the increasing trend towards early discharge after surgery. While existing studies have assessed POI risk with regression models, the role of deep learning's remains unexplored.

Assessment of Gender Differences in Letters of Recommendation for Physical Therapy Residency Applications.

Introduction: Letters of recommendation (LOR) are an integral component of physical therapy residency applications. Identifying the influence of applicant and writer gender in LOR will help identify whether potential implicit gender bias exists in physical therapy residency application processes.

Review Of Literature: Several medical and surgical residency education programs have reported positive, neutral, or negative LOR female gender bias among applicants and writers.