An Open-architecture AI Model for CPT Coding in Breast Surgery: Development, Validation, and Prospective Testing.

Objective: To develop, validate, and prospectively test an open-architecture, transformer-based artificial Intelligence (AI) model to extract procedure codes from free-text breast surgery operative notes.

Background: Operative note coding is time-intensive and error-prone, leading to lost revenue and compliance risks. Although AI offers potential solutions, adoption has been limited due to proprietary, closed-source systems lacking transparency and standardized validation.

Methods: We included all institutional breast surgery operative notes from July 2017 to December 2023. Expert medical coders manually reviewed and validated surgeon-assigned Current Procedural Terminology (CPT) codes, establishing a reference standard. We developed and validated an AI model to predict CPT codes from operative notes using 2 versions of the pretrained GatorTron clinical language model: a compact 345 million-parameter model and a larger 3.9 billion-parameter model, each fine-tuned on our labeled data-set. Performance was evaluated using the area under the precision-recall curve (AUPRC). Prospective testing was conducted on operative notes from May to October 2024.

Results: Our data set included 3259 operative notes with 8036 CPT codes. Surgeon coding discrepancies were present in 12% of cases (overcoding: 8%, undercoding: 10%). The AI model showed strong alignment with the reference standard [compact version AUPRC: 0.976 (0.970, 0.983), large version AUPRC: 0.981 (0.977, 0.986)] on cross-validation, outperforming surgeons (AUPRC: 0.937). Prospective testing on 268 notes confirmed strong real-world performance.

Conclusions: Our open-architecture AI model demonstrated high performance in automating CPT code extraction, offering a scalable and transparent solution to improve surgical coding efficiency. Future work will assess whether AI can surpass human coders in accuracy and reliability.