A lesson in implementation: A pre-post study of providers' experience with artificial intelligence-based clinical decision support.

Background: To explore attitudes about artificial intelligence (AI) among staff who utilized AI-based clinical decision support (CDS).

Methods: A survey was designed to assess staff attitudes about AI-based CDS tools. The survey was anonymously and voluntarily completed by clinical staff in three primary care outpatient clinics before and after implementation of an AI-based CDS system aimed to improve glycemic control in patients with diabetes as part of a quality improvement project. The CDS identified patients at risk for poor glycemic control and generated intervention recommendations intended to reduce patients' risk.

Results: Staff completed 45 surveys pre-intervention and 38 post-intervention. Following implementation, staff felt that care was better coordinated (11 favorable responses, 14 unfavorable responses pre-intervention; 21 favorable responses, 3 unfavorable responses post-intervention; p < 0.01). However, only 14 % of users would recommend the AI-based CDS. Staff feedback revealed that the most favorable aspect of the CDS was that it promoted team dialog about patient needs (N = 14, 52 %), and the least favorable aspect was inadequacy of the interventions recommended by the CDS.

Conclusions: AI-based CDS tools that are perceived negatively by staff may reduce staff excitement about AI technology, and hands-on experience with AI may lead to more realistic expectations about the technology's capabilities. In our setting, although AI-based CDS prompted an interdisciplinary discussion about the needs of patients at high risk for poor glycemic control, the interventions recommended by the CDS were often perceived to be poorly tailored, inappropriate, or not useful. Developers should carefully consider tasks that are best performed by AI and those best performed by the patient's care team.