Vulnerability of Large Language Models to Prompt Injection When Providing Medical Advice

Can commercial medical large language models (LLMs) be manipulated through prompt-injection attacks (ie, maliciously crafted inputs that manipulate an LLM’s behavior) to recommend unsafe or contraindicated treatments?In this quality improvement study using a controlled simulation of 216 patient-LLM dialogues, webhook-simulated prompt-injection attacks succeeded in 94.4% of trials and 91.7% of extremely high-harm scenarios, including US Food and Drug Administration Category X pregnancy drugs such as thalidomide.These findings suggest that current LLM safeguards remain inadequate to prevent prompt-injection manipulation that could induce life-threatening clinical recommendations.

Corresponding Author: Jungyo Suh, MD, Department of Urology, University of Ulsan College of Medicine, Asan Medical Center, 88, Olympic-ro 43-gil, Songpa-gu, 05505, Seoul, Republic of Korea (uro_jun@amc.seoul.kr).

Link to the article in your story

We encourage you to link out to this article in your story using the link below. It includes an access token that will give free access to the article for your readers up to one year after publication. (The link will be live after the article publishes and embargo is lifted.)

Please see the article for additional information, including full author list, author contributions and affiliations, conflict of interest and financial disclosures, and funding and support.

Need more information? Contact us.