In January, OpenAI, the company behind ChatGPT, launched a version of their popular chatbot dedicated to helping people navigate health-related questions. The new tool invites users to upload their electronic medical charts and wellness app data to receive personalized health advice on demand.
But while the company vows to keep users’ data secure, people should know that once they turn over their records, they may lose control over what happens to them, warns I. Glenn Cohen ’03, the James A. Attwood and Leslie Williams Professor of Law at Harvard, in a recent paper written with Ifeoma Ajunwa and Ravi B. Parikh.
Around 25% of Americans admit turning to ChatGPT, Google Gemini, or another chatbot for medical advice. That’s no surprise, Cohen says, given that large language models are inexpensive, accessible — and increasingly, fairly accurate. But he adds that, unlike physicians, hospitals, and other traditional healthcare providers, these chatbots aren’t typically governed by the Health Insurance Portability and Accountability Act, commonly known as HIPAA.
“This is not the equivalent of asking your doctor something, or of getting a second opinion on a health problem you have from a clinician,” he says.
Cohen’s paper, “When Patients Share Everything With an AI Chatbot: Risks and Opportunities of Large Language Models,” was recently published in the Journal of the American Medical Association (JAMA). It argues that people who upload their full medical records to large language models should know that they face unique risks, such as privacy violations, poor-quality advice, and even discrimination.
In an interview with Harvard Law Today, Cohen shared more about the paper and what people stand to gain — and lose — from sharing health data with chatbots.
Harvard Law Today: What inspired you and your coauthors to write this paper?
Glenn Cohen: We were kind of surprised — we saw some of the large language models going strongly towards the health vertical. We have known that people were using chatbots in this way, but it’s different to see the companies encourage and invite this kind of use. And we thought that many people were doing it with their eyes closed to the legal and ethical issues that are being raised.
HLT: What are the major risks associated with uploading one’s medical records to a large language model like ChatGPT Health or another chatbot?
Cohen: If you think this is the equivalent of asking your doctor about something or getting a referral to another doctor for a second opinion, you should know that there are some systematic ways in which it’s very different. One is the protection of the privacy of the information. Unlike healthcare information that typically resides within an entity covered by HIPAA, such as a hospital system or doctor’s office, this use is not typically HIPAA protected. That’s because you’re the one taking information that might have been protected within the healthcare system, and you’re sending it to a third party that is not a covered entity under that statute. That means you may be putting yourself at the mercy of whatever privacy rules Claude or ChatGPT has — and remember, those terms of service can and do change over time. There have even been companies that have tried to make changes to their terms and claim the changes are retroactive, like what 23andMe did with genetic data.
HLT: Are there any laws besides HIPAA that could protect our health data once it’s shared with a chatbot?
Cohen: There could be pertinent laws pertaining to fraud, or contract law, perhaps, if there’s a contract. There may also be some state privacy laws that apply for U.S. persons, and other laws for those outside the U.S. But the main thing to emphasize is that most interactions between a user and a chatbot are not governed the way other healthcare interactions are. When your health record leaves the hospital system, it’s losing a lot of the protections that would otherwise apply under federal law.
HLT: Why might that be concerning?
Cohen: Your health information could be exposed — large language models could be subject to cyberattacks or to prompts that allow someone to identify an individual within a data set. Either way, that’s obviously a violation of privacy. Additionally, patients typically don’t have great anti-discrimination law protections at the federal level against discrimination on the basis of genetic information. Once exposed, your health information could be used in all sorts of ways.
HLT: How is sharing my chart with a large language model any different than what my doctor does when they send it to another medical professional?
Cohen: When information is transmitted to a specialist for an opinion, physicians don’t typically send the entire patient record. For example, say you have a patient with a new breast mass, and you are sending them to a surgical oncologist. You send imaging, pathology, pertinent history. You don’t send a decade-old psychiatric record. Instead, the doctor curates the record for their colleague. And the reason for doing that is to protect privacy, but also to help the other physician advance the ball by telling them what to focus on. When you upload an entire medical record to ChatGPT, it has no way of knowing what to focus on in this way.
Large language models could be subject to cyberattacks or to prompts that allow someone to identify an individual within a data set.
HLT: What other major concerns do you identify in the paper?
Cohen: As I mentioned, large language models don’t have experience and often lack the ability to contextualize information, and that can exacerbate existing health inequities. For example, given the way medical records are created in the United States, sometimes they’re documented in ways that might be stigmatizing to particular racial groups. Let’s say a Black patient shows up to the emergency room with sickle cell disease — which is a condition that causes agonizing pain — and the triage note talks about them being drug-seeking. An experienced clinician can hopefully understand the context and the underlying biases at work. A large language model is not designed with that sensitivity in mind, so there’s concern that disparities in charting procedures might themselves be carried forward and propagated further by large language models.
HLT: You’ve discussed the potential risks, but in your paper, you also talk about how there may be upsides to using large language models in this way.
Cohen: There definitely are possible benefits. There is a lot of frustration with the American healthcare system, for understandable reasons: It’s expensive, there are not enough primary care physicians, you may wait a long time for an appointment, your insurance may not cover something important. A large language model offers the possibility of having 24/7 access to medical information that could be personalized. I completely understand why patients are interested in thinking about this. I just want people to go into it with eyes open and also ask themselves whether uploading their entire medical record is the best strategy for what they need.
A large language model offers the possibility of having 24/7 access to medical information that could be personalized … I completely understand why patients are interested in thinking about this.
HLT: Since 2018, the National Institutes of Health has conducted the All of Us research program, which collects genetic and health information from hundreds of thousands of Americans, with the aim of finding causes and treatments for common and rare diseases. Couldn’t we benefit in a similar way from the vast quantities of information the chatbots are taking in?
Cohen: It’s an interesting comparison, but with large language models, we have privatization of control and privatization of benefits. By contrast, the All of Us program was created by the government, and is responsive to the public, its terms are transparent, and it has had lots of community engagement along the way. It has levels of access to the data that’s based on the sensitivity of data. The program has worked with bioethicists. Maybe all that could be true of a company like OpenAI — but maybe not. They’re just not responsive to the public in the way a government agency is. They’re not public actors, and they don’t have the obligation to share the benefits of what they learn with the public.
HLT: How accurate is the medical information coming out of some of the major chatbots, anyway?
Cohen: We don’t yet have any studies on exactly this scenario — where you upload your entire medical record and it spits out a response. But we do have studies showing that large language models are performing better and better. There’s a great new paper from researchers at Beth Israel here in Boston that showed really good performance by large language models on text-based ER tasks using real cases. So, I’m very enthusiastic about the use of large language models in medicine, but I tend to think we should consider whether this is an adjunct to the physician — that is, a tool for the physician — rather than a substitute for the physician. Indeed, one of the most quickly-adopted large language model-based programs in medicine is called Open Evidence, which seems to do a great job of allowing an individual clinician to query the literature and access up-to-date treatment guidelines and peer reviewed studies almost instantaneously while working with real patients.
Edited for length and clarity.
Want to stay up to date with Harvard Law Today? Sign up for our weekly newsletter.