Medical AI systems could be vulnerable to adversarial attacks

A new paper published in Science by a team of researchers from Harvard University and MIT suggests that medical artificial intelligence systems could be vulnerable to adversarial attacks.

The paper was co-authored by Harvard Law School Professor Jonathan Zittrain ’95, faculty director of the Berkman Klein Center for Internet & Society at Harvard University; John Bowers, a researcher at Berkman Klein Center; Samuel Finlayson, an M.D. Ph.D. candidate at Harvard and MIT; Isaac Kohane M.D. Ph.D., chair of the department of Biomedical Informatics at Harvard Medical School; Andrew L. Beam Ph.D., an instructor in the department of Biomedical Informatics at Harvard Medical School; and Joichi Ito, director of the MIT Media Lab.

In their article, “Adversarial attacks on medical machine learning,” the authors note that medical machine-learning systems may be uniquely susceptible to such attacks because of the specific systems and incentives at play in the medical industry. With competing interests within health care and billions of dollars at stake, various players in the health care system could be motivated to influence the system in “subtle, impactful, and sometimes ethically ambiguous ways,” making medical AI systems a likely ground zero for the emergence of adversarial attacks.

“These vulnerabilities allow a small, carefully designed change in how inputs are presented to a system to completely alter its output, causing it to confidently arrive at manifestly wrong conclusions,” they write.

To address these emerging concerns, the researchers call for an interdisciplinary approach to policy-making—one that brings together specialists in health, law, ethics and technology to address the problem before it becomes prevalent.

In 2017, the Berkman Klein Center for Internet and Society at Harvard University and the MIT Media Lab launched the Ethics and Governance of Artificial Intelligence Initiative aimed at bolstering the use of AI for the public good. Zittrain, whose research interests include the ethics and governance of artificial intelligence, recently developed and co-taught a new course with Ito on the subject.  This semester, Zittrain and Ito are co-teaching a seminar “Applied Ethical and Governance Challenges in AI.”