T he promise of AI agents has led to claims of imminent and rapid adoption ac ross fields. Companies have even promised to build AI agents that can autom ate all legal and scientific tasks. At the same time\, there are concerns a bout their misuse leading to catastrophic risks such as bio and cybersecuri ty risks. In this talk\, we will go over three case studies to foreground t he importance of evidence-based AI analysis. First\, while AI has been clai med to automate all of science\, existing adoption has been plagued by seve re reproducibility failures that lead to overoptimistic results across doze ns of fields. Recent empirical work shows that current models fall well sho rt of accomplishing far simpler tasks\, such as reproducing a paper’s resul ts even when the code and data are provided. Second\, for legal application s\, tasks that would lead to the most significant changes to the legal prof ession are also the ones most prone to overoptimism about AI capabilities\, as they are harder to evaluate. Third\, for analyzing safety risks\, it is important to analyze the marginal risk of AI over and above existing techn ology to evaluate the effectiveness of policy interventions. We conclude wi th a discussion of how to effectively conduct empirical analysis of AI.
Sayash Kapoor is a computer science Ph.D. candidate at P rinceton University's Center for Information Technology Policy and a co-aut hor of AI Snake Oil. His research focuses on the societal impact of AI. He is a recipient of a best paper award at ACM FAccT\, an impact recognition a ward at ACM CSCW\, and was included in TIME's inaugural list of the 100 mos t influential people in AI.
CATEGORIES:Speaker/Panel LOCATION:Lewis Hall\, 5th floor at the Berkman Klein Center's Multi-Purpose Room 515 GEO:0.000000;0.000000 ORGANIZER;CN="Jessica Weaver":MAILTO:jweaver@law.harvard.edu URL;VALUE=URI:https://hls.harvard.edu/events/ais-impact-on-science-law-and- society/ ATTACH;FMTTYPE=image/png:https://hls.harvard.edu/wp-content/uploads/2025/03/SayashKapoor_1x1.png END:VEVENT BEGIN:VTIMEZONE TZID:America/New_York BEGIN:DAYLIGHT TZOFFSETFROM:-0500 TZOFFSETTO:-0400 DTSTART:20250309T070000 TZNAME:EDT END:DAYLIGHT END:VTIMEZONE END:VCALENDAR