A lawyer uses ChatGPT to prepare a brief for court. He knows the chatbot invented a case to boost one of his key arguments — and he leaves the citation in. If the judge finds out, can the attorney invoke the Fifth Amendment to protect himself?
Amid a growing number of AI-generated filings containing mistakes and misrepresentations, courts across the country have begun to consider measures aimed at ensuring the accuracy of submitted briefs, motions, and other documents. Some of these proposed rules, such as one proposed in September by the U.S. Court of Appeals for the Tenth Circuit, would require that lawyers and pro se litigants certify that filings prepared by AI have been reviewed by a human for accuracy before submission.
The problem of fabricated court filings obviously predates the rise of artificial intelligence tools, says Duncan Levin, a lecturer on law at Harvard, criminal defense attorney, and former federal and state prosecutor. But Levin believes that new technology is poised to create novel questions about when lawyers can plead the Fifth, particularly if courts adopt additional certification requirements like those proposed by the Tenth Circuit.
Lawyers must already bear responsibility for the accuracy of what they submit to court, Levin notes. But such a rule would add a more specific representation about the review process: A filing could contain fabricated material, and the attorney could also have falsely represented that the required human review and independent verification had occurred, he says.
That additional certification could provide evidence of what the lawyer knew and did, particularly if the lawyer later gives a false account of the review, Levin says. He adds that misrepresentations of this kind typically lead to professional or procedural consequences. But in extreme cases involving intentional deception, the conduct could potentially expose the attorney to criminal liability. In an email interview with Harvard Law Today, Levin discussed what he sees as an emerging collision between judicial efforts to police lawyers’ use of artificial intelligence and the Fifth Amendment privilege against self-incrimination.
Harvard Law Today: There have been several high-profile situations now in which AI was used to prepare court filings and got things wrong or simply invented cases. But you’re concerned about something slightly different.
Duncan Levin: Yes, one emerging legal problem created by generative artificial intelligence may have little to do with hallucinated cases. It may arise instead when a judge asks a lawyer a simple question: “Counsel, did you use generative AI to prepare this filing?”
Most of the time, that question should be easy to answer. Using AI is not itself unlawful, and courts have every reason to insist that lawyers verify the authorities, quotations, and factual assertions they submit. But the question becomes more complicated when it is asked during an inquiry into suspected misconduct.
Imagine that a brief contains fictitious cases. The court has reason to believe the lawyer failed to conduct a required verification, made an inaccurate representation about the use of AI, or tried to conceal what happened. The judge begins asking the lawyer personally what he did, what he knew, and what he previously represented to the court. At that point, a very old constitutional protection may collide with a very new technology.
The lawyer may have a Fifth Amendment problem.
HLT: How does the Fifth Amendment apply here?
Levin: The privilege against self-incrimination is not limited to criminal defendants testifying at trial. It can apply whenever the government compels testimonial answers that could expose the speaker to criminal prosecution. The Supreme Court articulated the point in Hoffman v. United States (1951). The privilege protects not only answers that would themselves establish criminal liability, but also answers that could furnish a “link in the chain” of evidence needed to prosecute. At the same time, the danger cannot be imaginary. There must be reasonable cause to apprehend criminal exposure from answering. That distinction matters here. The court determines whether the privilege is properly invoked in light of the particular question and its context.
HLT: Does that mean that a lawyer can invoke the Fifth Amendment in any situation in which they are being questioned about how they prepared a filing?
Levin: No, not every lawyer questioned about AI gets to take the Fifth. Professional embarrassment is not enough. Neither is the possibility of malpractice liability. Nor does exposure to Rule 11 sanctions or bar discipline, standing alone, trigger the privilege.
As I said before, the Fifth Amendment protects against criminal self-incrimination. But that possibility is not difficult to imagine. A lawyer who simply failed to catch an AI hallucination may have been careless, negligent, or reckless in a way that can be sanctioned. That does not necessarily create criminal exposure. Even a knowingly false citation does not automatically establish a crime. The question is whether the surrounding conduct creates a reasonable risk of prosecution for a particular offense, such as criminal contempt, obstruction, or perjury where the applicable legal requirements are met.
Change the facts, however, and the analysis changes with them. Suppose there is evidence suggesting that a lawyer knowingly made a false statement, fabricated an explanation, concealed evidence, or engaged in some other independently criminal conduct. If the judge then asks questions whose truthful answers could furnish evidence in a criminal prosecution, the Fifth Amendment may come into play. The Constitution does not disappear simply because the questioning is coming from the bench rather than from a prosecutor.
HLT: Is there a Supreme Court case that illustrates how the Fifth Amendment protections could apply to attorneys in this situation?
Levin: In Spevack v. Klein (1967), a New York attorney invoked the Fifth Amendment during a disciplinary investigation and refused to testify or produce certain records. He was disbarred. The Supreme Court reversed. Although the justices divided over aspects of the case, five agreed on the essential point: The lawyer could not be disbarred for asserting the privilege against self-incrimination.
Spevack may feel like a case from another era. In the age of AI regulation, it suddenly looks highly relevant. Lawyers have special obligations. They are officers of the court, members of a regulated profession, and fiduciaries to their clients. But they do not surrender their personal constitutional rights when they enter a courtroom.
HLT: You’re saying that lawyers have long been able to plead the Fifth in situations where responding to the inquiry could expose them to criminal liability. So, what is unique today, as courts begin to adopt AI-specific rules?
Levin: Some judges now require lawyers to disclose the use of generative AI, certify that AI-generated material has been independently verified, or otherwise assume personal responsibility for the accuracy of a filing. Those requirements make sense. A court should be able to insist that lawyers check what they submit.
But a certification does not necessarily answer the later Fifth Amendment question. Suppose a lawyer signs a certification stating that AI-generated authorities have been independently verified. The court later discovers that several of the cited cases do not exist. The certification itself remains evidence of what the lawyer represented to the court. The Fifth Amendment does not ordinarily erase a preexisting filing merely because its contents later become incriminating.
But now imagine that the judge asks: “Did you actually perform the verification you certified?” The constitutional posture may now be different. The court is no longer merely relying on a prior representation. It is seeking new testimonial evidence about the lawyer’s knowledge and conduct at a time when those answers may help establish a crime. A prospective certification can create an obligation and memorialize a representation. It does not necessarily give the government a prospective right to compel incriminating testimony about whether the lawyer later violated it. The certification may become Exhibit A. It does not necessarily make the lawyer Witness A.
And there is another complication.
HLT: Tell me about that.
Levin: It is often said that no adverse inference may ever be drawn from a person’s invocation of the Fifth Amendment. That is too broad. In Baxter v. Palmigiano (1976), the Supreme Court permitted an adverse inference from an inmate’s silence in a prison disciplinary proceeding where other evidence supported the decision. Outside a criminal prosecution, invoking the privilege is therefore not always consequence-free. But Baxter does not answer the harder question: whether the government may impose a penalty simply because someone asserted a valid Fifth Amendment privilege.
That issue runs through another line of Supreme Court cases.
In Garrity v. New Jersey (1967), police officers were effectively told that they must answer potentially incriminating questions or lose their jobs. They answered, and those statements were later used against them in criminal prosecutions. The Supreme Court held that the statements could not constitutionally be used.
In Lefkowitz v. Cunningham (1977), the Court rejected a New York scheme that imposed substantial governmental penalties on political party officers who refused to waive immunity and testify. The core principle is straightforward even if its application can be difficult: Government cannot force a person to surrender the Fifth Amendment privilege by imposing penalties for asserting it. Potentially incriminating testimony can be compelled through a lawful immunity process that protects against both its use and the use of evidence derived from it in a criminal prosecution.
Spevack brings these principles directly into the legal profession.
Courts unquestionably have broad authority to regulate lawyers appearing before them. They can require candor. They can enforce filing rules. They can sanction misconduct. They can refer lawyers to disciplinary authorities. But it does not necessarily follow that they may compel potentially incriminating testimony from a lawyer personally by threatening professional punishment for invoking the Fifth.
HLT: So, how can courts balance the need for accuracy and honesty without violating an attorney’s constitutional rights?
Levin: A judge may be entitled to ask a question and still encounter a valid assertion of the privilege. A lawyer may have a constitutional right not to answer while remaining exposed to sanctions based on evidence the court independently possesses. And, depending on the type of proceeding, an adverse inference may sometimes be permissible.
The Supreme Court has not decided how these doctrines apply to judicial inquiries about generative AI. That is what makes the issue interesting. The technology is new. The constitutional architecture is not. The answer, however, should not be less accountability.
HLT: Why do courts need to have rules about AI at all? Why don’t the old rules suffice?
Levin: Existing rules already do much of this work. AI-specific requirements may clarify verification duties, but courts should first ask what they add to obligations lawyers already have. Courts need effective ways to police fictitious cases, false quotations, inaccurate certifications, and reckless filings. Generative AI can produce highly persuasive nonsense, and lawyers who rely on it without verification can create serious problems for courts and clients alike. The better question is how to design accountability without unnecessarily colliding with constitutional protections.
HLT: How might courts do that?
Levin: One answer is to regulate the filing, not the confession. Courts should focus first on accuracy and verification rather than on the particular technology used to create a brief. The central professional obligation should not be “do not use AI.” It should be “do not submit legal or factual propositions that you have not adequately verified.”
A fictitious citation generated by ChatGPT is dangerous because it is fictitious, not because a machine created it. A false citation supplied by a careless associate, copied from another brief, or invented by the lawyer presents essentially the same institutional problem. Technology-specific rules may therefore miss the more basic point.
Where courts use certifications, they should generally be objective and prospective. Lawyers can be required to certify that cited authorities exist, quotations are accurate, and factual assertions satisfy the applicable standards for evidentiary support. That creates a clear standard and a contemporaneous record without requiring lawyers to provide a technological autobiography explaining how every sentence in a filing came into existence.
HLT: What could courts do if they suspect that something has been fabricated in a document?
Levin: When misconduct is suspected, courts should begin with evidence rather than interrogation. The filing itself may establish much of what matters. So may earlier representations, docket records, electronic evidence obtained through proper means, testimony from other participants, or other nonprivileged sources. The Fifth Amendment does not prevent a court from investigating wrongdoing. It may, however, limit the court’s ability to compel the suspected wrongdoer to provide testimonial evidence against himself.
Courts should also distinguish between correcting a defective filing and establishing a lawyer’s personal culpability. A court can strike a filing, require a corrected submission, protect the client from prejudice, demand additional verification, impose sanctions supported by the record, and make an appropriate disciplinary referral.
None of those measures necessarily require compelling the lawyer to explain potentially criminal conduct from the counsel table. And when a lawyer has a legitimate basis to invoke the Fifth Amendment, the invocation should be treated as what it is: the exercise of a constitutional privilege, not itself a confession of professional wrongdoing.
That does not mean misconduct disappears. It does not prevent courts from acting on independent evidence. Nor does it mean that every adverse consequence following an invocation is unconstitutional. It means only that accountability should not depend on compelling a lawyer to become a witness against himself.
There is a practical symmetry here familiar to lawyers who have worked on both sides of criminal cases: the government’s legitimate interest in obtaining an answer does not, by itself, eliminate the privilege against giving one.
HLT: Is there a deeper issue here, without even taking AI into account?
Levin: Yes, lawyers occupy an unusual position in court. They are advocates, officers of the court, members of a regulated profession, and participants in the administration of justice. Judges ask lawyers questions constantly, and the system depends on lawyers answering candidly.
But the lawyer is also an individual. Those roles usually coexist without difficulty. They become harder to reconcile when the lawyer himself becomes the object of the court’s inquiry. One moment, a judge may be asking counsel to explain an argument made on behalf of a client. The next, the judge may be asking the same person to explain his own potentially criminal conduct.
The physical setting has not changed. The constitutional posture may have. A lawyer standing at counsel table does not cease to possess constitutional rights because the judge has turned the questioning toward him. Recognizing that principle is not an argument for protecting bad lawyering. Courts should aggressively police fabricated authorities, false representations, and reckless submissions. AI makes those obligations more urgent, not less.
But accountability and constitutional protection are not opposites. As courts respond to a 21st century technology by creating increasingly detailed AI requirements and conducting increasingly searching inquiries into lawyers’ use of that technology, they may eventually confront one of the oldest questions in American constitutional law: When may the government compel a person to provide evidence against himself?
Someday soon, a judge confronting a suspicious AI-generated filing may ask a lawyer a seemingly straightforward question: “Counsel, did you know this was false when you filed it?”
And, depending on the circumstances, one legally permissible answer may be the one the judge least expects: “Your Honor, I respectfully invoke my Fifth Amendment privilege.”
Want to stay up to date with Harvard Law Today? Sign up for our weekly newsletter.