I spent the last week testing GPT-4 specifically on law firm problems. Contract analysis. Due diligence. Legal research. Document review. The results were better than I expected.
Here's what actually works and what doesn't.
Contract Analysis: Strong Win
I gave GPT-4 a real (anonymized) consulting agreement and asked it to flag unusual or concerning clauses. It:
- Identified a unilateral termination clause that favored the client
- Flagged indemnification language that didn't match industry standard
- Caught a confidentiality clause that was overly broad
- Explained why each was potentially problematic
Would I rely on this alone? No. But as a starting point for a junior associate, or a second pair of eyes on something a partner drafted? Very useful.
Time saved: An associate who would normally spend 1-2 hours doing initial review now gets a summary in 30 seconds and can focus on verification and judgment calls.
Due Diligence Document Summaries: Good But Requires Verification
I gave GPT-4 a 15-page M&A document and asked for a summary of key points, risks, and outstanding items.
It produced a solid summary. Captured the main facts. Identified the material issues.
Problem: It missed one nuance about a carve-out that mattered contextually. Someone familiar with the deal would catch that, but a junior person might miss it.
This is useful as a time-saver (3 hours of reading vs. 30 minutes of review), but it requires someone smart on the deal to verify.
Legal Research: Solid
I asked GPT-4 to explain case law around a specific contract law question. It provided:
- Three relevant precedents (all real, which is new)
- How each applied to the question at hand
- How courts have recently been interpreting the relevant statute
- Clear limitations of the analysis (what you'd need to verify)
This is the kind of research someone would normally hand to a junior associate. GPT-4 does it in seconds.
Caveat: You still need to verify the case citations and make sure the analysis is jurisdictionally accurate. But it's a legitimate starting point.
Document Review at Scale: The Real Opportunity
This is where GPT-4 gets interesting for law firms.
I uploaded 10 contracts and asked GPT-4 to identify which ones had problematic indemnification clauses. It got 8 of 10 correct.
Two it flagged that didn't have issues (false positive). Two it should have flagged but didn't (false negative).
That 80% accuracy is not good enough to replace human review. But it's good enough to pre-screen. You could use this to say "review documents 1, 3, 5, 7 carefully and skip 2, 4, 6, 8, 9, 10."
That's a meaningful efficiency gain in large-scale document review.
The Bar Exam Question: Real but Overblown
GPT-4 did pass the bar exam, but there's nuance there. It passed because it understands legal concepts, not because it's ready to give legal advice.
On multiple-choice bar questions, it performed well. On essay questions (which require nuance and fact application), it did okay but a good lawyer would write better answers.
Don't use this as proof that GPT-4 can replace lawyers. Use it as proof that GPT-4 understands law well enough to be a useful research and analysis assistant.
What Doesn't Work
Judgment-heavy analysis: "Is this deal fair?" or "Should we settle this case?" GPT-4 can provide frameworks for thinking about it, but it can't substitute for judgment.
Jurisdiction-specific subtlety: "How would a California court interpret this clause in this context?" It can give you a framework, but you need someone who knows California law.
Confidential analysis: You can't paste client details and confidential information into GPT-4 (it goes to OpenAI's servers). So it's not useful for analysis that requires client-specific context.
The Practical Implementation for Law Firms
Here's where I'd start:
- Contract review: Use GPT-4 to generate an initial checklist of things to review. Have a junior associate verify the list. Then review only those items carefully.
- Legal research: Use GPT-4 to generate research summaries. Have the responsible partner verify the citations and analysis before relying on it.
- Document summaries: Use GPT-4 to create executive summaries of long documents. Have someone knowledgeable verify for accuracy.
- Initial due diligence: Use GPT-4 to screen documents and flag potential issues. Have experts review the flagged items.
In all of these cases, you're using GPT-4 to handle routine work so your smart people can focus on judgment. Not replacing people. Augmenting them.
The Timeline for Law Firms
GPT-4 is available now for Plus subscribers and API users. Adoption will accelerate as more lawyers test it.
Within 6 months, any law firm not experimenting with this will be behind. Within a year, it'll be table-stakes to have thought about how to use it.
The Bottom Line
GPT-4 is not a replacement for lawyers. It's an assistant that's actually useful for the things lawyers spend time on but don't love doing.
If you're a law firm, you should be running a pilot right now. Not because GPT-4 will change how you practice, but because your competitors are.