For millions of people, an everyday legal dispute never justifies the cost of a lawyer or a private mediator, no matter how much the outcome matters to them. Judge Victoria Wood saw that problem over and over during her years on the Napa County Superior Court bench. This week, Wood joins Judicaid Chief Strategy Officer Valerie Clemen to explain how those years led to Judicaid, an AI-assisted mediation platform built to help people resolve everyday disputes before time, expense, and emotion push them deeper into litigation.

Wood traces Judicaid’s origins to two problems she kept running into as a judge and mediator. Traditional settlement conferences arrive late in a case, after the parties have spent real money and dug into their positions. Then there are the lower-value landlord-tenant, contractor, neighbor, probate, and small-claims disputes where professional mediation rarely pencils out at all. Wood puts a number on the access problem. The State Bar of California’s 2024 Justice Gap Study, released in 2025, found that Californians received no legal help, or inadequate help, for 85 percent of their civil legal problems. Judicaid aims at a slice of that gap by giving people an earlier and cheaper chance to communicate and negotiate.

Clemen walks Greg Lambert and Marlene Gebauer through Judicaid’s “shuttle-style” mediation process. Each participant talks privately with an AI mediator named Jude, so the two sides never have to speak to each other directly. Jude gathers each side’s account of the dispute, identifies priorities and possible settlement terms, and moves between the participants while filtering out insults, anger, and inflammatory language. If the parties find common ground, the platform prepares a proposed settlement for review and electronic signature. Wood is careful about one boundary throughout the conversation. Jude is a facilitative mediator, and it stays away from evaluating legal rights. It will not decide who is legally correct, predict who will win, or give legal advice.

The conversation then turns to Judicaid’s pilot with Napa County Superior Court, and the role courts could play in expanding AI-assisted dispute resolution. In the court model, a court subscribes to the service and hands litigants’ access through a QR code, with no integration into the court’s technology systems. The pilot has already surfaced a behavioral lesson. Offering mediation as an optional service does not mean parties will use it, and Wood and Clemen see more potential where courts actively encourage or require litigants to attempt dispute resolution before proceeding. Language is another piece of the story, since Judicaid lets participants who speak different languages work through the same mediation without arranging multiple interpreters.

Wood and Clemen also see mediation as just the starting point. Wood uses the phrase “intelligent dispute resolution” for a broader category of AI-assisted tools covering mediator proposals, parent coordination, and other structured approaches to conflict. The bigger ambition is a change in habits, where people reach for structured communication and settlement before a disagreement hardens into a lawsuit. Clemen boils that aspiration down to three words she hopes become part of the vocabulary of everyday disputes: “Just Judicate it.”

Listen on mobile platforms: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Apple Podcasts⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ | ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Spotify⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ | ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠YouTube⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ | ⁠⁠Substack⁠⁠

[Special Thanks to ⁠⁠⁠Legal Technology Hub⁠⁠⁠ for their sponsoring this episode.]

Email: geekinreviewpodcast@gmail.com

Music: ⁠Jerry David DeCicca⁠

LINKS

• ⁠Judicaid⁠

• ⁠State Bar of California, California Justice Gap Study⁠

• ⁠2024 California Justice Gap Study⁠

• ⁠Napa County Superior Court, Small Claims⁠

Transcript

Marlene Gebauer (00:00)
Hi, I’m Marlene Gebauer from The Geek in Review, and this week we’re talking with Judge Victoria Wood and Chief Strategy Officer Valerie Clemen from Judicaid. But first up, here’s a word from our great sponsor and partners at Legal Technology Hub.

Marlene Gebauer (00:13)
I have Sam Moore here from Legal Technology Hub. He’s going to tell us about AI governance from the advisory side.

Sam Moore (00:19)
Thank you. Good to see you again. Well, in the advisory team at Legal Tech Hub, we’ve been having a lot of conversations with clients about AI governance. Now, most law firms and departments have at least an AI policy of some kind, and there are some broad standards starting to emerge. A trend we’re seeing at the moment is a shift away from policies written in 2023 and 2024, which largely stated what you couldn’t do, and toward a more informative approach, which

Marlene Gebauer (00:37)
Okay.

Sam Moore (00:47)
defines different risk categories and

Marlene Gebauer (00:48)
You...

Sam Moore (00:49)
makes distinctions between routine, low-risk use of AI and other more substantial use cases. I think that’s appropriate for where we are, and it’s also what clients are coming to expect from their advisors. But one of the bigger challenges we’re seeing right now is how you express an AI use policy in such a way that any member of your legal team could explain it to a client. I think that’s still a big challenge for the industry.

Marlene Gebauer (00:55)
So, okay.

Sam Moore (01:17)
And I don’t think clients are all that happy if their attorney says, “I’ll have to get back to you about that,” because the attorney is probably using AI day-to-day. So they should be able to explain the policy themselves. How else do you know they’re complying with it? We’re doing some interesting work at the moment helping law firms and law departments take those first-version AI use policies that are very much “thou shalt not”

Marlene Gebauer (01:18)
So...

Sam Moore (01:43)
and turn them into more readable, appropriate governance positions that inform the conversation with clients. If anyone wants to find out more about our advisory services, they can find me or Cheryl Wilson Griffin on LinkedIn, or they can visit

https://legaltechnologyhub.com

.

Marlene Gebauer (02:01)
Thank you, Sam. I mean, it is really important to have AI policy in plain language so people understand it.

Sam Moore (02:07)
Absolutely.

Marlene Gebauer (02:15)
Welcome to The Geek in Review, the podcast focused on innovative and creative ideas in the legal industry. I’m Marlene Gebauer.

Greg Lambert (02:22)
And I’m Greg Lambert.

Marlene Gebauer (02:24)
Today we’re exploring whether artificial intelligence can help address one of the justice system’s most persistent problems, resolving everyday disputes without months of delay, escalating costs, or procedures that ordinary people struggle to navigate.

Greg Lambert (02:39)
Yeah, and to help us do that, our guests today are the Honorable Victoria Wood, a retired judge from Napa County Superior Court and founder and CEO of Judicaid. Joining Judge Wood is Valerie Clemen, Judicaid’s Chief Strategy Officer. Judicaid is an online, AI-assisted mediation platform designed to help people resolve disputes privately, affordably, and without first filing a case.

Marlene Gebauer (03:08)
Judge Wood and Valerie, welcome to The Geek in Review.

Valerie Clemen (03:12)
Thank you for having us.

Vickie Wood (03:12)
Thank you. Happy to be here.

Marlene Gebauer (03:16)
Judge Wood, Judicaid grew out of what you witnessed during your years on the bench in Napa County. What recurring problems convinced you that the traditional court and mediation systems were not adequately serving people with everyday disputes, and that technology could provide a better path?

Vickie Wood (03:34)
There were a few things. One of them, I’d say, is that I handled a lot of settlement conferences while I was sitting on the trial court bench, but usually those come very near trial, later in the case, when people are very entrenched.

Marlene Gebauer (03:50)
When the judge says, “You go in the other room and figure it out.”

Vickie Wood (03:54)
Yeah, yeah.

And they’ve already become very angered in their positions. They’ve already invested so much emotionally and financially, usually through attorneys’ fees, and it’s hard to get those settled. Sometimes that’s what it takes because people understand the dynamics and the finances, and they’re finally at their wits’ end and they’ll settle. But it comes at a cost. There may be families that could have had relationships saved if they could have talked earlier on and understood each other’s positions better.

So that’s one thing, the opportunity for earlier mediation or communication facilitation. For example, I handle a fair amount of probate disputes, and some people say probate disputes are essentially family law after someone dies. They’re highly emotional for lots of reasons that go back many years. But what very often happens is it seems like there’s a lot of money at stake, but by the time they go through the process and get to mediation, sometimes the entire amount is spent on fighting. And it’s really sad. People never thought that would be the case.

I often think to myself, boy, if I only could have been with these people earlier in the process and let them know what would likely happen if they continued down the litigation path. So that’s one of the things that inspired it. The other thing is that there is this broad swath of disputes that are lower value and really don’t justify the expense of resources for attorney representation or human mediators. But those disputes still mean a lot to people, even though they’re low-dollar-value. In fact, that’s the reason why I decided to go to law school, being in a very low-dollar-value dispute with a landlord who I felt had done us wrong, and I was gonna

Greg Lambert (05:56)
Ha ha.

Vickie Wood (05:57)
fight it. And we did sort of win, but boy, it took a lot of time.

But these cases mean a lot to people, and they don’t have the same access to alternative dispute resolution as higher-value cases. So that was the other problem I was looking to fix. In fact, there is a State Bar of California justice gap study that was recently released. It was a follow-up study from 2019, then redone in 2024, with those results coming out in 2025. They reflect, let me get this right, that Californians receive no legal help or inadequate legal help for 85% of their disputes.

And I thought that was astonishing.

Marlene Gebauer (06:59)
I wish I thought it was astonishing. It’s like, I...

Greg Lambert (06:59)
Yeah. Yeah. And...

Marlene Gebauer (07:03)
I think...

Greg Lambert (07:03)
I don’t think Californians are unique in this either.

Marlene Gebauer (07:06)
No, no.

Vickie Wood (07:07)
Yeah. So this was a California study, obviously, but that’s 85% of disputes where people don’t have help, don’t have adequate help, or they throw their arms up and say, “The justice system can’t help me.”

After the 2019 study, the California Judicial Council really made efforts to try to close this justice gap. The problem was that the results from 2024 showed things got worse, not better, despite their efforts. So that was striking to me as well.

When I left the bench and started my mediation practice and started studying and using AI, it came to me that there is probably a portion of these cases that could benefit from what AI and LLM technology can do. That’s how Judicaid was born. I came up with the idea and the name, and then I happened to have a family member who is brilliant in the tech world. I mentioned it to him and he said, “I can build that for you.” And he said, “That will be easy.”

So, more than a year later...

Greg Lambert (08:27)
Yeah, I imagine in California...

Marlene Gebauer (08:28)
Words you like...

Greg Lambert (08:29)
Yeah. Well, it’s nice...

Marlene Gebauer (08:29)
to hear when you’re building tech, “It will be easy.”

Vickie Wood (08:32)
Yeah, yeah. “That’d be easy.”

Greg Lambert (08:33)
to be in California where you can’t swing a cat without hitting an engineer.

Vickie Wood (08:39)
Yeah, needless to say, it was not easy. It proved to be much more challenging, but he and the other part of the tech team did it. We’re really proud of the tech product, the rest of our product, and the platform, and excited to be getting it out in the world. So thank you for the opportunity to be here today.

Greg Lambert (09:00)
So, Valerie, I want to jump into what Judicaid does and how it’s set up. But let’s start off with the user experience. Let’s take a typical user of the system, maybe two people having a landlord-tenant, contractor, or small-claims dispute.

What happens from the moment one person opens a matter on Judicaid until the parties either reach a settlement or determine that the mediation won’t resolve it? Can you walk us through the process of Judicaid?

Valerie Clemen (09:37)
Sure. The product just went live, I want to say, like six weeks ago. There are actually two aspects to it. There’s the consumer-facing version, where people can go on, pay $34.99, and start a mediation. But we also have the court-subscription version. Napa Superior Court currently has a pilot in its small-claims cases, so it’s a little different depending on which one we’re talking about.

Essentially, it’s a website, a standalone website. Parties can go there, create their own account, and put in some basic information about who the other side is and what the dispute is about. They never speak directly with the party they’re having a dispute with. An email goes to the other side and says, “We’re sorry you’re in a dispute. The other side would like to try to work this out with you. Please sign up as well.”

Hopefully they sign up if they’re willing to talk. Once they sign up, there’s literally an AI mediator that first goes to the initial party and goes back and forth asking questions like: When did this happen? What occurred? What is owed? Usually it’s over money or property. What attempts have there been to resolve it? What do you want today to resolve it? Once it has this information, it says, “Sit tight,” and then it goes to the other side.

When it says “sit tight,” people can be doing this on their phone, laptop, or tablet, and they don’t have to be online at the same time. It basically sends a message to each side when it’s their turn to speak to the AI mediator. It’s essentially the LLM, the AI engine, asking the questions and doing this.

So it goes to the other side and asks the same kinds of questions. What’s your version of the dispute? How do you feel about it? What’s your bottom line? What do you want? Once it has that, it starts going back and forth and tries to ask questions, get more information, and bring both sides to an agreement, hopefully in the end. It’ll keep going back and forth as long as it takes.

We’ve done a lot of pressure testing. Vickie and I are both attorneys, so we were like, what would a litigant say if you’re really upset? They might tell the AI mediator to F off and say, “You’re an idiot,” or whatever. So we did all of these things, and it’s great. It’ll hear these things and say, “I understand you’re upset, but what is your bottom line?” It’ll never go to the other side and say that this side insulted you. It’s very good at filtering

Marlene Gebauer (12:16)
Yeah.

Valerie Clemen (12:17)
and focusing on the problem and how to resolve it. Once it senses there’s an agreement, maybe there’s a payment plan worked out, it’ll prepare a draft agreement and send it to both sides and say, “Could you live with this? Is this what you want? Is this the agreement you want?” If both sides say yes, it creates an agreement, sends it to both sides through DocuSign, and you have a signed settlement agreement.

Greg Lambert (12:43)
Okay. Well, let me jump in on one question. What incentive does the counterparty have? I can see an incentive where I want to pay the $35 to get the subscription, but what’s the incentive on the other side to join this mediation?

Valerie Clemen (13:08)
Usually there’s a threat of a lawsuit. Like, “I’m gonna take you to small claims. I’m gonna call the Contractors State License Board.” If it’s a home-improvement project, there’s some threat because someone’s upset and wants something. So either you resolve it with that person or they’re gonna take the next step, which could have professional consequences. You might have to deal with litigation. You’re gonna be sued. In the world of Google, if you get sued by someone, somebody Googles your name and that shows up forever.

So the ability to resolve something before it turns into something worse would hopefully be motivation. But, you know, humanity...

Vickie Wood (13:46)
Yeah, but Greg, you ask a really good question because that is a sticking point. The platform generates an automatic email invite to the other side, and it says this person has either filed a lawsuit or is considering filing a lawsuit, but they would like to first try to resolve this peacefully.

You don’t lose any of your rights if you’re not able to resolve it. I personally would be very motivated to try that, knowing what it’s about, but what we’re finding is that a lot of people don’t even know exactly what mediation is, and they don’t know what they’re getting into. We try to highlight that this is completely confidential. The court’s never gonna know anything that is said. You preserve all your rights.

But that is a tricky part of this, and we’re hoping that if it gets out there and becomes more normalized, that’ll take away some of that reluctance. On the other side of the coin, I think people in general would really like to avoid going to court, seeing this person face to face, and talking to a judge. These are scary things. People are getting less used to talking to people face to face, so we’re hoping it’s going to play into the natural progression of how people would rather address conflict.

Marlene Gebauer (15:18)
So I’m curious about how the AI is trained. I’m assuming that it trains on contract law, landlord-tenant law, things that would come up as part of these disputes. I’m also curious, we talk about personas here sometimes in terms of different types of training and things like that. I’m wondering if judge personas were part of the training.

And then how do you keep the system from crossing the line into giving legal advice? I heard a little bit, if we sense there’s an agreement, we’ll start with the decision, but how do you avoid legal advice or saying, okay, right, wrong?

Vickie Wood (16:03)
Mm-hmm. Great questions. First off, no, it’s not trained in the law.

It doesn’t give any legal advice whatsoever. It is essentially a facilitator of the communications. As a mediator, most mediators know there’s a spectrum of how you approach mediation. You can be on one end, mostly facilitative, where you’re exploring people’s values and priorities in trying to get the case resolved, and you’re facilitating communication about how they feel and their thoughts on the case.

On the other end of the spectrum is a mediator who becomes quite evaluative and says to someone, “You’re gonna lose this case,” or, you know, “Here’s...”

Marlene Gebauer (16:48)
You know, “You might want to think about this.”

Greg Lambert (16:50)
Mm-hmm.

Vickie Wood (16:52)
“This is where I’m really concerned. These are your risks.” Generally, at least in how I mediate, and I think it’s one of the better ways to approach mediation, you start out facilitative and then, as the day goes on, you can get a little more evaluative and try to push things toward that end.

Judicaid stays facilitative. So there’s no legal advice. It’s not trying to tell someone, “You’re gonna lose this case,” or, “They’re right on this point and you’re wrong,” or vice versa. It will remind them of the risk and hassle, the time and expense of going to court, and the risk that they may not win their case, encouraging them to continue to think about compromise and keep engaging so they inch their way toward a palatable compromise where eventually they go, “Okay, I can say yes to this and be done and not have to deal with court.”

So that’s the idea, and that’s why we’re able to offer it nationally, because it doesn’t deal with laws that would be individualized to each state.

The training aspect is really, there are two main tech pieces. There’s the application that’s coded, which is our AI engine for the shuttle-style back and forth. That took a really long time to create. Then there is the mediator disposition architecture, and that’s where I had a lot of input as the subject-matter expert. It’s like a 38-page prompt that...

Marlene Gebauer (18:42)
Ha ha ha.

Vickie Wood (18:42)
That...

Greg Lambert (18:42)
How?

Vickie Wood (18:43)
guides Jude, our mediator’s name is Jude, by the way. It tells Jude...

Marlene Gebauer (18:48)
Do you say, “Hey Jude,” when you talk to it?

Vickie Wood (18:51)
Jude. We actually...

Marlene Gebauer (18:53)
I’ll leave now.

Vickie Wood (18:54)
never intended to have a name, and it just sort of came about. Since it’s Judicaid, and Jude’s sort of a gender-neutral name, we thought, why not? We’ll go for it.

But it’s the disposition, sort of prompting architecture, that tells it how to respond and how to continue to inch people toward a compromise. There are lots of guardrails so it’s never giving legal advice. It can suggest, if someone says, “Do you have any ideas on how we might be able to resolve this?” it can say, “Well, one thing you might consider is this,” or, “You could take payments over time,” or, “Maybe take a smaller payment up front and then the balance over time.” It can suggest things like that, but it’s very generalized in that way.

That’s a big part of why it was never intended to replace attorneys or human mediators. As a mediator, I know that most complex cases that go to human mediation are far too complex, both legally and emotionally, for an LLM to manage. So this is not trying to replace that. This is focused on filling the gap for those lower-value cases that are generally fairly simple.

There is emotion involved, and that’s something an LLM is very good at managing in communication. It’ll calm the person down, and they can get as upset as they want. It can be empathetic and say, “I hear you’re feeling really frustrated about that, and that’s understandable. But why don’t you let me go talk to the other person?” It’ll go to the other side, and it’s never gonna convey the anger or the bad words being said in the other room. It’s gonna say, “They’re feeling concerned about this.” So it paraphrases things in a way that makes them far more palatable to hear when you’re the adversary.

That’s a big part of what I do as a human mediator. When somebody is cussing and swearing and saying horrible things about the other side, I go in the other room and say, “Yeah, well, they’re starting to come around.” So...

Greg Lambert (21:17)
Yeah, I was gonna say, do you have it default back to those instructions so it doesn’t get caught up in what each side is feeding it, but rather goes back to its baseline to make sure it stays on the course it needs to stay on?

Vickie Wood (21:40)
Yeah. Like I said, the prompting architecture is vast, and it’s very carefully thought out in phases to guide things through. But it has priorities around guardrails, and if there’s any sort of safety issue, somebody says, “This is terrible. I’m gonna kill myself,” we make sure it’s gonna say, “If you’re struggling with that, then you need to consult resources in your locality. This may not be the place to do this.”

And there’s a button to say, “Stop mediating,” and terminate the mediation at any point in time. So, yeah, a lot of thought went into that...

Marlene Gebauer (22:25)
I have one. I have one.

Vickie Wood (22:27)
Aspect.

Marlene Gebauer (22:29)
Does it learn? I mean, from session to session or are those isolated?

Vickie Wood (22:34)
They’re isolated. We have made the decision at this point that we’re not training the AI model directly in the way that an open-source LLM would be constantly training itself.

Yeah, it’s truly intended to be confidential, just like mediation is confidential. You’re not supposed to go tell a court, “Well, in mediation she said this or that,” or you’re not allowed to, at least in California.

Greg Lambert (23:03)
And that’s when you hold up your hand saying, “I don’t want to hear that. I don’t...”

Vickie Wood (23:06)
Yeah. Yep. Exactly.

Marlene Gebauer (23:08)
No, no.

Vickie Wood (23:09)
Exactly. Privileged. So...

Greg Lambert (23:12)
Well, we talked about the user side, so let’s go over and talk about the court side. Courts appear to be a really important path for Judicaid going forward. Valerie, I’ll start with you, and Judge Wood, if you want to jump in, please do. What would a court-facing model look like in practice, and where do you see it fitting into the life cycle of these disputes?

Is it before the case is filed? Is it after, but before a hearing? Is there someplace else? Where do you see the path of leveraging either the threat of the court or the actions of the court to help Judicaid?

Valerie Clemen (23:55)
Yeah. Initially, it would be good to have it in self-help centers, because a lot of times, especially for small claims, landlord-tenant, or those types of claims, those parties are unrepresented. You just can’t afford to pay an attorney’s hourly rate for a case that’s honestly below $20,000. It doesn’t make any sense to pay an attorney, so they’re trying to figure out the system themselves.

Knowing they have this option to resolve their dispute without going through the paperwork, getting it served, and filing is helpful. So having that information in self-help centers and in the courts pre-filing is helpful. Once a case is filed, it would be helpful for courts to provide that option and, honestly, not make it voluntary. Make it mandatory to give it a try, to talk to the other side using some type of dispute resolution or mediation, but also provide Judicaid as a free resource for litigants to use and try to resolve the dispute.

The information would come at the time of filing. Like, here’s your complaint, you have to serve the other side, and also serve the other side with this dispute-resolution information to see if you can resolve the case before your hearing date comes up. In some courts, when your hearing date is set out six months or a year, being able to resolve it and put the thing behind you before that happens, or before you spend money having the sheriff personally serve it or hiring a process server, is helpful. There are so many costs and so much time associated with litigation.

The hope is that letting litigants know there’s this option would be helpful. Of course, a lot of unrepresented litigants are doing this for the first time. They’ve never filed a small-claims suit. They’ve never been in a legal dispute. A lot of them probably don’t even know what mediation is. I’d say at least 10% get the flyer and are like, “Meditation? I know I’m angry, but I don’t need that.”

Marlene Gebauer (25:48)
Yeah.

Valerie Clemen (25:49)
Like, just not having any idea.

Vickie Wood (25:51)
It should go hand in hand, though.

Greg Lambert (25:51)
I need it, but not in this case.

Valerie Clemen (25:54)
Yeah. Because honestly...

Greg Lambert (25:57)
So are you educating law librarians and self-help centers on this, so they have it as a tool they can point people to as an alternative, or...?

Vickie Wood (26:11)
We’re trying to. We’re trying to. We just...

Marlene Gebauer (26:15)
That’s why they’re on the podcast, Greg.

Vickie Wood (26:18)
But yeah, in Napa County, where we’re running the pilot, we have definitely put it out there to all of the people who might be interested. We’ve got flyers in the law library and in the regular county library. While it’s being offered for free, government agencies have a line where they don’t want to be advertising something that’s for profit.

But while they’re piloting it, or if they were, and actually, let me jump back to one of your questions, Greg, about what this would look like through the court. The platform itself will be identical. The courts would pay for a monthly subscription, and then they would be given a QR code that they could provide to litigants upon filing. They could even put it on their website, in the small-claims area, for people who are thinking of filing a claim to try it before they file.

Those people would enter the site through a landing page for that court with that QR code. There’s a little gateway where they have to say how they got there. If they didn’t get there through that court, we’re going to redirect them to the paid portion. But if they got there through the court through the QR code appropriately, then they bypass the payment piece and jump into the platform exactly the way everybody else does.

So, yeah, we’re trying to get this out there. We’re on social media. We just got back from a national court conference in Delaware. We were having conversations with courts across the U.S., getting them aware. I think the trick in trying to get courts on board with this is that most of the technology out there needs to be integrated into the courts’ networks and processes.

Their first thought, I think, is, “Here’s yet another AI-based app or process that we’re gonna have to implement and then be tied to this company for however long, and tied to their schedule of upgrades and new features and their pricing changes and all of that.” We’re completely different from that. The only thing the court has to do is give that QR code to its litigants or potential litigants, and any questions go through our support page. That’s it. That is it.

I think courts are having a hard time comprehending that it could be that easy. Hopefully we’ll get the message out. What was that, Marlene?

Marlene Gebauer (29:01)
This could be very helpful for them.

Greg Lambert (29:02)
Yeah.

Marlene Gebauer (29:05)
It could be very helpful for them in terms of the docket. So...

Vickie Wood (29:11)
Yeah, that’s the idea. And it’s so affordable that it’s sort of a win-win-win proposition, and it feels like a no-brainer to us. But still, it’s gonna take some time to get people to first be aware of it and then have a true understanding of how simple it is.

Valerie Clemen (29:29)
Yeah. I’ve had clients come to me, we’re in a small town, so I have clients who come to me with certain issues, and sometimes they come with things that are just so minor. They have a dispute with their neighbor and the money involved is so low, and I’m like, “Look, I can’t honestly have you pay me for this, but try this.” I’ve had a couple of clients go try Judicaid, and they were like, “It was easy to sign up and send the invite, but the other side didn’t respond.”

But it was nice. Sometimes clients come to you and they have things that are too minor, and you’re like, “Look, the budget for this doesn’t make sense.” Giving them something, some tool where they might resolve it on their own, was nice instead of saying, “I can’t take this. Good luck.”

Vickie Wood (30:12)
Yeah. So that is another sort of distribution channel, if you will. Exactly what Valerie just said, for attorneys to be able to give something of value to those clients they have to turn away. Say, “There’s another tool here.” We think it’s something attorneys would really like having in their back pocket for that situation. The problem is there is that immediate reaction of, “What is this taking? Is this going to take away our business?” Or, for...

Greg Lambert (30:45)
Yeah.

Vickie Wood (30:45)
mediators, “Is this going to take away our business?”

And I’m saying, no, it’s not that. LLMs are obviously changing the legal scene drastically, but at this level of sophistication, they can’t replace humans. You absolutely need human mediators. So hopefully, if there are attorneys out there listening, this is something for you to have to give to the people who you can’t otherwise help.

Valerie Clemen (31:16)
Yeah. It’s like...

Marlene Gebauer (31:16)
Yeah, I’m kind of looking at it like it’s an initial step because you’re absolutely right. I was very curious, a human mediator can sense when there’s unequal bargaining power or someone’s more legally sophisticated, and they can run their mediation with that in mind. But I’m curious how Judicaid does that as a technology tool.

Vickie Wood (31:43)
Well, in some ways it’s more neutral. Because it’s just going to be kind of...

Marlene Gebauer (31:47)
Yeah. It’s neutral. That’s it.

Vickie Wood (31:54)
Yeah, yeah. It’s not gonna be impressed by the person who is sort of a bully or sounds like they really know what they’re talking about. It’s not gonna be more persuaded by this side or that side. Again, it’s facilitating that conversation in a really neutral way.

And to your point about it being a starting point, yes, I do think cases will get resolved through this. But if they don’t, I also think there’s another value that goes beyond monetary. Let’s talk about this, it isn’t limited to small claims. It could be used for limited civil cases. It could be used for eviction cases. We’re gonna add a feature to try to get this into family law for parent-coordination kinds of things. I have unlimited features in my head, so it’s just a question of getting them to happen.

What I would see with small claims is, like I said in the beginning, people are passionate about these cases and they spend days’ worth of time preparing for them and putting everything into them. Then they get to court and there are 40 cases on the docket for trials. They’re gonna sit there for a long time, their case gets called, and they get maybe 10 minutes. And...

Greg Lambert (33:13)
Yeah.

Vickie Wood (33:14)
even if they win, but especially if they lose, they walk away not feeling good about the process and about the justice system, because many times this is these people’s only real interface with the courts. They walk away not feeling good about it, and that’s not good for any of us. That’s not good for society in general.

So if they start with a product like Judicaid, where they’re able to kind of be “heard,” and I’m putting that in air quotes for anyone who’s only on audio, to air their feelings about it and also hear the other side’s position and thoughts, and then have to think through the potential outcome that it may not go their way and decide they want to take that risk, well, now they’ve had much more participation in the ultimate outcome. I feel like that’s going to be an improvement for everybody across the board.

Greg Lambert (34:12)
Yeah. Well, I want to touch on your pilot with the Napa County Superior Court. Tell us about what you’re piloting. Beyond finding out whether the technology works, what are you hoping the courts will find out, and what are you hoping to find out from this pilot on your side?

Vickie Wood (34:35)
Yeah, so we’re new with the pilot. We’re kind of just starting to get through the period from where we started to where cases would be coming on to trial. We have had people engaging, but we’re not getting the mediations to happen yet.

So we’ve learned one really important thing, which is that people are gonna be reluctant to do this right now if it’s purely voluntary. That was a very valuable thing to learn. We knew there would be reluctance, but there’s significant reluctance.

One note, though, is that Napa County is a tiny county, so the number of filings is a very small sample to be working off of.

Greg Lambert (35:18)
I’m envisioning the mediation being over a glass of wine. That’s all I see.

Vickie Wood (35:22)
Of course, Greg, of course.

Marlene Gebauer (35:22)
Ha ha.

Vickie Wood (35:25)
If they’re not drinking wine, then the dispute is about wine somehow. But yeah...

Greg Lambert (35:30)
Yes.

Vickie Wood (35:32)
It’s a small county. But yeah, once we start getting engagement, and we’re connecting with some larger counties, one in particular is a large county that already mandates mediation for both small claims and unlawful detainers. They offer some human mediation options that are lower-cost than most human mediation, but still significantly more than Judicaid. So this would be another thing to offer people.

One of the greatest features about Judicaid is that it can be done in many different languages. You can have somebody who speaks only Spanish and someone who speaks only Russian mediating together. That would be very expensive to provide with a human mediator because you would probably have to have two interpreters for one mediator, and the cost skyrockets.

So, yeah, we’re hoping to get a bigger sample, especially one where it’s mandated, because we think if it’s mandated, a lot of people will opt for this because it’s so easy. You don’t have to schedule a set time. You can do it literally over your phone, dictating.

Greg Lambert (36:47)
And you mentioned that this isn’t just California, right? You’re looking at going...

Vickie Wood (36:51)
Correct.

Greg Lambert (36:54)
going into other states as well.

Vickie Wood (36:56)
Correct, correct. Yeah. And maybe, I mean, honestly, I don’t know why it couldn’t go outside the U.S. too, because again, it’s not tied to any particular laws. So I don’t know. I don’t want to think too big just yet, but...

Greg Lambert (37:09)
Well, then you have to deal with the EU and GDPR and all that, so, you know...

Valerie Clemen (37:14)
Yeah, yeah.

Vickie Wood (37:15)
Yeah. Well, I’ve been really bored lately, Greg. That would give me something to do with...

Greg Lambert (37:19)
Yeah.

Vickie Wood (37:19)
my time.

Marlene Gebauer (37:19)
Yeah.

So we kind of close the podcast by looking back and then looking forward. This is kind of our “what’s different now” question. If we had spoken with you a year ago, other than the fact that maybe Judicaid didn’t exist, what would you have told us is materially different today? What have users and court partners, or the experience of building the platform, taught you that changed your original assumptions?

Vickie Wood (37:56)
When I conceived of it, which was closer to two years ago, the way ChatGPT or any of these large language model platforms functioned was astonishing. I mean, it still is astonishing, but more people were seeing it for the first time or had not seen it yet. So the concept of what Judicaid does was almost like magic. Now people are aware that having a computer talk to you like it’s a human can be done.

What we have, that I don’t think is done anywhere else, is this shuttle-style approach where it’s truly exchanging the communications. But when my CTO told me, “This will be easy,” I thought we would be launching in like two months. Then all these months later, it’s not quite as novel as it was at the time.

At the same time, there’s also a fair amount of skepticism of AI and concern about AI taking jobs. There are people who might see that it’s an AI-facilitated platform and want nothing to do with it, without considering that this is something trying to provide a positive benefit and fill a gap. Those were unexpected things, but we’re rolling with it because I think it’s gonna require a little bit of awareness about what we’re doing.

Also, AI is becoming so pervasive that I think eventually people are...

Greg Lambert (39:41)
Yeah.

Vickie Wood (39:42)
just gonna get tired of being so against it, I guess. But yeah, I think the other thing is we focused on mediation because that’s the piece we wanted to nail down. Now that we have that core AI engine, we can branch off in so many different dispute-resolution ways, which gets me so excited.

We’re now calling it intelligent dispute resolution because it may not just be mediation. We might start having mediator’s proposals. We might have a parent coordinator to help with little custody disputes.

ODR, online dispute resolution, has been around for quite some time, but I think intelligent dispute resolution is new. So I’m gonna say you heard it here first.

Greg Lambert (40:29)
All right, all right. Well, that leads in perfectly to our crystal ball question. Valerie, I’ll let you start this off if you don’t mind. So...

Valerie Clemen (40:38)
Sure.

Greg Lambert (40:39)
Picking up on Judge Wood’s discussion there about intelligent dispute resolution, in three to five years, where do you see things when it comes to AI-assisted dispute resolution? What’s it gonna look like, and what all do you think it will cover?

Valerie Clemen (40:58)
I mean, our hope is that it would be offered in courts nationwide, because this underserved population doesn’t have access to dispute resolution or mediation, and this would provide that. So we would like to see it used in courts nationwide.

I would also, I mean, this is hopeful, but it would be nice if people just thought to “Judicate it”...

Vickie Wood (41:19)
Yeah.

Valerie Clemen (41:19)
when they have disputes, to go there first. You see people arguing on Nextdoor. Sometimes you think we should have guerrilla marketing and just go in there and say, “Just Judicate it.” Like, there’s so many...

Vickie Wood (41:32)
I don’t know if Nextdoor is national, but it’s this neighborhood platform. And yeah, so...

Greg Lambert (41:35)
It is. You know. No, we all...

Valerie Clemen (41:36)
It is.

Greg Lambert (41:38)
know. “Was that fireworks or gunshots?” is the most popular...

Valerie Clemen (41:41)
Yes, exactly. Exactly. “Does anyone...”

Vickie Wood (41:46)
Yeah.

Valerie Clemen (41:44)
“recognize this car?”

Greg Lambert (41:46)
Ha ha.

Valerie Clemen (41:47)
But people have disputes, and it’s really hard to get people talking to each other. Oftentimes, when people start talking to each other, they resolve things. It’s that initial step, because it’s uncomfortable, it’s adversarial. But once people do it, they can often resolve a lot of issues.

So the idea that people could use this as a tool to help get things started, it’s really hard to walk up to someone...

Marlene Gebauer (42:12)
Okay.

Valerie Clemen (42:13)
and confront them and talk about something, even if you have it simmering. I mean, people have a hard time with that with family members. You’ve had this thing simmering forever, but you can’t address it.

This is a tool that helps you be a little protected but still share your thoughts and feelings and resolve disputes. The idea that people could use it for all kinds of disputes, I mean, it is a very civil way to resolve disputes and quash things, and not have things linger for a long time.

But obviously, in the legal sense, it would be nice when people do have disputes to the level where they’re ready to file a lawsuit or do something more, that they have a way to either solve it beforehand or try to solve it once the lawsuit has started. They don’t have attorneys helping them in a lot of these cases. They’re probably using Claude or ChatGPT to help them with their case. So this is another resource to help them resolve it instead of keep fighting.

Vickie Wood (43:09)
Yeah, or have it escalate to something more than it ever needed to be. And just to tack on to that, I think that’s exactly right. A lot of times it’s barriers to communication, and that’s what Judicaid is trying to resolve. Whether it’s lack of skill with communication, a power dynamic, or a language barrier, that’s really where Judicaid provides this platform at a very base level.

And hopefully, this is my tagline that I have in our little video, my hope is someday that instead of saying, “Let’s take it to court,” we’ll say, “Let’s just Judicate it” instead. So that’s our hope.

Marlene Gebauer (43:49)
I like it. I like it.

Well, Judge Wood and Valerie Clemen, thank you for joining us and giving us a closer look at Judicaid, at both the opportunities and difficult questions that come with using AI and expanding access to dispute resolution. And thanks to all of you for listening to The Geek in Review. If you enjoyed the show, please share it with a colleague. We’d also love to hear from you on LinkedIn and Substack.

Greg Lambert (44:17)
And Judge Wood and Valerie, where can listeners go and find out more about Judicaid?

Vickie Wood (44:26)
Judicaid.com. We have a lot of information on there, including an instructional video that’s on YouTube, but you can access it from the homepage, along with how-to instructions. We’re on Instagram, Facebook, X, and LinkedIn, but that’s probably the best place to start.

Greg Lambert (44:47)
Soon to be on Nextdoor.

Marlene Gebauer (44:48)
All the places.

Valerie Clemen (44:50)
Yeah.

Vickie Wood (44:51)
I actually am on Nextdoor. That’s true. I haven’t really spent a lot of time on there yet, but yeah...

Marlene Gebauer (44:55)
Ha.

Vickie Wood (44:56)
I’m going there, Greg.

Greg Lambert (44:59)
Awesome, awesome.

Vickie Wood (45:00)
Thank you so much.

Marlene Gebauer (45:02)
And as always, the music you hear is from Jerry David DeCicca. Thank you, Jerry. Goodbye, everybody.

I ran across a post on X this weekend that pointed out 10 anti-slop skills that are supposed to take AI writing and filter out the things that make it stop writing like AI. I went through all ten of those skills and then I also asked my AI tool to clone the ten repositories and read through them, and create a report on the commonalities and unique features of each skill.

It’s easy to see why I would want to do this. Anyone who spends any amount of time on places like LinkedIn sees the slop everyday. Tells like ‘delve’, ‘tapestry’, and the three em dashes in one paragraph from a writer who never used an em dash in their lives before 2023. We’ve learned these signals of AI writing, and when we see it over and over again, we stop paying attention to what’s written, and start asking how much of this writing is actually from you.

What exactly showed up in the ten skills? A lot of it was repeated instructions that banned words like delve and tapestry. Typical words that a Claude or ChatGPT puts in its writing. Of course, em dashes are out, along with the run of sentences that have things like “in today’s rapidly evolving landscape.” That’s all well and good. Each one of the ten was good, but just like AI writing, it gets you to some type of average that ends up looking like everyone else again.

Where it gets really interesting is in the parts where the ten skills differ from each other. It took some side-by-side evaluation to see where those difference stood out

One that I really liked was a project called slopbeth. Which is going to be the name of my new Metallica cover band. Slopbeth pointed out something that was missing from the other nine skills. That was that creating a banned word list was merely treating the symptoms of the AI slop problem. Because AI tools are set up to predict the next likely token (word/phrase), over and over so that it comes up with a pretty average sentence that isn’t really directed at a specific audience. Banning words just means it is going to pick the next generic word and the underlying structure of the writing doesn’t change at all. So slopbeth pulled a concept from 1984 writer, George Orwell and designed a skill that was built around Orwell’s six rules essay. Attack the problem as the writing is being done instead of cleaning up the draft after the fact. Use short words whenever possible. Delete unneeded words. Use active voice as the default. The sixth rule may be the most important of the batch, which is that you should break any of the previous five rules if it means they are causing you to write something ugly. For an eighty year-old essay, it holds up pretty well in the age of AI slop writing.

The second lesson I learned came from Peter Yang’s skill, and it reminded me of something that I was told when I started editing podcasts. It’s okay to let someone take a breath. In other words, don’t over edit something because it makes it sound artificial. That is a problem with some of these anti-slop passes that it scrubs the writing so hard that it is perfectly clean, but with no life left in it. That’s the sort of thing that just grates on a reader’s nerves. Yang phrases this as a question. Would the writer defend their writing choices if you called them out on it? I know that some of you don’t like things like double exclamation points. I’m fine with it. I let some of my Oklahoma and Texas style show up in my writing. That’s also something that I rather like in my writing. But the mannerisms that AI puts in because there is a statistical pattern telling it to put them in there, just don’t have anyone behind them to defend those styles. That’s a big difference.

The third skill that I found useful is blader’s humanizer, and this one reminds me of the ‘human in the loop’ argument that we all make, especially when talking to lawyers about using AI to write their material. This was taken from the Wikipedia editor guide called Signs of AI writing, which is a great read as well. The big rule here is to never let a cleanup pass create a new fact. Or, more importantly, never let it invent a fact. Made up information doesn’t just expose AI slop writing, it can, at a minimum, be embarrassing. On the other end of that embarrassment, it can lose your client’s case, and get you sanctioned.

I went ahead and pulled this altogether, and used Claude for what it is really good at, and that is helping to write a skill. We took the rules from the group of Anti-AI Slop skills, and put them together into a fresh skill. I’ve labeled it as slop-sweep. The skill works in five steps. One, it identifies what parts of the writer’s voice need left alone and protected. Two, applying Orwell’s rules, it attacks the problem as it is writing, and knows to break the rule rather than write something ugly. Three, sweep for those tell-tale signs of antithesis writing (all those “not X, but Y” tells). Four, fix the rhythm by breaking the rhythm, like same sentence or paragraph length. And five, ask the AI to re-read the material and point out which parts of the draft still reads as if AI wrote this.

At this point you are probably saying to yourself that this is a lot of work to get AI to not sound like AI. You’re right. These are skills that I found helpful, and I’m sure there are dozens of other rules out there beyond this that all work toward cleaning up AI slop. No matter how many tools you use to clean up AI slop, if you’re not applying your own ideas, discipline, and uniqueness to what you’re creating, then all the anti-AI slop tools out there won’t matter in the least. The best skill file in the world can’t help you if you don’t apply your own personality and skills to what you are creating.

We are all pushing out more AI created content every day. In law firms, we are seeing more client alerts, newsletters, RFP responses, and thought leadership articles than ever before. Some of it (most of it?) may be leaving before any one checks it for AI slop. Use the AI slop-sweep skill to help you identify slop before your readers do, but remember that this only gets you part of the way there. You still need to rework the material so that you are ready and willing to stand behind it.

What happens when a 40-year-old legal data company decides its employees should start building their own software? This week on we talk with Best Lawyers CEO Phillip Greer and Senior Vice President of Research and Product Strategy Elizabeth Petit about an internal AI transformation that reaches far beyond adding ChatGPT to the corporate toolkit. Best Lawyers is experimenting with generative engine optimization, internal agentic systems, vibe coding, and an AI development environment where employees across research, finance, marketing, and other departments build applications around the company’s data.

Greer begins with a challenge facing every law firm marketing team: traditional search is changing. Google AI Overviews and answer engines such as ChatGPT, Claude, and Gemini increasingly give users answers without sending them to the familiar list of blue links. Greer argues that SEO still matters, but law firms now need to think about Generative Engine Optimization, or GEO, and the signals AI systems use when deciding which sources deserve trust. Structured data, schema markup, substantive content, and third-party validation all become part of the equation. For Best Lawyers, its long history of peer-reviewed rankings offers an interesting advantage. The company’s data serves as an independent signal that AI systems might weigh differently from content produced by a firm’s own marketing department.

Petit explains how Best Lawyers is applying the same thinking to legal marketing through Smithy AI, a system designed to help attorneys and law firm marketers develop profile content without endlessly copying the same biography across websites. Smithy draws from Best Lawyers’ structured information and existing lawyer content to produce a starting point that attorneys and marketers then edit. The larger goal is authenticity. As generative systems make producing generic legal content almost effortless, Greer argues that distinctive expertise, voice, and credible third-party signals become more valuable rather than less.

The conversation then moves inside Best Lawyers, where Greer has taken a far more unusual approach to AI adoption. After building a secure data layer connecting systems including SQL databases, HubSpot, Gong, Google Analytics, and accounting data, he created an internal Best Lawyers App Store where employees use natural language to build applications against company data. What began with roughly 30 percent of the workforce vibe coding has grown to around 40 percent, according to Greer. Petit describes building research and KPI dashboards despite coming from a research rather than software engineering background. Projects that once required Excel formulas, Power BI reports, development queues, and weeks of waiting now sometimes move from a question at 9:30 to a working internal application by 10:30.

That shift also changes the role of professional software engineers. Rather than spending their time building another reporting screen or internal form, Best Lawyers’ engineers increasingly concentrate on architecture, data infrastructure, performance, governance, and the guardrails surrounding employee-built applications. Greer describes moving parts of the company’s data architecture toward Elasticsearch and developing “Bestie,” an internal agentic AI team member. Yet speed introduces another problem. Petit and Greer describe an “AI vampire” effect, where instant feedback encourages people to keep working because the machine never gets tired, goes home, or stops responding. Human judgment includes knowing when the human needs to stop.

The discussion closes with what these changes mean for legal practice itself. Best Lawyers introduced Artificial Intelligence Law as a formal practice area in the 2026 edition of The Best Lawyers in America, reflecting how AI work has spread across technology, intellectual property, privacy, employment, compliance, litigation, and counseling. Looking further ahead, Greer and Petit do not expect AI to erase the billable hour overnight. They do expect clients to ask harder questions about what they are paying for, how legal work was produced, where AI contributed, and where the lawyer’s judgment created value. If AI makes routine production dramatically faster, the economic question for law firms becomes less about how many hours technology saves and more about how firms explain, price, and defend the value of human expertise.

LINKS

• Best Lawyers

• Smithy AI overview from Best Lawyers

• Best Lawyers Artificial Intelligence Law rankings

• Best Lawyers on the introduction of Artificial Intelligence Law

• Best Lawyers ChatGPT legal search app

• Phillip Greer on AI search, AIO, and GEO

• Phillip Greer on vibe coding and the Best Lawyers App Store

• Phillip Greer on Bestie, the Best Lawyers AI team member

Listen on mobile platforms: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Apple Podcasts⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ | ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Spotify⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ | ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠YouTube⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ | ⁠Substack⁠

[Special Thanks to ⁠⁠Legal Technology Hub⁠⁠ for their sponsoring this episode.]

Email: geekinreviewpodcast@gmail.com

Music: Jerry David DeCicca

Transcript

Greg Lambert (00:01)
Hey everyone, I’m Greg Lambert from The Geek in Review, and this week we are talking with Phillip Greer and Elizabeth Petit from Best Lawyers. But first up, here’s a word from our great sponsor and partners at Legal Technology Hub.

Marlene Gebauer (00:15)
I have Sarah Glassmeyer here from Legal Technology Hub, and since it’s almost time for ILTACON, Sarah’s going to share with us how you can use the Legal Technology Directory to prep for ILTACON.

Sarah Glassmeyer (00:30)
Yeah, so this is probably one of the more exciting times of the year for legal tech, kind of leading into ILTA. Everyone’s excited to see all the new changes, see people we haven’t seen for a year. So Legal Technology Hub can help you pregame and get ready to get the most out of it because, while you’re there, it does seem like you live now in Nashville and you’ve been there your entire life, but it goes fast. You only have two and a half, three days to talk to vendors.

So a couple of things I would suggest you do. One, look at the list of who is exhibiting at ILTACON. If you go to the ILTA website, they have a legal tech directory, but guess what? That is us.

We partner with ILTA. It is a mirror of our directory. You can see the little badges, and that will tell you who is exhibiting at ILTACON. Don’t wander in there. Think ahead and think, what am I looking for this year? Am I looking for a new kind of document automation tool? Am I looking for a new AI legal assistant?

So you do some filtering, make a list, see who’s going to be there, and start thinking ahead. Through the directory, you can see, these people don’t integrate with X tool, so they’re kind of off the list for us. You can do a little pre-filtering, pregaming, and figure out who you want to talk to, who is going to make the most of your time.

And then from there, if you are back on our platform in the Legal Technology Hub directory, if you’re logged in, you can make notes. So as you’re wandering through the exhibit hall visiting the people you want to visit, you can make notes saying, here’s this person’s email address that I spoke to and I want to follow up with them, or maybe show Bob this when I get back to my office, or these people are off the list, forget about them.

And so you keep track of who you’ve talked to, what you’re thinking about things live, and it’s all in one place on the Legal Technology Hub directory, not a swag bag full of business cards and flyers. So that’s how we help you get the most out of what you’re doing.

But also, I do suggest wandering around the exhibit hall, because everyone who’s there does have a listing on Legal Technology Hub, so you can double-check. But I think it’s good to wander, see who catches your eye, have a spontaneous conversation, because you might find a vendor you never considered or something you weren’t thinking about ahead of time. So it is a fun time to get yourself absorbed in what’s happening new in legal tech.

Marlene Gebauer (02:46)
Yeah, serendipity is definitely part of the ILTACON experience, but this is great advice in terms of using the Legal Technology Hub directory because it can be overwhelming, and this will allow you to focus on the vendors and people that you definitely need to talk to before you leave.

Marlene Gebauer (03:14)
Welcome to The Geek in Review, the podcast focused on innovative and creative ideas in the legal industry. I’m Marlene Gebauer.

Greg Lambert (03:21)
And I’m Greg Lambert, and today we are exploring how a legacy data institution is completely rearchitecting itself from the inside out for the generative AI era.

Marlene Gebauer (03:34)
We are thrilled to welcome Phillip Greer, CEO of Best Lawyers, and Elizabeth Petit, Senior Vice President of Research and Product Strategy. Phillip and Elizabeth have been leading a radical internal and external transformation at Best Lawyers, turning the organization into a pioneer for agentic workflows, vibe coding, and generative engine optimization. Elizabeth and Phillip, welcome to The Geek in Review.

Phillip Greer (03:58)
Thank you so much for having us. We’re quite excited to be here today.

Greg Lambert (04:03)
Right. Well, Phillip, I’m excited about this because I think a lot of us have been hearing about SEO for years, and now recently, with all the AI, we’re hearing more about generative engine optimization, or GEO.

Phillip Greer (04:18)
Mm-hmm.

Greg Lambert (04:20)
Last year you noticed that there was a massive drop in traditional web traffic due to all of these AI overviews that you see with Google and others, to where people are getting answers rather than getting links out to the websites and then going and finding the answer themselves.

So you talk about your pivot away, or, well, I’m sure you still use SEO, but you’re also having to implement this new generative engine optimization, and that differs a lot from the SEO that a lot of us here at law firms have grown up on and realized, in order for us to get high up on the Google rankings, we need to be good at that.

So how does GEO differ from traditional SEO? And why is it that these third-party reviews or peer reviews are becoming the primary, what you call a trust signal, that keeps firms from being compressed out of the AI search answers that we need to get involved in?

Phillip Greer (05:26)
Yeah, so I think it’s helpful to give a little bit more context. You started that process.

We’ve all been chasing SEO for over two decades, right? Google came out, and then, how do we get to the top of that Google ranking so that our customers and new clients can see the work that we’re doing? And we did that through so many things, backlinking efforts, keywords, thought leadership pieces, marketing associations, marketing partnerships with different types of organizations, et cetera, et cetera.

It’s been popular to say that SEO is dead. So first off, I don’t think SEO is dead. Traditional SEO is still important, and it’s the foundation of AIO.

So a few years ago, Google changed their search engine, and when you started searching, all of a sudden the middle of the funnel, or that blue-links world, started to go away, and you were using that Google Gemini. So you would say, “Hey, I need to find a lawyer for an issue I’m having,” and then it would give you a summary.

And then what was happening was so many law firms, and any type of business, were losing 15, 20, 30 percent of their web traffic because the blue links aren’t being clicked anymore. The users are going in, they’re getting that overview, they’re getting what they need, and they’re kind of moving on.

Well, those original AIO types of overviews were being fed from those SEO efforts.

Marlene Gebauer (06:51)
Yeah.

Phillip Greer (06:51)
The shift that we did over the last couple of years is, first, make sure that our content is structured in a way that, when these LLMs such as ChatGPT or Google Gemini or Claude are coming to our website and looking at our data in traditional SEO forms, we’re tagging it and presenting it in ways that are helpful.

So you think about SEO stuff you used to do. It was all about meta descriptions, keywords, page titles, and then, again, we talked about some of the backlinking techniques. Those things were structuring it so that, when the crawlers are finding your website, they know how to get that data.

Well, that’s kind of amped up with AIO and then eventually what we’ll talk about with GEO. You not only needed the meta descriptions, you needed to have good schema markup. You need to describe sections on the page, to be able to ask questions, answer questions, so that when the AI engines were gathering data, it wasn’t only finding it, they could start to interpret it, because that interpretation is what becomes a little more important when you think of things like ChatGPT or Google Gemini.

It has to take the structured data, and then it has to interpret it and create insights and say, okay, now you can start showing up in these overviews.

The latest shift, and I think it’s the biggest shift, is this GEO, and that was your opening question.

Greg Lambert (08:14)
Right.

Phillip Greer (08:14)
The generative engine optimization approach. And that is making these AI systems not only get your data, but understand that they can trust your data. And so, how do you trust data?

Data has to come from quality places, things that are not produced by a marketing staff. So we’ve all gamed the SEO world. Every year, Google changed the algorithm again. What do we do? How do we get to the top? We spend more dollars. We make sure that we have more content. We flood the internet with it. We do backlink strategies, right?

That would work for SEO, and then it would work until they changed the next algorithm again.

With GEO, you have these AI systems that have some insight and intellect in the sense that it knows whether or not the content is being produced by you, your law firm, putting more information out there, or are you reading a service like Best Lawyers that’s known for producing peer review surveys and, ultimately, lists of lawyers? It’s an annual process. It’s been around for 40 years. It’s got a strong standing, a big domain authority. It is a trusted source. It’s third party to the industry that is putting out their data, right?

So that type of association is important. That’s where GEO plays a role.

So we think about our customers and clients who are part of Best Lawyers, who make those rankings. They get a leg up immediately by making it onto our list because they’re getting a signal, an AI signal in the industry for these LLMs to say, okay, if they’re on Best Lawyers and they’re ranked in corporate law, then their value has a higher weight.

And there are other types of services that you should be paying more attention to, and you want to make sure they’re credible. And so that’s why, when you think about what is your plan going forward, I think for law firms you should continue to think SEO is important, but it’s not the only thing. And that’s the takeaway.

SEO is important. It’s not the only thing. Structuring our data is important, and let’s make sure that we’re also thinking deeper about the schema so that we can have questions asked and answered. And then how are we putting our data in? How are we associating? What are we saying? Go see us on, go connect with, what type of services, and what’s their credibility level?

That’s what is creating the big buzzword for the last year, the AI signals. And all those together create a strong AI signal for your law firm.

Greg Lambert (10:39)
Yeah.

Well, one of the things that I’ve been hearing from marketing folks is much more long-form articles, client alerts, more showing your expertise in an area. Are you also seeing the generative AI or the AIO, and it sounds like a song,

Phillip Greer (10:58)
Yeah.

Greg Lambert (11:00)
or the GEO relying upon, instead of straight-out structure and how things are presented, deeper content as well? Are you seeing that?

Marlene Gebauer (10:59)
Yeah.

Phillip Greer (11:13)
So first off, I refer to this as the E-I-E-I-O because it is 100 percent that effect. The Old MacDonald version.

Greg Lambert (11:18)
Yeah, there you go. The Old MacDonald version.

Phillip Greer (11:24)
For a new world. Yeah, so I think the goal is not only to have longer content, but it’s authentic content that speaks to your messaging.

Greg Lambert (11:31)
Okay.

Phillip Greer (11:31)
So you can create a lot of long-form content with AI, but the problem, if it’s not authentic to your brand and your expertise, you’re going to lose a lot of credibility.

So, thinking about what is it that makes your lawyer special? What makes them stand out? How do you speak on it? And how do you keep that authentic?

And sometimes it’s not directly in the world of law for the authenticity to stand out. So figuring out what your niche is there and then driving it home. That is what is going to create unique content that gives you that authority and will stand out among these GEO initiatives because they can say, this is not a whitewashed, white-labeled version of the same type of content that continues to be posted online. It’s something unique.

And if you have a series of this type of content, it can create that narrative of, okay, this is the type of lawyer, this is the type of firm, this is the type of credibility, this is the type of messaging that you can expect, and stand out among your peers. So yeah, I definitely agree with the longer-form opportunities there.

Marlene Gebauer (12:39)
So Elizabeth, for law firms trying to adapt to this new AI search environment, creating unique, machine-readable content is a difficult hurdle. Best Lawyers recently launched Smithy AI to help lawyers and marketers scale their profile content. How does this tool solve the duplicate-content penalty while keeping human editorial control firmly in the loop?

Elizabeth Petit (13:05)
Yeah. So first of all, we’ve always viewed ourselves as partners with lawyers and law firms. So the products that we build and deliver, we want to solve your problems. And one of the problems we know that you have is too many things to do and not enough time to do them.

So one of the things that Smithy AI was developed to address is getting lawyers to fill out their profiles.

Have either of you gotten lawyers to fill out their profiles? That’s a tall order.

Greg Lambert (13:35)
Yeah, all the time. All the time.

Phillip Greer (13:37)
Yeah.

Elizabeth Petit (13:41)
So the easiest thing, the old way to do this, the easiest thing to do is that most teams would copy and paste whatever you finally got written on your own website. You’re going to go copy and paste it everywhere else on the internet. And you got penalized for that.

So we wanted to, again, partner with law firms and develop an easier way to assist you with your workload.

So what Smithy AI does is it reads our structured data and the content you’ve already developed on your lawyer profiles on your website, and it makes a recommendation for you. Now, everybody knows that AI is not a replacement, it is an assistant, and you should still use your human judgment.

So it does say, recommended generative content. It pulls in information about the awards, things that it thinks are appropriate. You review it, and then you publish it with the click of a button.

The name for Smithy AI does come from one of our original co-founders, Gregory White Smith, who was a Pulitzer Prize-winning biographer.

And so we also wanted to name it in his honor for both founding Best Lawyers and as a writer who was a craftsman as well. So Smithy is developed to help you with your mountain of work, both with profiles, but also there are features now to help with press releases, email announcements, social assets. So we’re trying to be your extra hands when you’re looking at the clock, looking at your watch, looking at your calendar, and you don’t have enough hours in the day.

Greg Lambert (15:18)
I’m curious when it comes to that because I know, for example, one of the things that got pushed out on platforms like Substack is there’s kind of that AI content identifier.

Phillip Greer (15:34)
Mm-hmm.

Greg Lambert (15:35)
And so is that one of the things that you try to look for with using a tool like Smithy, giving them something to work with, but then, is it a full copy and paste, or is it something that they take and then they kind of do their own smithing on it as well? How do you encourage your clients to use the tool?

Elizabeth Petit (16:00)
Yep, so we envision it to get you like 90 percent of the way. So it’s not a copy and paste. It doesn’t pull exactly what you’ve already written. You can also hit that button a few times and say, slightly rewrite it, slightly rewrite it, polish it for me. So it can iterate a little bit for you.

But as with all wordsmithing through AI that we all do, we all know we do this, you should never take whatever it spits out for you and then publish that. This is where human judgment always needs to be that final set of eyes, that final check before you are comfortable publishing to your website, our website, or anywhere that you publish content.

Phillip Greer (16:44)
Yeah, and I would say also, going back to your point about the long-form content and where I was going with brand authenticity. So one of the great things about Smithy is, as Elizabeth is saying, we’re going to get the way you’re talking about yourself on your profiles, and we’re going to take the data we have on Best Lawyers and put it through our own system to say, how do we think it should build content?

So it’s going to have your unique voice with our data, but you still want to ask yourself, is this me?

If not, you do have the ability to be replacing it with another version of yourself. And I know with AI, it’s a whole different topic where there’s a lot of, what does it mean to clone a lawyer? Well, we don’t want you to clone your brand. We want you to play a role in helping that editing of the brand.

I think back to college, the hardest thing was always to start writing the paper. The first paragraph is always the hardest. The thing about Smithy is it’ll write the first four paragraphs for you, but you should come in with that closer. You should come in with that closer and do a little bit of editing on the first four that it produces for you.

Greg Lambert (17:52)
Phillip, I want to turn to something that you’ve done that I think we’ve seen a few CEOs do, and that was there at Best Lawyers, you kind of famously mandated that your entire team continue doing their job, but also look at becoming essentially software builders.

Phillip Greer (18:13)
Mm.

Greg Lambert (18:14)
Now you’ve got, and I think the last estimate was about 30 percent of the company, vibe coding their own apps. And Elizabeth, I think you’re a prime example of this. You’re a researcher, not an engineer, yet you were involved in building a real-time KPI dashboard there at Best Lawyers.

So it seems like people are taking you up on your challenge, Phillip. So I want you to walk us through something that you’ve built that takes the worst part of your job off the plate. And then, once the repetitive data wrangling is automated, how do managers shift their focus from doing that kind of grunt work to focusing on more of the higher creative work?

Because that’s something that we’re insisting our lawyers do. So I’m hoping you’ve got the secret sauce and can help us on that.

Phillip Greer (19:20)
Yeah, I’ll start here, and then I’ll want to pass it over to Elizabeth. But I’ll start.

So the end of last year, I’ve been playing with a lot of AI and product development for years now. My background is engineering. I used to build a lot of software. I built a lot of the core systems we still use here at Best Lawyers for producing our rankings and doing our survey processes.

And then I’ve been the CEO and running it for quite some time. And when AI came on the scene, I started experimenting. What can we do? What can we do?

And about December of last year, I had the breakthrough. Claude Opus came out, and its coding abilities were far superior to every other AI coding that had existed before.

So I had this wild thought where, what if I build an environment that was safe, that our team could fail fast in and not worry about hurting our data, and it wouldn’t go and be spread out to the outside world? It wouldn’t be trained on outside systems. We wouldn’t have to worry about them doing some shadow IT type environment where they’re taking sensitive data and going straight to ChatGPT and dumping it out there.

So I felt an obligation to create that environment. And it’s an exciting time because it felt possible. So I spent all of the holiday break in December and then all of January building out, first, an environment.

And what I mean by that is a place where all our data connects. So I needed to create this API layer so that all of our data, when I say our data, I mean our SQL data, our accounting data, our HubSpot, Gong, Google Analytics, any type of systems that we are accessing and using on a daily basis, I wanted it all to come to one place and then inform itself and know about its own data.

So I created a data lake, if you will, custom. And ultimately, I don’t know, for those who are a little more technical on these calls, it’s a deep coverage model where the data in Gong understands how it relates to our data inside of HubSpot, and that understands how it relates to our data inside of Google Analytics, and so on.

So when you’re asking questions, it references each of the data sets, and it knows what a firm is or a lawyer is inside of those areas.

That’s a complicated way of saying I made a place where all our data came together safely and made sure it knew how it was associated.

And then I needed to create that safe environment. So I built a system called the Best Lawyers App Store, where you could go in and start vibe coding through our Claude environment.

And when you vibe code, you’re safely in that App Store. It’s using only our data sets, and you can build, eventually, what we now are using as an operating layer for most of our business.

So I did this. I put all this together. I got it in a prototyped form, and I met with the team at the end of January and I said, “All right, all right, I got an idea. Everyone’s going to become a vibe coder. Every department’s going to be a builder. Elizabeth is going to be a builder. Nancy is going to be a builder.”

I went full Oprah Winfrey on, you’re going to, you’re going to, you’re going to.

Greg Lambert (22:25)
Yeah.

Phillip Greer (22:26)
And they were all on board. I was like, okay, no convincing needed, you’re totally in for this.

Elizabeth was one of the first people I sat down with. I think I spent all of 30 minutes with you, Elizabeth. It was a short session. I got you set up in the environment. I gave you the tools, told you how the data works and how to connect it, and I said, go and dream and build.

And Elizabeth, I’d love to hear your take on it. That’s my more technical explanation of what happened. I’d love to hear what your side was.

Elizabeth Petit (22:56)
Yeah.

Phillip Greer (22:56)
I’d love to hear what your side was.

Elizabeth Petit (22:58)
Yeah. So I knew Phil had been working on this passion project. Phil’s always got multiple passion projects going. So I knew he’d been working on this, and then he made a statement and he’s like, “I’m going to ask you guys to come join me in this.”

And then he did that ping and he’s like, “No, I mean it.”

And I’m like, “Yeah, I know. I’m here. I’m ready.”

So my personal mantra has always been, my team knows this, work smarter, not harder. And that doesn’t mean we don’t work hard. I love to work hard. But it means work more efficiently. If there’s a better way to solve your problem, do it the more productive way. Don’t do something the same way over and over again because you’ve always done it that way.

So what vibe coding unlocked for me was this new ability to work smarter and not harder, and take some of the repetitive, mundane parts of everybody’s job and make them more fun, make them more creative, automate them.

Whether it’s through KPI reporting that eats up time every day, or taking some of these important projects or problems we want to solve that sit on that shelf, that you know have value, but you never have the time to take off the shelf.

And say, okay, now that I have optimized some of this basic operational work, I’m not spending all day, two days of the week, pulling together my KPI reports, let’s take a bigger, meatier project off the shelf and let’s go start working on that.

Phillip Greer (24:38)
Yeah.

It’s been exciting watching how everybody came to the table to build.

Elizabeth built research dashboards to monitor the process for Best Lawyers as the nominations were coming in and voting was happening. Nancy, our head of content, developed expense-paid tracking systems for all of our budgets that were going out for paid media, as well as her marketing team developed this marketing dashboard that pulled in our Google Analytics and gave full-funnel conversion visibility.

I mean, we went from, like a lot of companies, working heavily in Excel and Power BI or Tableau. We’ve stopped using Power BI. We don’t use any Tableau. We do everything now inside of our dashboards that are developed independently by departments.

And the only time we’re in Excel these days is when we’re exporting data sets to give to our business partners or if we’ve got to send them out somewhere. But we work primarily now inside of our BL App Store environments.

When you’re asking for data and looking for new trends and analysis, it used to be, okay, we’ll export everything out to Excel. We’ll write a bunch of complicated formulas. I’m Googling how to write that complicated formula.

Greg Lambert (25:55)
Yeah.

Phillip Greer (25:55)
My God, why do we have to write these complicated formulas?

Greg Lambert (25:58)
Yeah. How do I do that VLOOKUP table again?

Phillip Greer (26:00)
Yeah, that VLOOKUP table.

Elizabeth Petit (26:01)
Yeah.

Phillip Greer (26:03)
My God. And then you always have the one person in the office who’s your Excel person, and that’s what their life is all about. “Ask me about my sheets,” you know?

But now everyone goes straight into their environment. The data’s all there in our data layer. It’s trained, it’s learned, it knows itself in the deep coverage model.

And you work through our App Store and ask it questions. And instead of developing an Excel sheet, you develop an entire new page that is the best visual representation of your data and answers a question. And you use that to start making decisions.

So our speed to decision is pretty great. A week-long, two-week-long Excel project is now done. We have an idea in a 9:30 meeting. By 10:30, we have a new page that’s been pumped out through our finance team that’s explaining the answer.

And then we need to make that next decision and, okay, how do we execute on this data?

Greg Lambert (27:02)
The follow-up I have on that, and this is something I’ve been writing on, especially since you have so much data that you’re working with, and it sounds like you’ve solved part of the problem through creating these data lakes or this ability to feed information into the right places so that you set up your AI to access clean information in a relatively simplistic way rather than having to go out to 15 different places, some external, some internal. You’ve got it all built.

So the topic that I’ve been writing on a lot, and I think this is going to be, I thought it was going to be 2027 fodder, but it might hit before then, is a lot of us are building these harnesses around both our data and our AI tools with the expectation of a lot of this information.

You don’t need to dump everything into the LLM and get an answer. You need it to be precise, and you need it to be consistently right so that the inputs and the outputs are accurate.

So I know, Phillip, you designed the initial, you vibe coded the system around it. Do you have a technical team now that keeps that up to date, updates or adds new things that come in, or are you still able to use the talent that you have on staff that they build on top of what’s already built?

How do you take that 9:30 question and turn it into a 10:30 report?

Phillip Greer (28:47)
Yeah, so I built all the environment for the initial App Store, and my intention was always to hand off the day-to-day maintenance because adding more data to our data lake or data API library is going to be a constant workflow.

And any company who’s working with any type of data environment, governance, or what have you, whether it’s SharePoint or an actual Azure data lake, et cetera, you’re going to have to have somebody managing that.

So once I got everything built, the initial systems, we found one person internally who had the most interest in the AI infrastructure and the future of AI and how it’s used in programming. He was one of our programmers, and his name’s Seth.

And so I trained him for a week on the system, and he took it over. And now he is running that API library.

And people, all the vibe coders, as they need more data or they need more feature abilities in the App Store, they go through Seth. And it’s been a good system.

Because my original vision was, I knew what Elizabeth and I knew what our research and our finance and our content team would need to get started, but I didn’t know what they were going to build, right? And that’s the wild thing, what they end up building and how they use the data.

I’m watching some of the requests come through and some of the systems they’re building, and I know because I designed the actual data that came together, and I would go, that’s not going to work. They don’t have that data. That’s not going to work. They don’t have that data.

So I had that moment where I was like, I need to hand this off to someone. I’m the CEO of this company. I can’t keep adding to that API layer.

Greg Lambert (30:28)
You don’t want to be the head vibe coder and CEO?

Phillip Greer (30:31)
Head vibe coder and CEO.

So I got Seth involved, and now whenever they start to build with more data, he continues to extend the actual API layer.

But what is great is the actual building is still happening from the non-builders. Well, they used to be non-builders. I’d call them all builders now.

That 30 percent number, as of, I onboarded a few more people this last week, we’re up to 40 percent of our company who are vibe coders now. It’s growing because the mentality is, if you want to do this, we want to help you do it. There’s no reason you have to sit on the sidelines.

So I handed that over, and then what I realized quickly by handing this over is there’s an opportunity to build more things.

So that’s been kind of exciting. And we can get into it. But I built out what we lovingly call here Bestie, which is our first type of AI support team member that uses our different models and data sets.

So yeah, you have to have someone who’s going to own it. You’ve got to have someone who’s going to make sure that they’re adding to it. But that’s the easy part.

The harder part is thinking about what you want to build. Because when you can build endlessly, narrowing that scope down to building things that are functionally helpful is that next muscle you have to start learning how to train.

Greg Lambert (31:55)
Yeah, it’s not asking what do I build, it’s what should I build, right?

Phillip Greer (32:01)
Yeah. One of the opening things you mentioned was about, how do you open up room and bandwidth for creativity?

Well, you have to allow yourself to be creative. And there’s a lot of people who work in an office environment that have not worked that muscle in a long time because you make that Excel report, you put it through Power BI, you drop it out as a PDF, and then you take it to the board. And you do it again, and you give the lawyer the thing.

The creativity is lacking.

And now all of a sudden, all of that administrative burden is done within 9:30, the idea, 10:30, the actual result. Now what’s the creative solution that you move forward and make something functional to execute on?

Marlene Gebauer (32:43)
So transforming a workforce as quickly as you have doesn’t happen without some friction. Phillip, you’ve spoken about the AI vampire effect causing extreme decision fatigue.

Phillip Greer (32:52)
Mm-hmm.

Marlene Gebauer (32:53)
And Elizabeth, you’ve had to manage an existential crisis among your software engineers. How did you help these technical engineers transition to feeling like they’re authoritative system architects rather than feeling displaced?

And how are you managing that cognitive burnout when AI offers answers in seconds instead of in weeks?

Elizabeth Petit (33:20)
Yeah. So first of all, the existential crisis is understandable.

These team members of ours, many of whom Phil and I have worked with for over 10 years, went to school to be experts in these fields. They are craftsmen. They are builders. This is their superpower.

And then seemingly overnight, IKEA came to town, and no longer is the craftsman superpower the same. And that’s hard. And I understand that, and we did understand that.

And it helped that Phil was a developer and came from that background, so he understood that as well.

And it was always our goal, continues to be our goal, to use AI to upskill our team members as an opportunity for professional development, whether you are an engineer or another member of the business team. It is not to replace anybody’s job.

And so while the business team is learning a superpower they never thought they would have, they never thought that they were going to be this type of craftsman, and they’re not going to be this technical craftsman, they do some things.

What it has done for our engineers is it has allowed them to also build faster, solve more complex problems. All the things that AI does. It allows for better debugging, it allows for better regression testing, it removes some of that mundane work that they were doing.

So we went into this as, we are all going to learn this together. This is new emerging technology. We don’t have all the answers, and we want to figure out how this makes the most sense for you and your roles, and us within our business.

Phillip Greer (35:16)
Yeah. At the end of January, at the same time I was meeting with the executive team, Elizabeth pulled all of our engineers together in an in-person meeting, and we had a Q1 huddle.

And I sat there and explained to them that I’ve built this App Store. I’ve got this huge data layer. All our data comes together nice and clean. It’s learning off of each other. And now everyone’s going to become builders.

And it was a lot of disbelief because, again, craftsmen, like, okay, I make quality code. There’s no way a machine’s going to do it.

And so there was a lot of disbelief.

After that, I met with Elizabeth and team and taught them how to vibe code, and they started building apps.

We brought everyone back together again and showed them what these non-builders were putting together and how it was pulling data from all of our systems.

And then even our engineers were in the mindset of, my God, this is a real thing. This is not some fancy fad happening, Phil says everyone’s going to be doing this. They’re building financial systems. They’re building research systems, marketing systems, sales systems.

And it was clear that they wanted to be part of it too.

And so Elizabeth put on, I thought it was an intelligent thing, as we had the adoption, every two weeks we’d have sessions where we’d encourage the engineers to show, in a safe space, the other engineers what they’d been vibe coding.

Because there’s also this stigma, like, I’m an engineer. I don’t vibe code. I know how to code.

But that’s not what vibe coding is. Vibe coding is not a replacement for, I don’t know, a good analogy. Here’s a good analogy.

I play guitar. But when someone says, “I play Guitar Hero,” and they’re pushing those buttons on the little plastic machines,

Marlene Gebauer (37:02)
Ha ha ha.

Phillip Greer (37:03)
I would go, “Ha, you don’t play guitar. I play.”

It’s not that. It’s a completely different thing. It’s not analogous to Guitar Hero versus playing guitar.

Both people are coding. Both people are building. It’s only one person’s doing it craftsman style, line by line. The other person is effectively doing it through a learned generative system that knows how to put all the pieces together, and you get to be that creative fountainhead.

So watching them come along the journey, it took a little while. We had to walk with them. We had to have these sessions where they would show what’s possible.

And the way it’s aha moments. After you start to realize, I made my day a little less boring, or I removed this ridiculous, redundant task that I have to do over and over, you start to go, now what do I do?

So that’s been a fun thing to watch. And I’m using the word fun because everything’s hindsight. In the moment, it was like, come on, guys, you don’t understand. I promise we’re not playing Guitar Hero. This is real. We’re producing real things.

Greg Lambert (38:06)
Yeah. I can’t imagine the eye strain from the eye rolling from your engineers when you first announced this.

Phillip Greer (38:16)
Yeah. Yeah, it was significant.

Marlene Gebauer (38:16)
Ha ha ha.

Greg Lambert (38:20)
But I’m wondering, Phillip, how do you leverage the intelligence and the experience of your engineers? Is there anything that they help people like Elizabeth better understand about structure and what the end results are and how you want to get there?

How do you leverage that expertise to help the non-engineers?

Phillip Greer (38:50)
So Elizabeth made the comment that our goal was not to reduce our engineering staff in this process. We wanted to support and invest more.

And they ultimately have, to the point you’re making, this understanding of our data.

No matter how much we bring it together and add to our API and have it learning on itself, they understand the repository, where it comes from, how it was put there, how it’s designed, the schema markup, how you scale it, and when you’re having more and more people access it, how much you pull on those data sets.

I’ll give you an example.

One person was writing an analytics system as a builder, and they were saying, give me all of our analytics in our BL App Store. We’re talking about hundreds of millions of data points, and they couldn’t get their app to do it. It was crashing constantly.

And so we had to have one of our architects come in and go, let’s walk through this.

And what they built, you’re using the word harness or harnesses earlier, or guardrails, they decided to change our API layer to understand that the user doesn’t understand the data when they say, give me all of our analytical data, give me all of our financial data.

So it’s my job to create ways that I feed it to them in an appropriate way.

Because when you have access to all of this and you don’t know how it’s formed, functioned, and put together, you create a lot of overly inefficient, heavy systems.

So what we’ve been doing is we’ve opened up the doorway and the communication to say, go talk initially to Seth. He’s in charge of the App Store. And then from there, you spend some time meeting with one of the engineers and asking questions about, how do I make my app do something a little faster? It’s getting stuck here.

And they walk you through their why and the how.

Because what they’re doing is they’re not only in charge of where the data is coming from, they understand it, and they’re building better ways to feed it.

So we had a big session with our engineers where we’re saying, how do we feed things even faster to our API layer?

So we had an architectural meeting where we’re trying to understand, if things are coming from SQL and those are our bottlenecks, it’s single-server environments, how do we extend that further? Let’s have a stopgap where it feeds into a system like Elasticsearch.

I’m so sorry for all the technicality here. Let’s have it feeding to a faster system.

Greg Lambert (41:11)
I think a lot of people that listen to this have heard Elasticsearch before.

Phillip Greer (41:16)
Yeah, yeah, yeah.

So we decided to move our API model from SQL to Elasticsearch, so the first read comes from a faster environment.

And these are the new types of things that our programmers get to think about. No longer are they creating forms on an intranet to feed data back to an end user in an Excel environment.

Now our engineers are freed up thinking, okay, if I restructure my data and take it from a single-server environment to a multi-server replica environment where I feed it faster, what do we accomplish?

How do I create a harness that feeds this data in a way that has nice guardrails, and they don’t have to think about ruining the data, over-encumbering the server? We’re not going to have server crashes.

And those are the projects.

Engineers always talk about, Elizabeth made the comment, taking that hard project off the shelf. That’s what the engineers want to do. They don’t want to have to write one more page on the intranet. They don’t want to write one more page that tells you what your voting reports are.

They will do it because that’s their job.

What they want to work on is making our server faster, building a better backend system, creating a new way to isolate our data and feed it in a multi-tenant scenario.

So that’s when their aha moments and their expertise start to be able to work on those types of systems.

For the first time, those are the types of things where we go, okay, you get the last two weeks of December. This is what you get. Build whatever you want in two weeks.

And now you have more opportunity to start thinking about that longer-term pipeline because each department, they’re servicing their own needs with the data.

Greg Lambert (42:59)
That sounds good, and I think it’s something that we all kind of face for how we transition into whatever the new version of, whether you’re a manager or an engineer, it seems like we’re all taking on more.

And I’ve noticed I’ve not seen people go home earlier on Fridays for some reason.

Phillip Greer (43:23)
Yeah.

Greg Lambert (43:24)
So I’m still waiting for my, it’s time for me to be rolling in income and sitting on the beach the whole time. So perhaps that’s a 2028 goal.

Elizabeth Petit (43:33)
Well,

Greg Lambert (43:34)
2028 goal.

Elizabeth Petit (43:35)
I think one of the reasons for that is because you move so much faster with AI, and specifically with vibe coding, it has unlocked all of this creativity and opportunity that, for those of you who’ve done it, is quite addictive.

Greg Lambert (43:55)
Yeah.

Elizabeth Petit (43:55)
Once you get in there and you start building, whatever you are, you’re prototyping something, you’re building something you’ve been wanting for years.

That used to be this cumbersome process of meetings, requirements, design, getting in the development queue. Three years later, you get something you asked for that you don’t need anymore.

You now all of a sudden get it quickly.

Okay, that’s great. But also, the feedback loop comes immediately. So instead of Friday morning you send off your information and think, I’ll get something on Monday to address, it comes right back to you.

Phillip Greer (44:32)
Mm.

Elizabeth Petit (44:33)
So now we have this cognitive fatigue concern because you are engaging so rapidly that the machine will never get tired. It will never go to dinner with its family. It will never log off for the weekend.

So at a certain point, you have to have the critical judgment to say, I am exhausted. You will never be exhausted, but I am.

And Phil shared this story that he found himself at one point being so in the go, and Phil and I are the same in that way. Once we get on that path, we are running.

And then he realized that he was not making the same little decisions that he would make. And not big, impactful things, but things that usually he would make a decision about.

He had that aha moment that he was having cognitive fatigue. And you don’t want AI to make those decisions for you.

So that’s when,

Phillip Greer (45:33)
Yeah.

Elizabeth Petit (45:34)
you have to remind yourself, AI will never say, go spend time with your kids, you know, go home early on a Friday.

Marlene Gebauer (45:41)
Not unless you

Phillip Greer (45:42)
Yeah.

Marlene Gebauer (45:42)
create an agent to remind you.

Greg Lambert (45:44)
Yeah, exactly.

Elizabeth Petit (45:44)
Yeah, exactly.

Phillip Greer (45:45)
Yeah.

Greg Lambert (45:45)
Exactly.

Phillip Greer (45:46)
And that runs into, you mentioned earlier, the AI vampire. That’s what it is. It’s feeding off of you, you’re feeding off of it, and you have to recognize, okay, I get to walk out in the sunlight. So I need to go do that some.

Marlene Gebauer (46:01)
Tear my eyes away from the screen, yes.

Greg Lambert (46:04)
Yeah, well, I always say when somebody asks me what’s the addictive game that I’m playing on my phone, I say it’s my $200-a-month Claude account. I’m on it all the time.

Phillip Greer (46:15)
Yeah, yeah.

Elizabeth Petit (46:16)
Yeah. Yeah.

Greg Lambert (46:18)
So yeah, you’re right.

Elizabeth, before we start wrapping up, I know that AI, of course, isn’t only affecting how we manage engineers, how we manage managers, and how we all become vibe coders. It’s becoming a part of Best Lawyers content as well.

And one of the things that you oversaw was the introduction of Artificial Intelligence Law as a formal recognition category in the 2026 edition.

Do you mind talking about what that encompasses, how you came up with it, and what you found?

Elizabeth Petit (47:00)
Yep. So, you know, it is always our goal to help connect clients in need of legal services, whether that’s professional or personal, with the right attorney for their case.

And so we always keep our eyes and ears open for developments in the legal industry to add new practice areas as they emerge over time.

So cannabis law, that was one that we added a few years ago because that was bubbling up.

And so about three years ago, we started to see that law firms were adding AI practice groups. And so we added this and started researching AI as its own practice area.

Now, when this first started, I envisioned this would be sort of a subset of emerging tech.

And we have technology law, and that this would kind of,

Greg Lambert (47:51)
This is what the blockchain lawyers morphed into, right?

Elizabeth Petit (47:54)
Yeah, exactly. This is sort of what I thought this would evolve into.

Phillip Greer (47:54)
Yeah. Yeah.

Marlene Gebauer (47:54)
Yeah.

Elizabeth Petit (47:57)
But as AI is so dynamic and evolving so quickly, the definition of an AI lawyer is quite nebulous.

Are you counseling AI corporations? Are you focused on data privacy? Are you looking at government issues? Even labor and employment. I mean, AI is truly touching everything.

And so because of that, it started off more as a subset of technology. But we will have to evolve with AI in our awards as this industry evolves so rapidly to make sure that it is reflective of how legal is addressing AI and the needs of clients.

Greg Lambert (48:49)
Are you thinking about

Phillip Greer (48:49)
We didn’t even get it.

Greg Lambert (48:52)
an AI-native law firm category? That seems to be the big buzz phrase these days.

Elizabeth Petit (49:01)
Yeah. We are getting ready to evaluate adding new practice areas for next year. So look for that webinar series coming up soon.

Phillip Greer (49:10)
Yeah, and whether or not we have AI law firms that we recognize, that’s currently not on the horizon.

But it is the question, when do I have my work done by AI and AI Inc.?

Greg Lambert (49:27)
Yes.

Marlene Gebauer (49:27)
Ha ha.

Greg Lambert (49:29)
All right. Well, before we get to our crystal ball question and we start looking at the future, I want to ask you guys to look back in the past. And I think I know where this is going based on our conversation.

Phillip and Elizabeth, what’s something that’s true today that wasn’t necessarily true for you and Best Lawyers a year ago? What’s changed in that year?

Phillip Greer (49:52)
Yeah.

Greg Lambert (49:52)
What’s changed in that year?

Phillip Greer (49:53)
I mean, the biggest thing, it’s everything we talked about, that we have non-builders, and they’re taking their data and they’re solving their own problems.

We have a decent engineering team as far as size because our company is research and tech. We use a lot of tech to build rankings and recognitions in over 76 countries.

And so we always built a lot of custom software, and we’ve always been able to do dynamic things for each department. But there was always a process. You got in line, got queued up, and then it handed off to the engineering team.

Greg Lambert (50:28)
Q3. Q3.

Phillip Greer (50:30)
And now that’s changed greatly because everybody is handling the majority of their own needs internally.

We’re still externally using our engineers. We’re not developing any software. All software that’s hitting our customers is still handled by our engineering team because that’s the right way to handle it.

But for all our internal needs, everyone’s a builder. So that was a big change I’ve seen. That is different now.

What about you, Elizabeth?

Elizabeth Petit (50:59)
I think the surprise that has come out of this is, I’ve been with Best Lawyers for over 15 years now, and I did not expect this career development opportunity.

Phillip Greer (51:14)
Mm.

Elizabeth Petit (51:15)
So I thought that I would continue to grow in leadership or other traditional career development paths, but all of a sudden to have an entire skill set that I did not have last year, that’s pretty amazing.

And I’m grateful to have been given that opportunity.

Marlene Gebauer (51:37)
All right, we’ve looked a little bit at the past, and now it’s time to look in the future.

If you would gaze into your crystal ball and look ahead a few years, as autonomous agents and vibe coding become standard inside both data organizations and law firms, what’s the single biggest shift you see coming for the legal profession’s economic model, specifically regarding the billable hour?

Phillip Greer (52:03)
Hmm. So I don’t think the billable hour is going to disappear overnight, right? But it will shift.

People are going to have more scrutiny of how you’re backing up the value of what’s coming from the work product.

And there’ll be more questions about how did you derive that actual result set.

It was already the question, you hire the partner and you ask yourself, am I getting the partner’s hours or am I getting the junior associate? That’s already been on the table for a long time. And some clients have been wanting to know that. And they want to know, I am paying for the partner’s time.

I think there’s going to be a lot of questions that more clients are going to be asking. I want you to break down the justification. Am I paying for the senior partner’s time, or how much am I paying for the AI’s time?

I don’t know what that’s going to look like as far as how it wants to be shown in transparency, but I think that’s going to be a big question.

Marlene Gebauer (53:01)
How about Elizabeth?

Elizabeth Petit (53:04)
I agree. I don’t think that this is, legal doesn’t revolutionize overnight. We’ve all been in this industry for a minute.

But a lawyer is not going to become twice as productive with AI. They’re not going to bill twice as many hours a day. That’s not possible.

But they’re expected to still bring in that same amount of revenue for the firm.

And so figuring out that balance between what is the appropriate use for AI, but still what requires that human judgment that AI will never replace.

And that’s ultimately what a client comes to a law firm for, not the administrative work that AI assists with. It is the human judgment.

That’s where I feel like, if firms tell their story to clients successfully, they sort of balance that out, get that efficiency that they need, but still have the confidence of their clients.

Greg Lambert (54:04)
Mm-hmm. All right. Well, I think that’s a good spot to end on.

So Phillip Greer and Elizabeth Petit, thank you so much for joining us and breaking down what it looks like behind the walls there at Best Lawyers. Appreciate it.

Marlene Gebauer (54:18)
Yeah, thank you.

Phillip Greer (54:18)
Yeah, thank you so much for having us.

Elizabeth Petit (54:19)
Thank you for having us.

Marlene Gebauer (54:21)
And thanks to all of you for listening to The Geek in Review. If you’ve enjoyed the show, please share it with a colleague. We’d love to hear from you on LinkedIn, YouTube, and Substack.

Greg Lambert (54:30)
And Phillip and Elizabeth, where’s the best place for listeners to learn more about you, more about Smithy AI, Best Lawyers, Bestie, all those things?

Phillip Greer (54:40)
So first off, for more information, go to bestlawyers.com. Also look us up on LinkedIn.

I’m posting a lot of videos about some of the AI work that I’m doing, some of the apps we’ve been building. And yeah, feel free to follow us there and check out what we’re doing.

Marlene Gebauer (54:56)
And as always, the music you hear is from Jerry David DeCicca. Thank you, Jerry, and goodbye, everybody.

By Greg Lambert

Earlier this month, Harvey’s research group released a data set that I’d been hoping someone in the industry would do for a long time now. They created an open-sourced version of a synthetic (fake) law firm.

The firm is called Calderwood & Harkness. I hoped this was a reference to Agatha Harkness from Scarlet Witch of Marvel Comics fame, but I was told by the folks at Harvey that it was not. And that I was a nerd. Regardless, the data was pretty nerdy as well, and in good ways.

The file structure was set up with 46 clients, 15 different practice areas, with 266 different matters, and a total of 9,288 documents. There were engagement letters, closing memos, due diligence summaries, and email communications between lawyers on who was covering what parts of the matters. It was all synthetic. So, none of it touched any firm’s real client data, and it was all made public on GitHub under an MIT license. In addition to this, they published 250 test questions to use on the data. These are questions that you’d typically see lawyers asking their document management systems. They also provided the answer key to the questions.

I immediately downloaded the data and started playing around with it.

The very first thing that jumped out at me was it was super clean data. And for anyone who has spent any time looking at law firm data, the one thing that we’d all agree on, is that our data is typically very dirty.

Fake Firm... Fake Tidy Data

A large majority of the data, some 8,055 files, were all MS Word. The 615 emails were all saved as .eml files. Excel spreadsheets made up 573 files, and there were 45 PowerPoint (.pptx) slide decks in the mix as well. The format mix wasn’t terrible, but it wasn’t very accurate either.

There were exactly zero PDFs. I don’t know about your firm, but mine loves PDFs. Nice clean, born digital versions, scanned and OCR’d versions, and those lovely scanned, non-OCR’d versions or the scanned at an angle PDF copy.

The other issue I spotted immediately was that the filenames were structured so perfectly, it would make a Records Manager weep with joy. The files were all named in lowercase type with hyphens between each word. With filenames like closing-checklist.xlsx or willful-infringement-memo.docx, I couldn’t find any exceptions to this naming convention anywhere in the entire data set.

I checked my own litigation matters in the DMS and did a sampling of fourteen workspaces and calculated that between 60 and 80 percent of the files were emails, by count. The C&H law firm count was close to 7 percent. The number of documents per matter was also much smaller than what I found in real matters. There were roughly 35 documents per matter in the C&H data, and in my small sampling size against real matters, none of those came close to only having 35 documents.

I pulled my hypothesis together, and I think that many of us would also come up with a similar theory. The AI search tools would test well with this type of clean synthetic data, but it would struggle. In other words, messy data like scans, duplicates, different formats and sheer noise just doesn’t work as well as clean data.

It seemed like a good theory to test. So, I tested it.

What I built and tested

I decided to build two additional versions, where each version changed only one thing, that way if the scoring moved, you’d know exactly what moved it.

I created a C&H-Enhanced. This took the original 9,288 files and added realistic metadata to each of the files, this was derived from that document’s own body text. That would include authors, create dates, and modified dates. These are data points that a DMS would pick up and expose for searches. I left the document text alone and it is a byte for byte duplicate of the original document. This allowed for the original 250 test question set and the corresponding answer keys to stay valid.

The second set is C&H-R, for realistic. I pushed this set a little further. In order to get more realistic numbers for PDFs, I converted 2,819 of the Word files to PDF. To continue the realism tactic, I made sure there were a variety of PDF types. I made 2,114 as born digital PDFs, where the originals are printed as PDFs from MS Word. There were 564 scanned PDFs that were rendered at 150 dpi in grayscale, with skew and noise added and a tesseract OCR layer underneath. That caused the output to carry copier producer strings in the metadata instead of the actual author names. And finally, there are 141 image only PDFs with no text layer. This was what I thought would be the hardest part of the data set.

I kept drafts, redlines, and templates as Word files. Seventy-five documents were kept as both Word and PDFs so that I could test how a system handles duplicate files. In the end, 30.1 percent of the data set came out as PDFs, which comes close to what I would see in the real world.

None of this was as easy as I was hoping. When I reviewed the original PDFs, they had timestamps that had been written with UTC digits under a fixed offset label. This was incorrect and I had to fix those with local time and daylight savings handled correctly. I then discovered that a chunked copy had nested a complete duplicate matter folder inside of itself, so there ended up being 150 additional files that needed to be removed.

Testing

Out of the 250 questions that were in the benchmarks, I used a sample of 30 of them with a fixed random seed. This will allow anybody to reproduce the same sample. The 30 questions create 369 individual grading checkpoints. I use this to conduct the actual scoring. The checkpoint is set up as a small yes or no answer to things like “the answer identifies the Meridian acquisition” or “the answer does not include matters outside the qualifying set.”

The questions went into Harvey exactly as they were published. There was no rephrasing, no hints about the answer key, and limited to one question per query. The process then took a fresh grader that had never seen any of the other runs, would grade every checkpoint against the published rubric.

The 30 questions were processed three times on each of the three data sets. So there were nine complete passes, along with nine independent graders, over 270 queries.

Here are the results.

The original C&H data set scored a median of 132 checkpoints out of 369. The runs ranged from 115 to 139. The set of metadata enriched files scored 130, with a range of 128 to 132. The C&H-R formatted version with all the PDFs and scans and image only pages scored 126, with a range of 113 to 127.

So after all the testing, it turned out that the score from best to worst, was only six out of 369.

The theory did not survive

My theory that drove my decision to build two new versions of the data, did not survive the tests.

Nothing we added, whether metadata, PDF conversions, scanning, OCR files, or image only files, did anything to move the initial results.

I checked with the three graders on how they evaluated the PDF versions, and they reported back no OCR damage anywhere across the 90 answers. There were no cases of garbled text or illegible pages, and none of the results produced an issue where the graders could not read the files.

The realistic document set is not the reason behind why these tools missed things. The ceiling on results show up the same whether it is clean Word files, metadata rich files, and on scanned PDFs.

This was not the answer I was expecting. But with most experiments where the results are not what were expected, it can point you to something outside your original hypothesis.

What I actually found

The results of the tests were solidly proving me wrong about the formats, but it was showing me something else that I almost overlooked.

For each of the datasets I ran the same 30 questions three times. Every setting on each of the runs was identical. The same data set, the same model, the same mode, everything in the tests were the same. The totals came back with a small, and acceptable spread of 139, 132, and 115.

The actual differences didn’t show up until I dug in a little deeper on each of the checkpoints.

In the three runs, eighty-eight checkpoints passed all three times. One hundred and ninety-nine of them failed in all three runs. And eighty-two checkpoints, or 22 percent of the tests, came out differently between each of the runs. So the totals were similar, but the wins and losses traded places and masked the true difference in the totals.

I’ve now run the numbers on three different versions of the data. Nine passes with nine separate graders. Every single time, the flip rate lands somewhere between 22 and 28 percent.

Let me put that another way. Any two identical runs agree with each other on only 78 to 86 percent of the individual checkpoints. That is the system working as designed.

A good example is question 195. It asks for closed M&A deals whose executed agreements actually contain indemnification sections. Three identical runs scored 12, then 17, then 6, out of 32 possible checkpoints.

If the only run result I’d seen was the one that scored 17, I would have called it a strength. If I’d only seen the run that scored 6, I would have said the opposite. Neither would have been true.

Question 208 pulled the same trick. First result of 24. Then the next run resulted in a score of 8. The final resulted in another 8 out of 28.

One run is a story. Three runs are a data trend. And the data keep saying the same uncomfortable thing.

One run is a story. Three runs are a data trend. And the data keep saying the same uncomfortable thing.

Let me pause a beat here. The first thing I thought was that the graders were part of the problem. So I checked on that by giving four fresh graders the task to re-score an identical set of answers, and they all agreed with each other within about two points. On a 26-point swing between two separate runs, it turned out about two of those points are about the grading. The other twenty-four points are about the product.

This means that when you ask one of these tools to dive in and find every matter where something is true, you get one list. Ask the same question again, and you’ll get a different list. Nothing was telling me what parts in the answer were the stable parts.

You have to ask the sweep questions twice, and then you need to compare the two answers. Where they agree is the reliability, where they differ are the parts you cannot rely upon. It requires an extra query to find the reliability, and it is the cheapest control method I could come up with. Nothing else even comes close.

Is this test even fair?

It might be a little embarrassing to admit, but this took me nearly two weeks to even consider.

Across nine runs, some 152 of the 369 checkpoints had never been satisfied. Not once. No matter the setting, or the document versions, or the grader. In my evaluation I had been calling that a failure of the platform.

The thing that I hadn’t checked was whether those checkpoints were even achievable by anybody at all. I’d been putting the blame on the product when it may actually have been the test itself that was flawed. Think of it like a college exam where everyone taking the same exam fails the same twenty questions. After a while, you have to look at the questions rather than the students.

I turned my attention to the 30 questions and presented those to an AI agent that could open the document folders and files directly, rather than having to go through a product. So, presented with a copy of the data set, and no answer key, the agents went through each of the questions, word for word. The grading was identical written protocol, and I made sure that the graders were not told whose answers they were reading.

These three graders scored those specific runs at 265, 266, and 267 out of 369. The graders agreed with each other over 99 percent of the time on the individual checkpoints. Out of the 369 checkpoints in total, only three were disputed at all.

By this basis, the ceiling is around 265. Harvey is sitting at around 132. So it was only getting about half of what the direct file access was able to achieve.

This altered how I looked at two things specifically.

Firstly, that of the 152 checkpoints that had previously never passed, the direct search method got 72 of those. That showed that Harvey had some real misses. The answers were in the files, and the answers were findable. The other 80 failures, about a fifth of the entire test, were not findable by any method I had tried. Keeping those in the results makes it appear much worse than it is. So if we only ran the results on the 289 checkpoints that we proved are achievable, Harvey scores 46 percent and the direct search method scored a 92 percent.

Secondly, I sorted the questions into four categories as easy, medium, hard, and extra hard based primarily on the number of checkpoints the questions carried. On the extra hard group, Harvey scored 29.1 percent. The direct file search, on the other hand, scored 92.5 percent. Whereas on the easy group, Harvey and the direct search scored much closer at 53 percent for Harvey, and 61 percent for direct search.

The questions that I placed in the hardest category weren't actually difficult. They were tedious. Questions where there were 26 checkpoints is a task that requires the system to see how many separate things they can go and find. So the categories I'd been calling difficult and labeling as extra hard, were really just extra tedious because of the breadth of scale in the task. It turns out that tools like Harvey really struggle when it comes to these types of tasks.

The Pinnacle problem

I did find a favorite question in the group, and that is question 178.

This question asks to pull everything that the firm has done for one specific investment fund’s portfolio companies. This is typical conflicts check questions that you would ask before bringing on a new client. The answer to this is 23 matters, and those matters happen to be in three client folders where the numbers are sequential. This is not a trick question by any means.

When the direct file search got the question, it found all 23. It hit all 26 of the 26 checkpoints in the test. I tested a second AI system as an independent run, and it got 25 of the 26 checkpoints correct.

Harvey scored a 1 out of the 26 checkpoints on that question, once, across a total of fifteen runs.

When I looked at the answers that Harvey was bringing back, I could see where it was failing. The C&H law firm’s synthetic data put in some additional investors into the data who had confusingly similar names. The AI tools with direct file access identified those decoys, and both of them filtered them out. Harvey, meanwhile, immediately picked up on the wrong names, every time, and went through and meticulously created well-organized output on the wrong investors. Every single time, on every single version of the data sets.

I made three additional attempts to see if I could fix this on the prompt side, and none of the three tries worked. In one attempt, I asked Harvey to list every entity in the vault that had names that resembled the fund I was specifically needing. It listed those without any problems. It found the twenty-plus entities, listed the exact legal names, tax ID numbers, fund vintages, and the results were all correctly listed and separated. In the same conversation, I went back and asked the question again, and it immediately went back to the wrong set. So, in the same run, it was able to name the single largest client by the correct name, and then it turned around and explicitly ruled it out as unrelated.

It has the ability to get to the right answer, but somewhere along the way it can’t reach the part of the system that determines which documents to retrieve.

The one question that just could not be answered

There was one question in the batch that just could never be answered.

This question asked which of the firm’s documents reference OFAC, the sanctions authority. The answer key listed three checkpoints. First, that 728 documents across 84 matters referenced OFAC. Second, surface all of the 728 documents. Third, do not mention any document outside of a specific list of 15 matters.

The reality of the question was that the 15 matters that were barred from being mentioned actually contained 81 of the 728 documents. That meant that there were 643 other documents that were in matters that the third checkpoint forbid you from mentioning.

This was an obvious contradiction from the other two checkpoints. This created a scenario where the best possible score to that question was a 2 out of 3, and none of the systems were able to get a perfect score, because there was no ability to get all three.

The question had been run fifteen straight times and scored a zero in all of those runs. I’d been continually counting this as a failure each time I drafted a report on the runs.

Luckily, two days after my audit discovered this contradiction, and completely independently from my testing, Harvey Labs shipped a revision to their answer keys. That specific question was rewritten and the three checkpoints were updated to remove the contradiction. The Harvey team found the same defect and fixed it.

That’s the kind of thing that you want an open benchmark to do.

What I am doing with all of it

I have placed both of the updated data sets on GitHub and made them public, on my fork, under the same MIT license as the original C&H law firm data. The C&H-Enhanced set is around half a gigabyte, and the C&H-R set is one and a half gigabytes. I placed the manifests that recorded every single change in there as well, along with the six scripts that I used to generate the whole thing deterministically. That way you don’t have to trust my zip files, you can rebuild it and check for yourself.

I also put the results in there as well, along with a note explaining exactly which version of the answer key they were scored against. Harvey revised 206 of the 250 questions on August 10th and a score against the old keys is not comparable to a score against the new ones.

Let me know if you run it on something else. I’d love to find out what you found.

I want to reiterate something that I mentioned earlier. None of this testing was possible until Harvey, along with Engram, put out the data set, along with the benchmark questions, and answer keys, and then invited people to download it and take a look. They did this with the understanding that someone like me would take it and run it against their own product and publish the findings. Their own paper diagnosed the same weakness I ended up measuring.

More vendors should follow suit. I’d like to run similar exercises on other platforms and see how they hold up and compare.

_{Both corpora, the manifests, and the regeneration scripts are public at github.com/xlambert/harvey-labs, under the same MIT license as the original benchmark.}

_{All figures are scored against the firm-knowledge rubrics as first published on 2026-08-07. Upstream revised 206 of the 250 task rubrics on 2026-08-10, so these numbers are not comparable to scores computed against the newer keys. Everything in the corpus is synthetic. No client information was involved at any stage.}

In this episode of The Geek in Review, we talk with Patrick Forquer, Chief Revenue Officer at Legora, about legal AI’s move from experimentation into daily legal work. Forquer explains why Legora has invested heavily in legal engineers, lawyers with practice experience who work alongside clients on adoption, workflow design, prompt and context engineering, and change management. The conversation also explores an emerging career path for lawyers who pair substantive legal knowledge with AI fluency, especially as firms search for people able to translate practice needs into working systems.

Legora’s acquisition strategy provides another lens on the company’s ambitions. Forquer describes a strategy aimed at building breadth across legal work while adding depth in litigation, commercial real estate, regulatory monitoring, and legal research. Recent acquisitions such as Wexler, Cadastral, and Graceview bring specialized capabilities into a broader agentic platform. Legora’s own 13-day acquisition process also serves as an example of how M&A diligence, document review, drafting, and analysis are beginning to move through shared AI environments.

A major portion of the discussion focuses on the difference between traditional workflow automation and agentic AI for legal work. Forquer draws a line between prebuilt automation and agentic systems: workflows follow predetermined steps, while agents receive a goal, gather context, form a plan, call tools, and work across longer tasks with human review. Context engineering therefore becomes increasingly important. Matter data, firm knowledge, permissions, legal skills, and connections to systems through tools such as MCP all shape the quality of agentic work. M&A due diligence already represents one area where longer-horizon agentic processes are gaining traction. Legora describes the same architecture through its agentic operating system, or aOS.

The discussion then turns to economics, pricing, and proof of adoption. Greg points to Crowell & Moring’s reported 91 percent attorney adoption and nearly 70 percent weekly usage, while Forquer argues login counts and activated licenses tell only part of the story. Legora tracks daily activity and depth of feature use, including tools such as Tabular Review, skills, playbooks, and extraction templates. Agent Pro’s shift to consumption-based pricing introduces another measurement challenge, with credits tied to usage alongside dashboards, spending controls, and project-level attribution. For law firms, a broader question follows: when AI compresses hours while increasing speed, scope, and output quality, traditional measures of efficiency and value start pulling in different directions.

The episode closes with a look at what law firm innovation leaders should prepare for next. Forquer identifies the data layer as one of the central issues behind successful agentic AI. Secure access to documents, matter-level permissions, governance, firm knowledge, and well-structured context determines how far agents progress into complex legal work. Talent matters alongside infrastructure, which brings the conversation back to legal engineers and new hybrid roles spanning law, AI, knowledge management, and data governance. The episode leaves innovation and KM leaders with a practical agenda: improve data governance, build legal engineering skills, align stakeholders around risk and outcomes, and measure value through work product, adoption depth, and client impact.

LINKS

• Legora

• Legora aOS, Agentic Operating System

• Legora Introduces Consumption-Based Pricing for Agent Pro

• Crowell & Moring Marks Six Months of Legora Integration

• Measuring the Impact of AI on Law Firms, Ari Kaplan and Legora

• Legora Acquires Wexler

• Legora Acquires Cadastral

• Legora Acquires Graceview

• How Legora Uses Its Platform to Close M&A Deals in Days

• Legora Series D Funding Announcement

• Legaltech Hub

• ILTACON 2026

Listen on mobile platforms: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Apple Podcasts⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ | ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Spotify⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ | ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠YouTube⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ | ⁠Substack⁠

[Special Thanks to ⁠⁠Legal Technology Hub⁠⁠ for their sponsoring this episode.]

Email: geekinreviewpodcast@gmail.com

Music: Jerry David DeCicca

Transcript

Marlene Gebauer (00:00)
Hi, I’m Marlene Gebauer from The Geek in Review. I have Sarah Glassmeyer here from Legaltech Hub. Sarah’s going to share with us how you can use the Legal Technology Directory to prep for ILTACON.

Sarah Glassmeyer (00:12)
Yeah, so this is probably one of the more exciting times of the year for legal tech kind of leading into ILTACON. Everyone’s excited to see all the new changes, see people we haven’t seen for a year. So Legaltech Hub can really help you pre-game and get ready to get the most out of it because while you’re there, it does seem like you live in Nashville and you’ve been there your entire life, but it goes really fast. You really only have like two and a half, three days to talk to vendors. So a couple of things I would suggest you do.

One, look at the list of who is exhibiting at ILTACON. So if you go to the ILTA website, they have a legal tech directory but guess what? That is us. We partner with ILTA. It is a mirror of our directory.

You can see the little badges, and that will tell you who’s exhibiting at ILTACON. Don’t just wander in there. Think ahead and think like, what am I looking for this year? Am I looking for a new kind of document automation tool? Am I looking for a new AI legal assistant?

So you do some filtering, make a list to see who’s going to be there, and start thinking ahead. And through the directory, you can see, for example, that some companies don’t integrate with X tool, so they’re off the list for us. So you can kind of do a little pre-filtering, pre-gaming, and figure out who you want to talk to, who’s going to make the most of your time. And then from there, if you’re back on our platform in the Legaltech Hub directory and you’re logged in, you can make notes. So as you’re wandering through the exhibit hall, visiting the people you want to visit, you can kind of make notes saying, “Here’s this person’s email address that I spoke to and I want to follow up with them,” or, “Maybe show Bob this when I get back to my office,” or, “These people are off the list.

Forget about them.” And so you just kind of keep track of who you’ve talked to, what you’re thinking about things live, and it’s all in one place on the Legaltech Hub directory, not just a swag bag full of business cards and flyers. So that’s how we can help you get the most out of what you’re doing. But also, I do suggest just wandering around the exhibit hall because everyone who’s there does have a listing on Legaltech Hub. So you can kind of double-check it. But I think it’s really good just to kind of wander, see who catches your eye, have a spontaneous conversation because you might find a vendor that you never considered or something you weren’t really thinking about ahead of time.

So it is kind of a fun time to get yourself absorbed in what’s happening in legal tech.

Marlene Gebauer (02:27)
Yeah, serendipity is definitely part of the ILTACON experience, but this is great advice in terms of using the Legaltech Hub directory because it can be overwhelming and this will allow you to focus on the vendors and people that you definitely need to talk to before you leave. Thank you.

Marlene Gebauer (02:54)
Welcome to The Geek in Review, the podcast focused on innovative and creative ideas in the legal industry. I’m Marlene Gebauer.

Greg Lambert (03:00)
And I’m Greg Lambert And today we are going to be looking at the next phase of legal AI work where vendors are moving beyond, or perhaps more accurately, adding to assistants and isolated pilots, toward more agentic work, more firmwide deployment and new pricing models.

Marlene Gebauer (03:20)
And our guest sits at the center of those commercial and operational questions. Patrick Forquer is Chief Revenue Officer at Legora, where he leads the company’s global go-to-market strategy and works with law firms and legal teams moving AI from experimentation into daily legal work. Patrick, welcome to The Geek in Review.

Patrick Forquer (03:40)
Hey, Marlene. Hey, Greg., thanks so much for having me.

Marlene Gebauer (03:43)
So one thing caught my eye recently. Legora is hiring a significant number of attorneys, not just software engineers. I heard Max on a podcast saying that you guys may have more lawyers than engineers, and that’s kind of unique in the industry. So you’re hiring legal engineers, legal associates, and lawyers with practice experience. What do those lawyers actually do day-to-day?

What skills make someone successful in those roles? And what does that tell you about where the profession’s headed?

Patrick Forquer (04:15)
Yeah, it’s a great question. I get this question all the time. And I think legal engineering is like one of the most exciting roles in technology now, just generally, because they’re on the front lines of AI and the AI revolution, doing the hard work of change management and adoption in firms. And so I think if you wind all the way back, I mean Legora was founded by engineers, right? One of the founding principles of the company is that we want to build with lawyers, not simply for lawyers.

That means taking their feedback and understanding the deep levels of requirements and subject matter expertise needed to build a successful legal AI platform. For us, that’s always been a founding principle, having lawyers in the business helping guide the direction of the product. More importantly these days, there’s this idea of transformation. We’re fortunate that Legora is popular right now. We get to have lots of conversations with many great firms and great companies out there in the market.

But at the end of the day, we have to help lawyers adapt AI into their ways of working, into their workflows. And candidly, you can’t just do that with standard consulting practices. You have to have deep levels of subject matter expertise. So, for example, people from an M&A background understand how an M&A deal gets run and the due diligence process. The same applies to litigation and the different aspects and practice areas of legal work.

And so our lawyers predominantly work in our legal engineering function. They embed with our sales and post-sales teams. First, they have deep levels of practice-area expertise. So they bring credibility to our clients: we understand your business, and we understand the types of deals, cases, and matters you’re working on. And then they have high levels of technology and AI proficiency, meaning they understand how to do things like prompt and context engineering.

They understand how agentic workflows work and how to build them. And they’re highly curious and adaptive in a client-facing environment where you can go in and work on a live deal or a live matter and help build sort of long-horizon workflows within Legora to help our clients deliver faster and better work. It’s really exciting, and I love spending time with our legal engineers. They’re some of the smartest, best people I’ve ever worked with.

Marlene Gebauer (06:36)
That’s a great endorsement.

Greg Lambert (06:36)
See Marlene, this is why we’re having a hard time finding these people. They’re taking them from us.

Marlene Gebauer (06:39)
That’s what I was going to say. I’m seeing discussions about what’s going to happen to our junior people, and those are all legitimate questions. I’m glad we’re having these discussions about other job opportunities that may be out there for people.

Patrick Forquer (06:58)
Yeah, and we definitely think that, Marlene. We think legal engineering is a new career path. It has always existed in different forms at different companies, and Greg and I were talking about that earlier. But the current form of building these bespoke workflows, skills, and capabilities within tools like Legora is not something that was done as commonly as it is now. And they’re making a meaningful impact on the client work that’s getting delivered.

And it’s showing up for our clients in their work with their clients. And I think that’s the most exciting part. I definitely think it’s a new career path that’s opening up a ton of opportunity for the folks that work for us. And we’re really excited about that.

Greg Lambert (07:45)
Just curious, Patrick, about the types of talent that you’re looking for. Is there a certain practice range? Are they, say, seventh-year or tenth-year lawyers? What’s the sweet spot you tend to look for in the talent?

Patrick Forquer (08:02)
We actually hire across the entire spectrum. We have everything from folks who’ve done a couple of years as an associate through partners who work with us and they tend to focus on obviously solving different problems. Especially within BigLaw, there’s so many different types of transactions and deals and matters and cases that are happening on a day-to-day basis and there are different needs, and how you train and enable a partner is very different from how you train and enable a first-year. And we just try to map practice area and sort of seniority to help solve the different problems we’re looking at so we can add the most value to the firms that we work with.

Greg Lambert (08:44)
You’re the revenue guy, so let’s talk a little money now.

Patrick Forquer (08:49)
Okay.

Greg Lambert (08:50)
I know that you guys have raised significant capital. I think I was looking at Series D so far. And you’ve gotten busy on the acquisition side including a recent acquisition of Wexler, which I think is like the fifth

Patrick Forquer (09:03)
Yeah.

Greg Lambert (09:04)
or sixth acquisition. So, instead of asking you specifically about the individual acquisitions, what’s kind of the strategy that you have when you’re looking at acquiring and building through the acquisition process?

Patrick Forquer (09:21)
Yeah, for sure. It’s a couple of different things. First and foremost, we’re hiring amazing talent. And so the folks that are coming into the business through this acquisition are high-slope, highly intelligent, hardworking folks that we want to work with. And they’ve built platforms that are solving big problems that are relevant to our customers and the firms we work with.

In this world, as you know, there’s so much going on in legal tech. There’s so much noise and so much chatter. A lot of times when you’re comparing sort of legal AI platform A to legal platform B, you’re looking at it in terms of like apples and apples, right? It’s like you’ve got drafting, we’ve got drafting, how are we doing it? How are they doing it?

And so on and so forth. So part of the strategy also is we want to be a wide application layer for the firms we work with and a platform that any practice area could work within, but we also want to be really deep. The deeper the capabilities we add, for example the litigation capabilities Wexler provides, the more we can be both wide and deep and that sort of keeps the comparison from an apples-to-apples comparison. So Max likes to make the comparison of apples to fruit salad. And so we want to add more of these bespoke, deep levels of capabilities.

We bought Cadastral for real estate work. We’ve got Wexler for litigation work. We brought a company called Graceview that we integrated into the platform in under 60 days for regulatory horizon scanning as an example. So not only do we want to be able to work with every type of lawyer, we don’t want to just do surface-level work. We want to go deep within each practice area and within each area.

So it’s really important to us that we provide the highest level of value possible. And we’re able to do that through our M&A strategy. And so we’re trying to provide more value and more bang for your buck when you work with Legora.

Greg Lambert (11:12)
Yeah. And when you’re doing these acquisitions, you talked about making it both wide and deep. Are you really looking at kind of becoming almost like a matter management platform where the attorneys are constantly working within the Legora platform over a wide range of activities? It’s not just legal research or document drafting. It may be directly accessing your document management system and bringing things in and working with them there or maybe even the accounting system.

How wide and deep are you going to go?

Patrick Forquer (11:49)
Yeah. With any type of work that you do, no one likes bouncing around to different systems to do different things. And insofar as we can add capabilities to our platform that make sense for the type of work that we’re doing, we’re going to look at it. And certainly for us, we want to look at M&A as an accelerant to adding value and keeping folks spending more time in the platform. So yeah, we’d love for you to be able to do an end-to-end M&A due diligence process, for example, in Legora.

Many firms are doing that today. And actually, that was our fifth acquisition of the year. And we’ve run all those acquisitions on Legora. And we did one of those acquisitions in 13 days, using our product to run the M&A due diligence. Obviously, we work with some amazing law firms on that as well.

So it’s been really fun to see it in action. The more capabilities that we have, the longer folks are going to spend in Legora, and that’s good for them and good for us.

Marlene Gebauer (12:46)
So let’s keep talking a bit about the workflows. We’ve certainly moved from asking, “Can AI draft or summarize?” And now we’re really talking more about these agents that can execute multi-step automated legal work. We hear a lot about it, and I’m sure that there are pockets where it’s done. Which legal workflows are genuinely ready for that sort of next step today, and where do you think it’s still a little bit of theater?

Patrick Forquer (13:22)
Yeah. Well, it’s a great call out, Marlene. From my perspective, you hear a lot about agents out in the market. Everyone’s got an agent doing X, Y, or Z. But if you really look under the hood, a common pitch is, “We have this number of agents.” Anyone who tells you they have like hundreds of agents, I think is perhaps misdirecting a bit.

So, is it a repackaged workflow? Workflows have existed for a long time. n8n technology has existed for a long time. And then there’s lots of different workflow tools that you can buy that have if-then logic. Are you buying something that’s a repackaged workflow or are you buying something that’s truly agentic? And for us, the way to think about it would be

Marlene Gebauer (14:04)
And maybe explain what’s truly agentic versus not.

Patrick Forquer (14:08)
Yeah, so the agent in Legora, like the Agent OS that we launched earlier this year, it can click every button you can click in the platform. So it’s not just a set of rules that it’s following, like “if this, then do this; if that, then do that,” which you have to then go in and like edit those rules or that logic within a particular pre-baked package. With the agent, the big difference is you’re sort of moving from prompt engineering to context engineering, right? So with agentic work, you want to give it a definable scope. You also want to have the data and the sources made available within the project in Legora.

So everything that you’re going to be doing is grounded in source and truth. And then you also want to have a human in the loop at each step to customize and make sure everything is working like you want it to. So if you give the agent a goal in our system, and then you give it some context of that goal around like what it is you’re trying to do, sort of guardrails and things to watch for, then our agent is going to come back and it’s going to ask you some questions, some clarifying questions around the scope, around the goal. It’s going to look through the documents and make sure it has all the context it needs to achieve the goal. Then it’s going to present you with a bespoke plan for that particular goal that you’ve outlined and you can layer in skills, tools, and resources within Legora to make it even more bespoke and more granular to improve the work product.

The plan that it’s showing you is then the prompt. So in the old school world, the “Improve Your Prompt” button was a big feature that we still have that people like, but with an agent, you don’t really need the “Improve Your Prompt” button as long as you give it enough context with a very well-defined goal and all the resources that it needs. It’ll create the plan. And the plan is in many cases a long-horizon task that can run for 20, 30, 60, 120 minutes in one go to solve these increasingly complex problems. As long as as it has access to all the legal skills that we provide it in the tool, all the context you give from the firm or from your client documents, the agent can then go run these long-horizon tasks and solve increasingly complex problems instead of saying, “Go run this four-step workflow” that was pre-baked when I logged in.

And so it’s quite different and it’s extremely flexible and bespoke to the particular problem that you’re working on. That education, though, and that sort of change have been a journey. These are new capabilities that only really emerged as the new models came out around Christmas and the beginning of this year. And so we’ve had to educate our teams on what they can do, what they can’t do. M&A due diligence, Marlene, to answer your question specifically, is an area where we’re seeing an increasing amount of that work being able to get done with the agentic work.

And we’re hoping there are increased capabilities with Wexler as we sort of bake their capabilities in the platform over time that you’ll be able to do that in the long run. But yeah, we’re still working on this together and figuring it out with our customers.

Marlene Gebauer (17:22)
Thanks for thinking of the litigators.

Patrick Forquer (17:24)
Yes, of course. Always thinking of the litigators.

Greg Lambert (17:28)
You were speaking about things coming out over the Christmas break. It was kind of funny because I was talking with one of our AI-forward partners that had developed a process earlier in the year, and he was sending an update and then one of the other partners was like, “Do you realize it’s only been six months since you created this?” And we were all

Patrick Forquer (17:49)
Yeah.

Greg Lambert (17:49)
like, “Holy smokes, is that all it’s been? It feels like forever ago.” I’m wondering, sticking with the agents, and especially with all the transformation we’re seeing, I have a two-sided question. What are you finding that law firms may be a little slow on when it comes to putting an agent to work? And then for those that may be overachievers, what are some things they think it can do that it’s not ready for yet?

Patrick Forquer (18:21)
Yeah, that’s a great question, Greg. I think one thing is to be transparent. The rate of change in AI capabilities, in some cases, outstrips what the organizations we work with are able to absorb. The change is happening so fast. We all have to be transparent and honest about the fact that these things are changing.

What you could do six months ago or the thing you designed six months ago may not work the same way. There are all kinds of organizational questions around that, like approvals to use different capabilities and all the risk and the training and the change management. So I always tell folks, if you’re using AI today only to summarize documents and draft emails, you’re probably not doing enough, right? But in

Greg Lambert (19:09)
Yeah, yeah.

Patrick Forquer (19:10)
many ways, the question is: Does the agent have access to all the context and data and resources that it needs to complete the task? We’re seeing an increase in requests for things like MCP builds. So we have a team of folks, for example, that can build secure MCP access to different systems that’ll enable multi-system execution across a number of different areas, but you have to have the right security and guardrails around data governance in place before you can do that. I think a lot of folks think, “I can eventually just tell the agent, go do this thing, and it’ll figure it out.” And the agents, in theory, have the capability to do that. But do you have that sort of matter-level isolation?

Do you have the document-level permissioning structure down? Have you given it enough context and enabled it with your firm’s embedded skills and context so it knows, “This is how we do this particular piece of legal research,” or, “This is how we do this particular type of drafting,” so it can produce content that’s in line with your expectations? And that’s a journey. Again, going back to our legal engineers, that’s why we have so many legal engineers, so that we can work with our clients in a more embedded way and help them develop those more complex use cases. Does it have access to all the skills and content that it needs to produce the output at the quality that you want?

Marlene Gebauer (20:34)
We’ve heard in the past that a lot of AI companies are saying lawyers are saving time using the tools. But we’re also starting to hear, because of that, how are we sort of looking at the economics of legal services? Danielle Benecke, on the podcast Max was on, was saying that they do this by outcome. There are a lot of different ways to do this. Her team works directly with clients.

So, when are we going to start talking about how to measure the changing economics of legal services and how are we going to price things when using AI? I think people want to talk about it, but I don’t know that people are ready to do anything about it. So I’m curious what your thoughts are on that.

Patrick Forquer (21:27)
Yeah, well, look, we could spend the whole podcast talking about this. We could probably spend all dinner

Marlene Gebauer (21:31)
True.

Patrick Forquer (21:31)
and all night talking about all that stuff too. And certainly I’m not here to tell lawyers how to price and package their work. But I have some thoughts on what you were going through. First, I think it’s a misnomer just to think about AI in terms of efficiency, right? If you’re using AI only to save time on things, I think that’s mostly the way the first wave of AI worked.

People are doing that sort of look back, right? I used to do this thing this way and it took me 10 hours, and now I do it with AI and it took me 30 minutes. That type of stuff has existed for a while. And I don’t think that’s particularly exciting to the market anymore. What we try to look at in terms of a look forward is, okay, if you’re commoditizing certain types of work product that used to take you 10 hours and now it takes you 30 minutes in terms of time, then what does that enable in terms of quality of output?

What does that enable in terms of adding new features or new services that maybe you couldn’t have done previously? What can you package or repackage as part of new work product that wasn’t available? So there are all kinds of 50-state surveys and all these different examples of things that would have been too onerous in a world pre-AI that you can now deliver in a really effective way using platforms like Legora. And when we think about value, anytime you’re looking at an economic system more broadly, if certain pieces of the inputs into that system are getting commoditized in terms of time, then what you typically see in other industries is that the ecosystem and the economy itself gets bigger. So we think there’ll be more legal work, that there will be more legal services, that this is accretive to the overall market.

And it’s a really exciting time to be a lawyer and be working in legal services. And, as I’m sure you both know, like if you talk to any litigation partner, like they’re busier than ever, right? There are more cases coming in, there’s more work to do. People talk about time as it relates to you know the billable hour. But I was talking to a CLO of a big company a few weeks ago and he was mentioning, if we were working with an external counsel and they used to say, “Hey, if we can do this M&A deal in six months,” today they might say, “Maybe we can do it in three or four months.” Is it more or less valuable to do it in three months than six months?

Again, I’m not here to answer that question for folks or tell people how to price things, but I certainly think there’s an interesting conversation to be had around value. And if you’ve got an amazing attorney today who’s working just with Microsoft Word and Outlook, and you give them a platform like Legora, is that attorney more or less valuable to their clients? And what are the economics of that work? Certainly I have every confidence that legal research went from analog to online, data rooms went from banker boxes in a conference room to platforms like Intralinks and Datasite, and paper moved to email. There’s been all kinds of these examples over time and legal services has only grown and thrived.

We fully expect that to be the case with AI. How that looks and the shape of it between lawyers and their clients, things will change and evolve for sure, but we think it’ll be very positive and that there’ll be a lot of opportunity.

Greg Lambert (24:56)
Before I go further into the economics, there was something that you mentioned in your last answer where you talked about change management being one of the hardest parts of the deployment. I read something earlier where I think Crowell & Moring was talking about how they had a 91 percent attorney adoption rate. A year ago, we were fighting to tell people whether it was safe to use these tools or not, and now we’re

Patrick Forquer (25:23)
Yeah, yeah. Yeah.

Greg Lambert (25:25)
looking at 91 percent adoption. And I think they said they also had 70 percent-plus weekly usage within like six months of deployment. So I’m curious when you do these types of rollouts, what are the success metrics Legora has? I can look at metrics and I can see people have logged in or people have run prompts, but I don’t necessarily see

Marlene Gebauer (25:52)
Or the skills they’ve used. Yeah.

Greg Lambert (25:54)
Yeah, I don’t necessarily see some of the details. So what are you looking at on your side as far as success and how your users are using the product?

Patrick Forquer (26:05)
It’s a great question. And this is an area where we’ve evolved and grown so much over the past few years. And it’s a huge emphasis for me. It’s one of the top three things that I focus on every day when I go to work. How are we doing on the implementations that we have active?

And we actually have an entire team dedicated to this particular piece of our customer engagement. So this isn’t something that’s done by sales or our post-sales engagement team. We have a team that’s dedicated and focused on this. We’re continuing to evolve our strategies as we learn. We’ve done thousands of these now, so there are a couple of areas to think about.

For us, the speed and quality of implementation is the number one leading indicator of client satisfaction and overall renewal metrics. So we look at whether we got it done in the timeline we had agreed to with the client, right? So we look at speed. Did we get it done on time? And that time varies by client, by size, by the type of implementation.

We have different implementation options. We run through those and collaborate with our clients to create bespoke plans for all these big rollouts that we do. And then, of course, we look at a ton of data. And as you mentioned, Greg, for us, we don’t look much at activated licenses or number of logins. What we really look for is deeper than that.

Greg Lambert (27:32)
Those vanity metrics are really nice.

Marlene Gebauer (27:36)
Love it.

Patrick Forquer (27:36)
Of course, monthly active usage is nice. But we want daily active usage. And we don’t only want daily active users. Going back to my earlier sort of anecdote. If folks are coming in and only summarizing documents or saying, “Respond to this email for me,” that’s relatively shallow usage, and it’s easily commodified across other platforms.

We know that if folks are using more of our products and doing more complex tasks in Legora, and, as Marlene mentioned, are they using skills? Are they using playbooks? Are they using extraction templates? How many parts of the product are they using? What parts of the product are they using?

So we have a product called Tabular Review. Tabular Review gives us the ability to do bulk document review and extraction. When I first saw Tabular Review and met Max, I thought, “My gosh, this is amazing, and this is going to change the way work gets done. Period. The end.” If people use Tabular Review, they have their light bulb moment with AI.

It’s like, “My gosh, I can get all this data out of this big document set and then turn that into a RAG that I can draft, review, research, and so on, on top of with the agent.” And it unlocks an incredible amount of complexity in terms of the things that you can do. Those are the types of things we look at. Of course, we look at monthly and daily active usage, but we also look at quality of usage. So we have a number of different mechanisms and nomenclature we use internally. We might see that Marlene logged in a certain number of times this week and used a certain number of tools within the platform.

We can’t see the data or prompts, but we can see the tools she’s using. That’s the deeper level of analysis that we look at. And typically when we see certain feature usage, we know that we’re going to have a happy customer who’s solving big problems and getting great results from Legora.

Greg Lambert (29:31)
Now I’m going to hit you with the hard questions so...

Patrick Forquer (29:34)
Great.

Greg Lambert (29:35)
Yeah, butter you up.

Marlene Gebauer (29:35)
We save them for the end. We butter you up, and then we save them for the end.

Patrick Forquer (29:37)
[Laughs]

Greg Lambert (29:41)
and then hit you.

Marlene Gebauer (29:42)
Yeah.

Greg Lambert (29:43)
So a few weeks ago you announced that Agent Pro, which is one of the high-end tools in Legora, is moving to consumption-based pricing. The more tokens you use, the more it’s going to cost. And I think this was not a surprise to the industry that this was coming, but you were one of the first to move to this. So as a customer of Legora, what should people be thinking about as far as planning? It’s easy to plan per-seat pricing because you know what the cost is going to be.

When you start thinking about tokens, credits, or consumption-based pricing, how are you coaching your clients on how to approach that?

Patrick Forquer (30:27)
Yeah, it’s definitely a model from a technology provider that’s newer for legal tech. It’s used very heavily in other databases and technology systems, as you know. So for us, there are pros and cons to every pricing model. So on a per-seat model, the question we used to get was, “Okay, if you buy 1,000 seats from Legora, but what if these people don’t use it? What if Marlene goes on vacation and it’s sitting there?”

Greg Lambert (30:52)
Can I get a refund on that?

Patrick Forquer (30:54)
Yeah. “Do I get credit back if she doesn’t use it or something like that?” I can tell you that there’s no perfect pricing model. There are pros and cons to everything. The second thing is, look, this is new, this is a journey that we’re all on together. We try to lead with a bunch of empathy and understanding and try to be extremely collaborative and transparent with our clients as we work through this together.

I’ve worked at companies that sold consumption before. There are some key tenets I like to think about. First, we have to be able to predict consumption across various scenarios pretty well. The example I always use is: if we’re going to sell you 100 licenses, sorry, 100 credits of Legora. And to be clear, we’re not simply taking the tokens we use and just adding a margin on that and doing cost-plus pricing.

The credits are an amalgamation of different document API and token usage in the platform. So we don’t simply take the number of tokens that you used on a given day and add some money on top of that. But if I sell you 100 credits of Legora and then three months in you use 1,000 and I send you a surprise bill for 900 tokens, sorry, 900 credits, guess what? You’re going to get super pissed and you’ll probably churn, right? That’s an easy way to lose a customer.

That’s not a good scenario to be in. On the flip side of that, if I sell you 1,000 credits and you’re very happy, and then you use 100, and then you renew your contract at 100 credits. If we take a 900-credit churn, that’s bad for us. And then our investors are like, “What the heck? You just took a 90 percent churn on this very happy customer.

What’s going on?” So it aligns incentives in a really nice way. It aligns incentives around whether you’re using the product and whether we’re able to accurately predict that. Does the usage account for experimentation and different usage patterns? And do certain user types use the product differently than others? Are we able to report on that?

Are we providing visibility and governance? So, can we do user-level caps? In our case, you can put usage metrics against a particular project or matter in Legora. So you have governance and control you can put in place to make everyone comfortable and prevent certain situations from happening. But at the end of the day, it aligns incentives in a really nice way.

And if we’re able to predict the way we should, and communicate your consumption on a weekly, monthly, and real-time basis, so you know exactly how much you’re using and the patterns we’ve laid out hold, then you typically get to a good outcome. At the end of the day, I always ask our teams: for every dollar of Legora you buy, like what do our clients get? For every credit of Legora you use, what do you get in the outcome? Is it only efficiency, or do we deliver the other value-added benefits that we were talking about? And one of the more exciting things for the second half of this year is that we’ve hired a value engineering team who are working with our customers on what is the ROI of working with a platform like Legora.

And it can’t simply be, “It saved Greg some time on something.” It’s got to be more robust. We’ve done some work with Ari Kaplan on this and published some reports, but this is really the first inning on this stuff. But if we can help our customers tell a narrative, tell a story with data showing that for every dollar of Legora we put in, this is what we get out, and we can do that in a way where we’re communicating and executing on the consumption pricing as I described, then I think it’ll be fine. But it’s new, and we’re working through it with our clients in a really collaborative way. I think we’ll be in a good place.

And like you say, this is the direction of travel for the industry more broadly and I think we’ll be in a good space.

Greg Lambert (34:57)
I think one of the questions I hear the most as we start looking at this, because it’s not just going to be Legora, there are going to be other

Patrick Forquer (35:06)
Yeah.

Greg Lambert (35:06)
AI companies that we work with that aren’t necessarily going to use credits and consumption, but instead use a pure model where if you use a million output tokens, you’re going to get charged a million output tokens. And if you let an inefficient process go awry, that’s on you because they’re not upcharging you, they’re charging for what you use. Are you guys thinking about safeguards so that, if somebody sets up an inefficient agent that runs into a loop and can’t get out of it, it breaks, or are there some kind of safeguards that you set up?

Patrick Forquer (35:45)
Yeah, a hundred percent. We already have a lot of these in place and we’re continuing to invest in this area around model selection, model routing, right? You don’t need to use whatever the latest model is for every single task in Legora, right? So model selection, model routing, guardrails against everything from prompt injection, matter-level isolation, all these different things. We we have to have a lot of guardrails in place in the product which we already have, and then we’re continuing to build what we need to make our clients feel more comfortable using Agent Pro in particular.

Marlene Gebauer (36:22)
So Patrick, we have a new question. We sort of look backward to the present and then for our crystal ball, we look for the future. So this is sort of looking backward and looking to the present. What’s one thing that strikes you that’s different today than last year in our industry?

Patrick Forquer (36:43)
I’m glad you asked because I was actually looking back on Max’s interview with you guys last year, and he made the prediction about these long-horizon agents. He said, “They’re coming. It’s going to come faster than you think.” Before we had the agent, we had the assistant. And the assistant could do a couple of steps of a workflow. I think our workflow builder capped out at about five steps because the prior models would run out of gas.

They get what they call context anxiety, right? And wrap something up before getting to the end. The ability for the agents to do increasingly complex long-horizon tasks is here. And it’s something that we were working toward last year, but the reality was we were still in workflow mode. Last summer we were in workflow mode, whereas this summer we’re in Agent OS, with all the things we were just talking about.

So that’s been really exciting to see that come to life. More importantly, we’re seeing our clients use it. And I hear from clients every day about something cool they did with Agent that they never would have been able to do. I was with a client yesterday down in Atlanta. He was talking to me about some exciting work he was doing for a business development use case.

It generated a deck he was excited to use in a pitch meeting. I think the agentic revolution, so to speak, is here and it’s all new and we’re figuring out new applications, new use cases for it every day, even those of us who work at Legora. I was in a demo the other day and one of our legal engineers asked the agent to not only redline a document, but create an HTML page of the issues list based on the redline. That way, the lawyer could have a visual representation they could share with their client based on the redline it had generated for the customer. I honestly didn’t know Legora could generate HTML like that.

I thought, “That’s so cool. I didn’t know

Marlene Gebauer (38:41)
Mm-hmm.

Patrick Forquer (38:41)
that.” And so there are all these things that we’re uncovering and, as you can tell, I get excited about it. Our team does too.

Greg Lambert (38:49)
Yeah, I heard someone say HTML is now the new markdown for agentic AI tools especially.

Patrick Forquer (38:58)
Yeah, for sure.

Greg Lambert (39:00)
So let’s jump to the crystal ball question and as we told you in prepping for the interview, you can answer this however you want, but I’m going to phrase it a little bit differently this time because I think this is an interesting and a little bit selfish of me to do this. Let’s say you and I swapped roles. You came in as a chief innovation officer in an AmLaw firm. What’s something that you think someone like me should prepare for over the next few months or a few years?

Patrick Forquer (39:36)
Well, I mean, I think your jobs will be increasingly essential to the success of the firms that you work with. And we think it’s really exciting. Yes, more money. More money. Well,

Greg Lambert (39:46)
So you’re saying we’ll make more money, right?

Marlene Gebauer (39:48)
[Laughs]

Patrick Forquer (39:50)
Hey, man, I read LinkedIn too. There’s a lot of stuff going on in knowledge management these days.

Marlene Gebauer (39:53)
Send that out to the...

Greg Lambert (39:57)
Yeah, we will neither confirm nor deny that we’ve passed that around to...

Patrick Forquer (40:02)
Yeah. It’s a talent war out there right now. But we might have to talk afterwards, Greg. I think the first thing is, and we’ve encountered this because we use Legora internally as well, and we use a number of AI tools. But the layer that gets often overlooked is the data layer.

And with all the talk about agents and all these different things, as we were discussing earlier in response to Marlene’s question, where does the data sit that you need to go execute these long-horizon tasks? And how do we teach systems thinking and design thinking to our internal teams? And maybe that’s the lawyers, or maybe that’s part of the KM function around how we think in terms of these step-by-step workflows that need access to specific pieces of context. The more context you give agents, the better they do, right? But accessing that context within the firm, for many reasons, is such a challenge.

We talk to firms all the time about this. It’s about getting the data in a way that’s secure within the right governance frameworks, but also understanding what your risk tolerance is for the different use cases that you want to do with different agents. That data layer is more important than ever. And that’s an area where our legal engineers are constantly collaborating and working with our customers to understand how it exists today and what we can change to make the agent more powerful. So the data layer is definitely an area that I would look at.

And I do think, going back to the legal engineering sort of type of conversation, that there are all kinds of new roles and new capabilities within those roles that are going to be available. So making sure you have the right team in place probably means I would ask for a bunch more headcount, Greg and Marlene, as I’m

Greg Lambert (41:54)
Mm.

Patrick Forquer (41:55)
sure you’re always doing yourselves and all the knowledge management leaders we work with, doing the same. There are new skills and new capabilities around AI specifically, data governance specifically, that I would want to hire for and make sure that we had that in-house so we could move faster on the AI adoption curve. Without this sort of data layer and without the talent in place, you’re going to be limited in the complexity of the tasks that you can do. And again, we’re all on this journey together as we sort of try to adapt these increasingly advanced capabilities and technologies into our platforms. And I think from our side, one thing that Legora wants to do a much better job of is to leverage the collective knowledge from our peers.

And that’s why I love this podcast and what you guys are doing because there’s so many smart people out there trying to solve these problems. But it can’t be KM or innovation alone. You need partners, customers, KM, innovation, and you need everyone on the same page. Can we align on what the goals and intentions are of the firm as it relates to AI? What are your expectations from us?

We need to make that alignment clear. For the people I talk to who do what you guys do, getting stakeholder alignment is difficult. Everyone has a perspective. People come from different places and may have higher or lower risk tolerance. But at the end of the day, you’re held to a standard, right?

You’re held to a level of expectation. And sometimes you have a great opportunity to go hit that expectation, and sometimes you’re limited because of the constraints put on you by the business. But I’m not sure everyone always appreciates that. From Legora’s side, our job is to try to bring those folks together so people have a deeper understanding of the challenges so that we can unlock the true potential here. Those are some things I would think about.

And you’re probably thinking, “I’ve already thought of all those things. This guy.”

Greg Lambert (43:50)
All right. Nope, I’m writing them all down. That’ll go in my monthly report now.

Marlene Gebauer (43:53)
We’re taking notes, man.

Patrick Forquer (43:59)
I know I’m not telling you anything you don’t know, but just to reinforce, as I’m sure you guys have talked about many times.

Greg Lambert (44:06)
Well, I appreciate that. And Patrick Forquer, CRO at Legora, thank you very much for taking the time to talk with us today. We’ve covered a lot of ground here and it’s been a lot of fun. Thanks.

Marlene Gebauer (44:19)
Thank you, Patrick.

Patrick Forquer (44:19)
Thanks, Marlene. Thanks, Greg.

Marlene Gebauer (44:21)
And thanks to all our listeners for listening to The Geek in Review. If you enjoyed the show, share it with a friend, share it with a colleague, and we’d love to hear from you on LinkedIn and Substack.

Greg Lambert (44:32)
And Patrick, where’s the best place for listeners to reach out and learn more about you and Legora?

Patrick Forquer (44:38)
So you can send me an email anytime, patrick@legora.com. We’ve got an amazing website, https://legora.com, and our LinkedIn page has all the latest announcements from the company. So give us a follow. Send me a LinkedIn connection or an email. We’d love to chat with your listeners, and I appreciate everyone listening today.

Marlene Gebauer (44:55)
And as always, the music you hear is from Jerry David DeCicca. Thank you so much, Jerry, and goodbye, everybody.

Patrick Forquer (45:02)
See you.

I was working my way through my own Substack subscriptions this morning, drinking my coffee, and scanning the titles to look for something interesting. I stopped when I got to Peter Diamandis’ Metatrends newsletter and his newest article called “Your About to Live Like a Millionaire.” Spoiler, we’re all going to live like millionaires by 2031. According the Diamandis, we’ll all have personal physicians, private tutors, robots to clean our houses, private flights, and cheap robot taxis. Sound great! Let’s do it!!

I’m a big fan of techno-optimism. These types of articles are really fun to read and think of ways to expand the boundaries of the possible. I’ve been writing with this approach on blogs, and talking about it on podcasts for nearly twenty years now. Diamandis spins a great story, and he’s really someone who also likes to push the boundaries of the possible, so I jump in. For the first part of the story, I was along for the ride.

As I kept reading, it began to cause what is commonly referred to in the industry as an “ick factor.” The common refrain of the tech-bro (the ultra rich tech bro to be more specific) of trust us… we’re going to make the world a better place. Also, don’t tax us, don’t regulate us, and move out of the way because we know what is best, and will bring everyone along to the new utopia.

It’s pure techno-solutionism wrapped up in market libertarianism. And it frustrates me.

Give Diamandis His Due

Diamandis is a smart man, who has accomplished a lot, and I mean A LOT, of things in his life. His article starts off with a good analogy of what the rich had in King Louis XIV’s era, and how this ultrawealthy aristocrat died of gangrene because the best medicine at the time consisted of attaching leeches to heal the illness. He compared this to what most first-world (implied) minimum-wage workers have access to today. I have a few issues with this, but I get where he’s going with it. Technologies like water/sewage, air conditioning, antibiotics, and smartphones have improved overall life for practically everyone in countries like the US.

His underlying point about advancements in technology areas have created a demonetization for things that were once far out of the reach of everyday folks is a real thing. The smartphone alone collapsed the price of a camera, a GPS device, a music store, and an encyclopedia into one device. Costs do fall, and access really does spread, that’s not my argument.

The Bumper St’ICK’er Slogan

Diamandis puts it as “The millionaire lifestyle doesn’t trickle down. It collapses down.”

That’s a great line that fits on a bumper sticker. But it’s also the line that puts the “ick” in bumper sticker. Add to this the part that Diamandis puts in that should be where he spends his time, but glosses past in order to get to the part where technology will save everything and everyone.

The question isn’t whether the abundance arrives. It’s whether we distribute it.

The ‘whether we distribute it’ may be the hardest issue that humanity ever faces on any technology solution. Technology has never been the solution for distribution. Distribution gets solved, if it gets solved at all, by institutions, law, labor leverage, and by policies. These are the parts that Diamandis hand waves away for the rest of his article.

Here’s just a few examples to back me up. First, let’s take something like insulin. My father was a Type-1 diabetic. When he was diagnosed at age 14, he was told that he would probably not live past age 30. While he lived to be 73, it was a constant struggle for him, mainly around how insulin was distributed during his lifetime. Over a century ago, Frederick Banting’s team handed the patent to insulin to the University of Toronto for a dollar. The idea was that Banting believed that something as life changing as insulin belonged to the world. The medicine and the molecule was cheap, while the system of distribution and access stayed expensive. My father died in 2012, and it wasn’t until after his death that Congress acted to make the abundance and price align.

Another example, with more of a legal argument to it, is PACER. Federal court records should be about as demonetized as information gets. But here we are, still paying by the page when the technology has been there since cell phones were the size of bricks. It can be done if the political will is there. I worked with a team in the 1990s in Oklahoma and placed the dockets online for free. But, if it weren’t for laws and policies, distribution of those court records would still be behind a paywall like PACER.

Tr’ICK’le-Down Technology Distribution

The article doubles-down on how technology will solve the distribution issues. As he wraps up his discussion, Diamandis makes requests to different factions to explain what they need to do to help make this happen. My serious eye-rolling came as he listed out the common arguments on why the technology creators are the saviors for us all:

• Don’t tax the rich

• Don’t regulate the tech

• Allow unfettered access to the markets for AI, robotics, and autonomous transit

The wealthy technology class will create a situation where abundance and demonetization won’t trickle down, it will “collapse down.” Let’s all just remove the rules and trust the folks holding the capital. Good things will come as a result.

I’m not one for making projections, but I think Collapse-Down Economics will be about as successful as Trickle-Down Economics was. It will be great for the rich, but the chasm between the have and the have-nots will get even wider.

In addition to not having great trust in Diamandis and his 2031 projection, I want to point out what his business is. His concierge healthcare projects include Fountain Life, which he co-founded with Tony Robbins and Dr. Bill Kapp. The APEX membership for this is around $19,500 a year. Maybe this will drop down to zero by 2031, but I don’t think it is too much of a leap to be skeptical of someone who is telling you not to regulate and industry, who also sells to that industry.

P’ICK’ing at the years in the middle

The article skips over the five years between 2026 and 2031. There are lots of new industries and distributions ready in 2031, without and explanation of what happens to the industries that exist in 2026.

We live in this gap right now as we contemplate the future advancement in technology abilities and what that means for jobs that exist in the present. We keep hearing, and watching the automation of real jobs. Law firms are setting up leadership roles to set strategies for changes in how we price the value of our work, and our workers.

And we have someone who is telling us that in 2031, our kids will be living as well as millionaire kids. But please, don’t ask any questions for the next five years as the technology (and those who control it), just figure it all out along the way. I hope everyone has a 2-5 year nest egg set aside as we make our way to Universal HIGH income.

Naming the fr’ICK’tion

Want to know what is the biggest road block in Diamandis’ future utopian world? The legal industry. The contracts, antitrust, consumer protection, IP, employment law, are all nasty little areas of the regulatory scheme that is holding us back from spending more time with our families on the beach while we roll in that sweet, sweet, Universal HIGH Income of our AI wealth dividend.

Again, I don’t think it is an unrealistic view to be a little suspect when a wealthy futurist outlines the functions of the legal industry as an obstacle to abundance. It sounds like he is simply wanting to create a world where the people who already hold all the levers get to also plan out all the distribution without any of the rest of us getting in the way.

Look, there’s no contradiction in wanting to find a balance between getting my AI tutor, and my cheap medical diagnostics, and my $80,000 house. But I want this to be open to the majority of people, and we have a history of that just not happening. The history that we do have says that part of “whether we distribute it” rests on that really boring part of institutional work that he keeps referring to as an obstacle.

I’ve had a little bit of fun, and a little bit of venting at Diamandis’ expense here. I think he can handle the criticism. Plus, I’ll keep reading his stuff because I do like the optimism and it is a story that should be told. The technology is real. The coming abundance is real. I just think that the only tools that have ever worked to truly distribute it in a meaningful way should also be real.

It turns out that tech companies don’t sit outside of government as just ordinary vendors. This week on The Geek in Review podcast, we welcome back Texas A&M University School of Law professor Hannah Bloch-Wehba to talk about accountability of Big Tech, AI regulations, government surveillance, and the intertwining of public authority and private tech infrastructure. Bloch-Wehba traced the dependency between the two powers all the way back to the 1930s in her article “How Tech Took Over,” in how the tech sector became a foundation for national security and economic growth.

Today’s hybrid form of governance, where Bloch-Wehba explains how a handful of private companies supply data and cloud systems, along with decision-making infrastructures across multiple governmental agencies. It is a struggle for traditional constitutional doctrines to adjust to the modern technology and the operations provided by contractors that are providing their core foundational operations.

The issues also enter into the criminal law enforcement areas and Bloch-Wehba’s “Rights, Knowledge, and Capture in the Datafied State,” discusses how trade-secret claims are throwing a barrier between proprietary data systems and criminal defendant’s ability to examine the systems that are being used to convict them in the courts. There is a strangeness in the judicial systems where corporate choices are shaping the legal process being followed, rather than corporate governance following established legal norms.

Bloch-Wehba’s “Information Law Pluralism” covers how privacy rules, audits, impact assessments, disclosure duties, researcher access, and independent review as parts of a broader system governing exactly how knowledge is shared, validated, and even produced. There seems to be no single device that transparently provides accountability. In addition, she lists how a political campaign program against states attempting to regulate AI companies and products is weakening state transparency even further.

Finally, we cover Bloch-Wehba’s “Rethinking Federal Support for Journalism” where she argues that platform payments give rise to the risk of replacing a governmental dependency gets switched for journalist and new organizations being financially tied to companies they must scrutinize. Ideas floated like an AI tax provide some alternative funding possibilities for supporting local and public-interest journalists.

LINKS

• Hannah Bloch-Wehba’s website

• Hannah Bloch-Wehba, Texas A&M University School of Law

• “How Tech Took Over,” SSRN

• “Rights, Knowledge, and Capture in the Datafied State,” SSRN

• “Information Law Pluralism,” Indiana Law Journal

• “Who’s Regulating Police Technology? It’s Not the Courts,” Tech Policy Press

• “Rethinking Federal Support for Journalism,” Knight First Amendment Institute

• Google Maps: Updates to Location History and on-device Timeline storage

• National Science Foundation Act of 1950

• National Science Foundation history

• Flock Safety license plate reader cameras

Listen on mobile platforms: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Apple Podcasts⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ | ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Spotify⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ | ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠YouTube⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ | ⁠Substack⁠

[Special Thanks to ⁠⁠Legal Technology Hub⁠⁠ for their sponsoring this episode.]

Email: geekinreviewpodcast@gmail.com

Music: Jerry David DeCicca

Transcript

Marlene Gebauer (00:01)
Hi, I’m Marlene Gebauer from The Geek in Review. I have Sam Moore here from Legal Technology Hub. He’s going to tell us about AI governance from the advisory side.

Sam Moore (00:10)
Thank you. Good to see you again. In the advisory team at Legal Technology Hub, we’ve been having a lot of conversations with clients about AI governance. Most law firms and law departments have at least an AI policy of some kind, and some broad standards are starting to emerge. A trend we’re seeing at the moment is a shift away from policies written in 2023 and 2024, which largely stated what you couldn’t do, and toward a more informative approach that defines different risk categories and makes distinctions between routine, low-risk uses of AI and other, more substantial use cases. I think that’s appropriate for where we are, and it’s also what clients are coming to expect from their advisors. One of the bigger challenges we’re seeing right now is how to express an AI use policy in such a way that any member of your legal team could explain it to a client. I think that’s still a big challenge for the industry.

I don’t think clients are all that happy if their attorney says, “I’ll have to get back to you about that,” because the attorney is using AI day to day, so they should be able to explain the policy themselves. How else do you know they’re complying with it? We’re doing some interesting work at the moment, helping law firms and law departments take those first-version AI use policies that are very much “thou shalt not” and turn them into more readable, appropriate governance positions that inform the conversation with clients. If anyone wants to find out more about our advisory services, they can find me or Cheryl Wilson Griffin on LinkedIn, or they can visit legaltechnologyhub.com.

Marlene Gebauer (01:50)
Thank you, Sam. It is important for AI policy to be in plain language so people understand it.

Sam Moore (01:58)
Absolutely.

Marlene Gebauer (02:07)
Welcome to The Geek in Review, the podcast focused on innovative and creative ideas in the legal industry. I’m Marlene Gebauer.

Greg Lambert (02:14)
And I’m Greg Lambert. Today we welcome back Hannah Bloch-Wehba, the law professor at Texas A&M and a leading scholar on law and technology. Hannah writes about how tech can be used to conceal power and evade accountability. Hannah, welcome back to The Geek in Review. We were just looking at the date. It’s only been seven years and two children.

Marlene Gebauer (02:34)
Ha ha.

Hannah Bloch-Wehba (02:37)
Thank you so much for having me back after seven years and two children.

Greg Lambert (02:42)
Ha ha ha.

Marlene Gebauer (02:44)
We are absolutely happy to have you, and you have been busy in your role at A&M. So we’re going to dive in.

Greg Lambert (02:51)
Yeah, well A&M’s been busy.

Congrats. What are you guys, like 22nd now in U.S. News & World Report and all the good stuff?

Hannah Bloch-Wehba (03:00)
Something like that. We have a new building, a new campus coming up. It’s a pretty exciting time at A&M.

Marlene Gebauer (03:07)
Terrific. Well, I want to dig into some of the things you’ve been writing about. In “How Tech Took Over,” you argue that the government didn’t merely fail to regulate Silicon Valley; it affirmatively helped make technology companies essential to public governance. What legal and policy choices were most responsible for creating that dependence?

Hannah Bloch-Wehba (03:30)
So in law and tech circles, we often talk about deregulation and hands-off methods of regulation that started when the internet was commercialized in the mid-’90s. But in “How Tech Took Over,” I go way back to the 1930s and ‘40s, and I look at choices that were made in science policy and innovation policy then. Essentially, the government knew that technology and innovation were going to be critical both to economic progress and to safeguarding national security. So they built institutions that promoted both scientific research and technological innovation. Once that ball started rolling, the government became a primary consumer of those innovations. From almost a century back at this point, the government has been making legal and policy choices around science policy and tech policy, like the National Science Foundation Act and military funding of basic research, which were intended to cultivate this industry for public benefit. I think that set the stage for tech’s expansion, but not in a predictable or linear way. So over time those policy choices have changed and we’ve had these different political moments where, for example, the balance between economic progress and national security needs has shifted somewhat. But overall, I think this is a really important sort of reservoir of federal support for the tech industry that tech law scholarship hasn’t really explored. And so that’s why I was interested in digging into this history and seeing why this happened and how.

Marlene Gebauer (05:25)
What was the response to the article?

Greg Lambert (05:28)
Everyone came out and said It’s all lies. It’s all lies, Hannah.

Hannah Bloch-Wehba (05:33)
You know, I think people have been interested in it in this moment when it’s clear that the relationship between the federal government and tech companies is not an arm’s-length regulatory relationship. We see CEOs at the inauguration lining up to support the president. We have press conferences about the urgent need for federal funding to help build data centers. The U.S. government is intervening in a permitting lawsuit in Mississippi against xAI. We are seeing a lot of weird dynamics in the last couple of years that we haven’t seen before. The conventional narrative that the federal government has deregulated doesn’t capture what’s going on. So I think people are aware and hungry for a better explanation of this relationship. The response has been interested and supportive. I will say, people are like, “This isn’t an article, this is a book,” which is true. It will eventually be a book, but right now it’s one article at a time.

Marlene Gebauer (06:44)
And we’ll have you back for the book.

Greg Lambert (06:45)
Yeah, definitely. It’s

Hannah Bloch-Wehba (06:47)
Well, that’ll help me finish it.

Greg Lambert (06:50)
Well,

Marlene Gebauer (06:50)
Ha ha ha.

Greg Lambert (06:51)
it’s definitely interesting times. You started talking about this almost commingling of tech and government. I think we’ve always assumed that the traditional assumptions of constitutional accountability attach to the government.

But we’re almost seeing this blending of authority between the private sector in Silicon Valley and the government. So how are you viewing that in your now article, but soon-to-be book, and explaining this blending between private and public accountability?

Hannah Bloch-Wehba (07:34)
So I think this, what I call the emergence of a hybrid form of governance, that intentionally takes advantage of relationships between private actors and public entities, is complicated for conventional approaches to constitutional law. We’ve always had this problem to some degree.

There have always been efforts to extend constitutional accountability to contractors, for example. We have a complicated body of doctrine about how and when that is and isn’t appropriate. We have state action doctrine that limits the application of the Constitution in general. Then we have little exceptions around the edges of the state action doctrine. The problem here, I think, is that the kinds of functions that many tech companies are performing don’t look that familiar. So one of the arguments I make in the paper is that tech companies are really enmeshed in providing the informational infrastructure for government decision making. So take an example like cloud infrastructure for federal government agencies.

A huge amount of that infrastructure is provided by a handful of tech companies. Is that a government function? It’s hard to analogize that kind of function to something that would have happened even 20 or 30 years ago. So we don’t have a lot of good comparisons to say, okay, well, this looks like this other form of public-private cooperation, so here’s how we should think about it.

So I think we’re feeling around in the dark for the kinds of principles that we ought to apply. It’s much easier to apply the traditional state action doctrine when you have a traditional government contractor relationship than when the contractors are providing the thing that makes the government run, and that is, to a significant extent, what’s happening right now. So I think it’s getting much harder to pull them apart. At the same time, a lot of those infrastructural activities are hard for normal people to see and understand. An ordinary person doesn’t care whether the informational infrastructure for a federal agency is provided by Google or Oracle or the government itself. We don’t have a lot of insight into what’s going on. So it’s something that I think is important and yet overlooked in constitutional doctrine and in scholarship, frankly.

Greg Lambert (10:30)
One of the things this brings to mind as you’re talking is, typically, we have a transfer of power, different administrations, different Congresses, but the underlying structure is consistent. Do you see this as causing an inconsistency as we transfer power from one administration to another, or when a new Congress comes in, that we’re kind of the...

Marlene Gebauer (10:58)
We’re looking short-term. We’re looking in eight-year segments as opposed to long-term.

Hannah Bloch-Wehba (11:04)
I guess I see it as related, but a little bit different. It’s never been the case that when a president enters into a long-term contract with an outside party, you’re going to redo that contract every Congress, right? Or with the changeover of every presidential administration.

So the system of contracting has always been, in some way, vulnerable to the accusation that it’s not sufficiently politically accountable because the people who made the agreement long predated the current administration. I think this introduces an interesting wrinkle, right? This is not a Trump administration problem. This is a problem that has been occurring and building, as I argue, over decades, right? But if you enter into a long-duration contract with a tech company to provide something like cloud infrastructure for the Department of the Interior, I’m making this example up, but you enter into that contract with Google, you are not going to then cancel it and enter into a different contract with a different vendor for the next administration because everybody knows it’s too much trouble and cost to switch from one vendor to the other. It’s enough trouble to switch from Gmail to Outlook. Nobody does that in their personal lives. Now imagine the entire informational infrastructure of a federal government agency is locked into one provider. It’s hard to destabilize that relationship. Regardless of what elected officials might want to do, there are serious switching costs and serious anti-competitive effects.

Marlene Gebauer (13:08)
I’m curious if there’s another model out there that perhaps clarifies the relationship that we could consider.

Hannah Bloch-Wehba (13:18)
I’m curious if you have one in mind.

Marlene Gebauer (13:21)
I don’t have one in mind. I just...

Hannah Bloch-Wehba (13:23)
I can’t.

Marlene Gebauer (13:23)
I’m just, I mean, I realize this is all...

Greg Lambert (13:27)
Are you talking about another government, another country or state?

Marlene Gebauer (13:30)
Perhaps in another country or another government, or something that someone has surfaced as a potential model, if there’s any work being done in that space.

Hannah Bloch-Wehba (13:43)
So I’m not a comparativist, but I do think when I’ve talked about this issue in front of European audiences, they’ve been baffled. They’re like, “We don’t have this problem.” There are a couple of reasons why they don’t have that problem. One is that I think they do a lot more public provision than the United States tends to do. So one model is for the government to build it themselves or to recognize that government investments give rise to a certain degree of state control. Historically, the United States has not taken that approach, but I could see a parallel universe where we might take that approach of public provision. I also think that the U.S. constitutional doctrine of state action is not one that I’ve seen in robust form in most European constitutional democracies. So there’s no reason why you have to have that doctrine as the delineator of what’s subject to constitutional constraint and what’s not. You could wish away some of these features of our order, but in the real world where we live and work, these are the constraints we’re operating within.

Greg Lambert (15:17)
Yeah. Well, it’s kind of like Sean West, someone who’s been on the show before, said, Europe’s superpower is its regulation. I think part of the traditional American superpower has been its somewhat hands-off approach to regulation. I can see how everything we do baffles the Europeans, so that doesn’t surprise me.

Hannah Bloch-Wehba (15:41)
Yeah, but I think they also, the last time I gave this kind of paper, I gave an early version of this paper in Berlin in 2022. I think it was before the Europeans had turned away from American tech platforms as reliable partners. So I think now that the Ukraine war has been dragging on for so long, and we see that the Europeans are trying to make this pivot away from partnering too closely with U.S.-based tech companies, they’re more attuned to these kinds of problems of capture and enmeshment and the risks they might cause for Europe.

Right. It’s one set of risks when it’s U.S. companies within the U.S. government. It’s a totally different situation when we’re talking about U.S. companies providing the infrastructure for a foreign government, right? I think that illustrates what the structural problems might be, even though the risks themselves look different.

Marlene Gebauer (16:56)
All right, I’m going to pivot a little bit in terms of our questions. I mean...

Greg Lambert (17:01)
We could stay on this topic all hour.

Marlene Gebauer (17:03)
We really could. I am fascinated.

Hannah Bloch-Wehba (17:05)
Yeah.

Marlene Gebauer (17:08)
But when the book comes out, Hannah Bloch-Wehba (17:11)
No.

Marlene Gebauer (17:11)
we’re going to talk all about it. So in “Rights, Knowledge, and Capture in the Datafied State,” you examine government use of privately developed data and algorithmic tools, particularly in criminal law enforcement. What happens to due process when someone can’t examine or challenge a system because a vendor claims trade secret protection?

Hannah Bloch-Wehba (17:33)
Yeah, I mean, this is an emerging problem for criminal law enforcement, and it’s been emerging for several years now. I’m definitely not the first person to write about it, but it’s not going away. The problem is that when a vendor claims trade secrecy to conceal information about how a product works, a product that was used to produce evidence used to arrest or convict somebody, then the defense has a right to challenge, to confront the evidence against them. They can’t do that because the trade secrecy claim stands in their way. So I think it undermines people’s ability to access sufficient information to exercise their rights.

It undermines their ability to challenge the evidence against them, but it also creates a bigger, more systemic problem, which is that a lot of these tools aren’t tested or validated in a meaningful way before they get used. The process of confrontation between the defense and the prosecution is the way the public gets information about whether this tool works or not.

If you can’t confront it, then you can’t produce that information. Then we don’t know whether it works or not. That seems to me a problem that has social effects beyond the individual defendant in an individual case. We’re using technologies that haven’t been tested or validated, and then, in the course of these individual confrontations, undermining the ability to produce evidence about their validity.

Marlene Gebauer (19:25)
Your Tech Policy Press article shows Google effectively determining whether and how police could conduct geofence investigations. What does that example reveal about capture when a company controls the relevant data, the investigative possibilities available to police, and the information available to the courts?

Hannah Bloch-Wehba (19:49)
Yeah, so this is an example of the broader dynamic, which is that these companies wield control over the government’s ability to engage in certain courses of action, and that control isn’t always visible or checkable. Google is a great example. For many years, they kept individuals’ location information by default in something they called the Sensorvault, which is kind of creepy and not what

Marlene Gebauer (20:25)
Yikes.

Greg Lambert (20:26)
Yeah.

Hannah Bloch-Wehba (20:29)
I would have called it if I had known that name would have gotten into the public domain.

Marlene Gebauer (20:31)
They needed a PR person on that one. They did not have one.

Hannah Bloch-Wehba (20:35)
They didn’t anticipate that becoming public, but they kept everybody’s location information by default in the Sensorvault. That made it possible for law enforcement to go to Google and say, “Hey, we want to know everybody who was in this location at this particular time.” Only because Google had made that choice was it possible for government to do. So, starting several years ago, once law enforcement figured out they could make this kind of request, they started doing it again and again. This is a geofence warrant, and it quickly became the most frequent form of judicial warrant Google was receiving. It became an appealing tool for law enforcement. Strikingly, Google was the only company that would comply with these warrants.

So it’s not that other companies didn’t have the data, but Google was the one that would give the data to law enforcement. To their credit, they said, “Well, we’re not going to give it to you willy-nilly. Our corporate policy is that you need to get a warrant rather than a subpoena or a court order. If you get a warrant based on probable cause, we will work with you.” Then they created this elaborate process by which they worked with law enforcement to deliver this information. It’s a striking example of the law of Google. This is not the law that was developed by

Greg Lambert (22:04)
Yeah.

Hannah Bloch-Wehba (22:05)
a legislature or a court. This is not Google trying to apply a ruling that was already made. It’s Google saying, “This is what we want you to do.” Law enforcement is so eager to get this data that they do what Google wants them to do.

And I think Google was right that a warrant is required, but that’s not the point. The point is that it’s not their job to decide. Right.

Marlene Gebauer (22:32)
They’re making the call.

Greg Lambert (22:34)
Mm-hmm.

Hannah Bloch-Wehba (22:35)
So I think this is a clear way in which it’s corporate policy that law enforcement has to get a warrant. It’s also the data infrastructure or the data governance choices Google is making internally, the collection by default, the indefinite storage, right? Those are the things that make it possible for law enforcement to get this information. So I think, in a clear way, it was Google that shaped the geofence warrant.

Then, in 2023, Google decided, “You know what? We aren’t going to do this anymore. It’s incredibly burdensome for us.” They turned off default storage of location information in the Sensorvault. That’s another example of Google determining whether this is okay or not.

Marlene Gebauer (23:32)
Their choice, yeah.

Hannah Bloch-Wehba (23:33)
Like, okay, after several years of complying, now we’re going to change our data governance practices, and this isn’t going to be possible anymore. Again, I think that was a win for privacy. I don’t think it’s a win for democratic control to have individual tech companies determining, based on what’s convenient for them, whether they’re going to comply with a warrant, what legal standards should apply, and whether the government should be able to get this data. That makes me uncomfortable.

Greg Lambert (24:12)
Well, we talked recently about the OpenAI and Hugging Face situation, and one of the comments we made was, if this were an individual, this would be a crime.

Hannah Bloch-Wehba (24:24)
Absolutely.

Greg Lambert (24:25)
So this situation where Google decided, “Okay, I’m going to turn this off.” There was a situation, and it’s a little apples and oranges here, but where someone, I think, was coming back into the country and tricked the government official into wiping the data from his phone.

Hannah Bloch-Wehba (24:43)
Yeah.

Greg Lambert (24:44)
Now he’s charged with a felony for doing this. So, individual crime. If the company did that, it would be a policy change, a change in internal policy.

Hannah Bloch-Wehba (24:57)
Right.

Greg Lambert (24:58)
So, yeah, it seems like we’re almost a little upside down.

Hannah Bloch-Wehba (25:04)
Yeah, I mean, to be clear, I don’t think that should be a crime. I don’t think it should be a crime to decide you’re going to delete your information. But I definitely don’t think we should have one set of rules that apply to big tech companies and a different set of rules entirely that apply to individuals. That seems bad to me, and I don’t think we’ve appreciated the extent to which that’s the case.

But it is true that these companies are making decisions about, you know, how to test their products that may end up violating the law and they’re not being held to account the way that an individual might be. And I think we should conceive of that as a potential problem, right? That the law might not be applied evenly.

Marlene Gebauer (25:56)
Then it’s about the data they collect and how they choose to use it, and there’s no oversight on that.

Hannah Bloch-Wehba (26:03)
Well, I think that, to the extent oversight exists, for a long time we’ve assumed that we’ve opted to comply with the terms of service as they’re delivered to us, right? So if the

Marlene Gebauer (26:16)
Yes, our click agreements. Yeah, because everybody reads those.

Hannah Bloch-Wehba (26:22)
thing that makes the difference is whether I’m an individual or a company that produced 40-page terms of service. That seems obviously laughable, but I think that is part of what’s going on.

Greg Lambert (26:37)
Well, again, another one we could have a complete show on. As if you haven’t done enough writing, one of the other papers you wrote was...

Marlene Gebauer (26:46)
There’s more. There’s a couple more.

Greg Lambert (26:50)
information. Okay. Yeah.

Hannah Bloch-Wehba (26:50)
To be fair, it’s been seven years since you had me on the podcast.

Marlene Gebauer (26:54)
Yeah.

Greg Lambert (26:55)
So one of the other papers you wrote was “Information Law Pluralism.” First, do you mind defining what you mean by information law here?

Hannah Bloch-Wehba (27:06)
Yeah, so by information law, I mean the body of legal structures that governs how knowledge is produced, validated, and disseminated. A lot of the time when we talk about information law, people mean one part of it, like privacy law or public disclosure law. I’m trying to bring that all together and say these are all ways in which we govern flows of knowledge.

Greg Lambert (27:38)
Well, in that, you talk about emerging tools, audits, impact assessments, document requirements, and independent assessment. Which of these has the greatest potential for producing meaningful accountability at the end of the day?

Hannah Bloch-Wehba (28:02)
Yeah, so part of the impetus for writing this paper was that I took a step back. I decided to do a project where I looked at every piece of proposed and enacted state legislation about AI. This was a stupid project. It was a couple of years ago.

Greg Lambert (28:18)
So small, just a little weekend project.

Marlene Gebauer (28:21)
Well, a fun little thing to do, yeah.

Hannah Bloch-Wehba (28:24)
It was a couple of years ago. It wasn’t quite as crazy as it would be to try that project now. But I thought, what are they trying to do? My sense was that there were some broad-stroke similarities and some important differences, and I wanted to capture those. As I looked at these laws, I realized they’re all trying to do the same thing, which is, in different ways and for different audiences, force the AI providers, right? Whether that’s a company, an individual, or a system is often unclear in the legislation, but force these providers to generate and share certain kinds of information about their products. It might be shared with individuals who are affected by those products, like when you apply for a job and have to use an AI interviewing system.

It might be shared with the public, lawmakers, or regulators. We have all these different audiences for all these different kinds of information. So what are they trying to do? That’s the thing I think lawmakers were genuinely uncertain about. I wrote the paper to explain that these different ways of producing information serve different purposes and should be used in complementary ways. So I’m going to avoid your question entirely and say, I don’t think there

Greg Lambert (29:49)
Well done.

Hannah Bloch-Wehba (29:50)
is one tool that will be the silver bullet for accountability. Luckily, I didn’t think until pretty recently that we had to choose one tool because I thought it would be up to the states to regulate AI as they saw fit, so we were going to have federalist experimentation. Little did I know.

Greg Lambert (30:13)
I think I found the flaw in your logic.

Marlene Gebauer (30:15)
Yeah.

Hannah Bloch-Wehba (30:19)
I might still be right about that. I think the claim to state capacity to regulate AI is strong, and that, but for threats to preempt that experimentation through litigation and threats of cutting off different kinds of aid to the states, we would be seeing a lot of that experimentation in action. The last couple of years have tried to deter it from happening. I don’t think you have to choose one of these tools. They all need to work together. I think the public needs a certain degree of information about how these systems operate in practice.

We need expert insight into how they’re working. That’s researcher access to the under-the-hood data a lot of the time. I think it’s important for these companies to produce information directly to the regulators charged with overseeing their operations. Unfortunately, I also believe there’s a successful political campaign against these laws, and that’s going to stymie the development of legislation in this area. So this is a paper that got...

Greg Lambert (31:44)
Is that argument about national security? Because we’re hearing, well, if we regulate our AI companies, China and Russia are not going to regulate, and we’re going to lose the advantage. Is that the sole argument, or is there something more nuanced in that?

Hannah Bloch-Wehba (32:04)
I think there are two related arguments, and they’re familiar instances of this debate. One, AI is essential to national security. We shouldn’t do anything that’s bad for national security. Therefore, we shouldn’t do anything that might be bad for AI development because we need to win the arms race.

Greg Lambert (32:26)
Seems super logical.

Hannah Bloch-Wehba (32:27)
It’s super logical at a high level. The problem is that, first, China does regulate AI, so the whole idea that they have hands-off development and we need it too doesn’t work. Second, it’s this assumption that you need a totally deregulatory environment to safeguard national security, but we would never have said that about nuclear weapons, for example. So it seems like a strange leap. I’m also not sure you can win an arms race, but that’s neither here nor there. The other part is about the federal role. Having failed to pass AI legislation in the U.S. Congress that would preempt state legislation, we’ve instead had a series of threats by the president to convene a litigation task force to sue states that regulate AI in an overly burdensome way. We haven’t seen that happen yet, but many states recognize that the threat is real. We also have a litigation campaign dedicated to undermining these laws. So xAI has been suing in California and Colorado to undermine AI transparency rules, and they’ve been pretty successful. Colorado revisited and substantially weakened its laws partly in response to these threats. I think they all hang together as one ball of wax. Whether you call it national security or something else, it’s a successful campaign to get states to change their positions.

Marlene Gebauer (34:16)
Okay. One last pivot.

Greg Lambert (34:19)
Yeah, and you said it’s been seven years, but I looked at the dates. All of these are 2026 articles, so...

Marlene Gebauer (34:25)
Well, we’re trying...

Hannah Bloch-Wehba (34:25)
I had a good year.

Marlene Gebauer (34:27)
We’re trying to keep it fresh. You have definitely been busy. So in “Rethinking Federal Support for Journalism,” you argue that requiring platforms to compensate news organizations might deepen journalism’s dependence on tech companies. And you propose a national journalism foundation. How would you design something like that to support knowledge production while protecting editorial independence, resisting political interference, and ensuring that funding reaches public-interest and local journalism rather than primarily benefiting established media owners? This is an important question in my mind. I Hannah Bloch-Wehba (35:09)
Yeah.

Marlene Gebauer (35:10)
think our whole information and news dissemination platforms are struggling right now.

Hannah Bloch-Wehba (35:18)
Yes, and I think they’re going to struggle even more in the year to come. I came up as a press freedom lawyer. I spent four years litigating on behalf of journalists and media organizations, so this is an issue that’s important to me. I think a lot of news organizations are intuitively worried about taking money from the federal government, understandably, because they feel like it might compromise their independence. These proposals are floating around to enable news organizations to bargain with platforms for a share of the money they’ve been losing based on platform competition with their products.

I don’t think that solves the independence problem. I think you’re no more independent

Marlene Gebauer (36:14)
Just trading.

Hannah Bloch-Wehba (36:15)
if you take your money from Google than if you take your money from the government. Google is an extremely influential company, and we should want independent news organizations to report on influential actors, whether they are private or public. Right? So I think this is a bad structure, and I think it’s better to have a structure that’s public but independent in a meaningful way. What I’m thinking about is modeled on the National Science Foundation, which tried to thread some of these conflicts when it began funding university research and basic science.

Similarly, researchers were worried that if they took federal money, they would be captured by a federal policy agenda. The National Science Foundation dealt with that through institutions of independent peer review, essentially. Now, all of that is contested. I don’t mean to say it’s easy, make a National Science Foundation and call it the National Journalism Foundation.

But I think it speaks to the fact that these are issues we’ve dealt with before, conflicts between independence and federal support that we’ve been dealing with since 1952, when the NSF was first stood up. We’re revisiting them today with contemporary debates about science policy and research funding. I don’t think it’s impossible to manage. I think it’s important to direct federal funding and support to the kinds of news organizations producing news of genuine civic value, particularly in underserved areas. We have a lot of news deserts in this country. It’s hard for many Americans to access high-quality news. My first order of business would be geographically underserved communities.

Greg Lambert (38:29)
Yeah, that got me thinking, and this might be a little far-fetched, but you have the National Science Foundation that you talked about. Then there are other, not agencies, but other ways of distributing licensing, like ASCAP for the music industry.

Hannah Bloch-Wehba (38:49)
Yeah.

Greg Lambert (38:50)
Do you see there being some type of underlying control over how information gets distributed and then reimbursed?

Hannah Bloch-Wehba (39:04)
Yeah, I mean, I haven’t thought about that end of it. ASCAP is an interesting corollary that I hadn’t considered. The two poles I have thought about are the NSF on one side and the Corporation for Public Broadcasting, which no longer exists, on the other.

Greg Lambert (39:23)
Yeah.

Hannah Bloch-Wehba (39:25)
One of the things they have in common is that both have turned into significant political footballs because their budgets are appropriated by Congress. At the beginning of debates about the Corporation for Public Broadcasting, they considered an alternative, which was to fund CPB through an excise tax on television sets instead of through the appropriations process.

And that would

Greg Lambert (39:57)
Mm.

Hannah Bloch-Wehba (39:57)
have been the funding. I think about this because there are so many proposals to tax AI or take a golden share of AI, and then what do you do with that money? The conversation about supporting an ecosystem that includes journalism should be part of the question of how you tax AI and what you do with that fund. I could see a revenue source that comes from some kind of AI tax and removes at least some of the levers political actors have to bully these agencies or force them to adopt a political agenda that would compromise the integrity and independence of journalistic institutions.

So I think that’s one big question I have. Where do you get the money to support such an institution, and how do you make sure it’s not captured by political interests trying to strong-arm news coverage? I think that’s an important question.

Greg Lambert (41:12)
Whew. Well, on that happy note, let’s wind this down. We’ve started asking a new question before we jump into the crystal ball question. Hannah, you get to be one of the first people we ask this. What’s something that’s true for you today that wasn’t true one year ago?

Hannah Bloch-Wehba (41:39)
Well, I was going to say that a year ago we weren’t at war with Iran, and that’s true, but I don’t think it’s that interesting. I do think it’s important for the overall

Greg Lambert (41:50)
Right.

Hannah Bloch-Wehba (41:51)
question of how the relationship between tech companies and national security is changing based on the nation’s foreign policy. The thing that I think is interesting and emerging quite quickly is that there is a popular movement mobilizing around two things. One is data centers. Opposition to data centers is becoming a high-profile grassroots issue that I didn’t expect or see a year ago today, although I think the ball was rolling. The other thing is Flock surveillance cameras, which

Greg Lambert (42:33)
yeah.

Hannah Bloch-Wehba (42:35)
are everywhere and have been for several years, and are also galvanizing a wave of popular opposition. So the thing I would say, which is a positive spin on it, is that issues related to tech accountability and tech’s enmeshment in the state are much more visible to people than they were a year ago. That means the political conversation about these things is changing quite quickly, especially because it’s an election year.

Greg Lambert (43:08)
Yeah. I saw a story this morning where this guy sat underneath one of the Flock cameras and had his computer set up. The police kept coming by and checking on him because they thought he was trying to hack into the Flock camera. He’s like, “So I’m the problem, right? Not this?”

Marlene Gebauer (43:26)
Yeah.

Hannah Bloch-Wehba (43:28)
Yeah. I mean, I do think Greg Lambert (43:30)
Yeah.

Hannah Bloch-Wehba (43:31)
it can be a huge diversion of resources. Do you want the police going by to check on that guy again and again? Don’t they have something better they could be doing? That’s the question I ask about Flock.

Greg Lambert (43:43)
Apparently not.

Hannah Bloch-Wehba (43:45)
Yeah.

Marlene Gebauer (43:48)
Well, Hannah, it’s time for the crystal ball question. I like this new combo because Greg’s question is looking back to the present, and now we’re looking forward. What is the single biggest change you see coming that you think we need to prepare for in the next few years?

Hannah Bloch-Wehba (44:07)
Ugh, I am so bad at predicting the future, you guys.

Greg Lambert (44:11)
Everybody is.

Marlene Gebauer (44:12)
Everybody is.

Hannah Bloch-Wehba (44:15)
I hate being on record.

Marlene Gebauer (44:18)
The good thing is, if we wait another seven years, we don’t have to hold you accountable for it. But hopefully

Greg Lambert (44:24)
Ha.

Marlene Gebauer (44:25)
it won’t be another

Hannah Bloch-Wehba (44:24)
True.

Marlene Gebauer (44:25)
seven years.

Hannah Bloch-Wehba (44:26)
I mean, I think that from where I sit in higher ed, there are significant changes coming to higher education. I don’t know what they are. If you talk to my husband, he thinks our kids aren’t going to go to college because higher ed will have changed that dramatically in the next 20 years. I’m not sure whether he’s right or wrong because I hate to predict the future. I do think we have certain core assumptions about the kinds of things higher ed does that are going to be different. That’s partly because of technological change in AI. I think there are going to be questions about the role of higher education as the workforce changes, which I think is going to happen fast. I also think a lot of higher education institutions are changing from within. I do wonder what universities and research are going to look like in the next few years. For people who have kids in college, going to college in the future, or going to graduate school or law school, these are going to be different places than they are right now.

Marlene Gebauer (45:53)
Yeah, I agree with that. I saw something not too long ago. My alma mater dropped a number of degree programs and is narrowing its focus to certain ones. That could be a problem. We’re training people for their professions as opposed to broadening their education.

Hannah Bloch-Wehba (46:20)
Right. We’re doing that at the same time as, as you said, journalism is starting to collapse. Book publishing is collapsing. Search engines don’t serve the functions they used to serve. So the whole information ecosystem, how people learn, is changing quickly.

Greg Lambert (46:40)
Yeah. Well, again, on that happy note,

Marlene Gebauer (46:45)
Yeah.

Greg Lambert (46:47)
Hannah, I want to thank you for coming in and talking with us. This has been fascinating. If I had predicted how well Texas A&M Law School would do in its first 15 years, it’s skyrocketed, and I’m sure you have contributed to that as well. So thank you for coming in.

Marlene Gebauer (47:09)
Definitely.

Hannah Bloch-Wehba (47:10)
Thank you so much for having me. It’s been a pleasure.

Marlene Gebauer (47:14)
Thanks to all of you for listening, and don’t forget to like and subscribe.

Greg Lambert (47:18)
Yeah. Hannah, what’s the best place for people to find more of your content and learn more about your work?

Hannah Bloch-Wehba (47:30)
You can find my content on my website, hannahbw.com, or the Texas A&M website, and on SSRN, the Social Science Research Network.

Marlene Gebauer (47:43)
As always, the music you hear is from Jerry David DeCicca. Thank you so much, Jerry, and bye, everybody.

Cooper reserved the small windowless conference room and walked in to see the same mismatched chairs and overworked whiteboard. Good things have happened in this room over the summer, and he was a little superstitious about it, so he decided not to look too closely at that.

Jesse had already logged into the call and was on the wall screen when Cooper had walked in. Leo had also beat Cooper to the room and sat there hunched over a laptop, looking like he had a couple of late nights in a row. Maya arrived last with her legal pad under her arm and a printout paperclipped to the front.

"Before we start," she said, taking the stable chair, "I did some reading of my own this week, since apparently everybody else was doing it too. I made it as far as a phrase called semantic interoperability, and then I closed the browser. So I want it on the record. If anybody uses the word semantic in this room today, I'm walking out, and I'm billing the hour to one of you."

"Noted and agreed," Jesse said from the screen. "Cooper. Before we get going here, did you keep your promise?"

"I half kept it." Cooper set his notebook down on the table. "Tuesday night I opened up the public viewer. I told myself I was only going to look at the front page. I ended up searching for one term, and I read through the definition on it, and then I made myself close the laptop and walk away. So I'm counting that as keeping the promise. Most of the promise."

"That's about what I figured would happen when I asked for it." Jesse pulled up his screen share. "Alright. Let me show everybody what Leo has been buried in this week.”

The Walkthrough

Jesse’s demonstration turned out to be a website. Down the left-hand side ran a list of categories. He clicked on one of them and it unfolded into branches. He clicked again and those branches unfolded into even more branches.

“This is SALI,” he said. “The Legal Matter Specification Standard. It’s built and maintained by a nonprofit standards group, and law firms and legal vendors have been contributing to it for years now. What you’re looking at is a taxonomy of the work of law. The areas of law are in here. The services a lawyer actually performs are in here, and it turns out that’s a different thing than the area of law. The industries that the clients belong to are in here. Thousands of terms in total. And here’s the part I want you to slow down on. Click on any one of them.”

He clicked. A term opened up, and underneath it was a definition, written out in plain sentences. Below the definition was a list of synonyms, all the other names that people out in the wild use for the same thing.

“Every term in here has a definition attached to it. Every term carries its synonyms, so the system understands that three different names can all point at one concept. And every term has a stable code behind it, which means software can refer to it without getting into arguments about spelling. If you want, you can export the whole thing as a spreadsheet, or format it for a machine to read it directly. The firm can adopt it and adapt if for its own internal use. The license allows you to do that.” Jesse opened up another page. “This is a definitions section for the whole industry, and it’s set up for you to start using it today.”

Maya was quiet, reading the screen through her glasses. Cooper knew the expression. It was the one she wore when an opposing brief turned out to be better than she expected.

“Now,” Jesse said, “Cooper, go ahead and do the thing we talked about.”

Cooper slid copies of a form across the table. It was the firm's own matter intake form. The form every new matter starts from, and a form Cooper had personally filled out wrong at least once in his career. He printed three copies that morning.

“Look at the questions on our form,” he said. “What area of law. What kind of work is being done. What industry is the client in. Who is the client. Now look up at the categories in the standard on the screen. It’s the same list. Our intake form has been asking ontology questions for as long as any of us have worked here. We collected the answers as free text, and nobody ever graded the quiz. That spreadsheet I spent Monday buried in, that’s what an ungraded quiz looks like after three years.”

Maya set the form down next to her legal pad and studied it for a moment.

“So, the industry already wrote the answer key,” she said.

“The industry already wrote the answer key,” Jesse said. “And I have been waiting all week to watch you figure that out.”

The second folder, again

Leo turned his laptop toward the room and took over the screen share. He clearly had not rehearsed any of what he was about to say. Somehow that always worked in his favor.

“Okay. So, you all remember the demo with the three folders. The second folder was the real NDA from the spring. The one with the non-solicit hiding in paragraph eleven. My harness caught it back then because non-solicits happened to be one of my six positions. I wrote those six checks myself. Cooper asked me to rebuild the clause checks, so they run against a real vocabulary instead of running against my list. So that’s what Jesse and I did this week. And now I want to show you that same folder from the demo, run through the new version.”

He ran it. The log wrote itself down the side of the screen the way the room had watched it do before. The status page came out in its same fixed shape. And there in the findings was the non-solicit, flagged the same way it got flagged in the demo, except this time the flag carried a term from the standard right next to it, and the term carried a code. Leo clicked on the code, and the standard’s own definition of a non-solicitation provision opened up on the screen for everyone to read.

“It caught it, just like before,” Leo said. “The difference this time is that the catch now has a name. Anybody who uses the standard means the exact same thing that we do. If we send work over to co-counsel, their system can read our tags without anyone needing to get on a phone and discuss it. The next time we have a vendor tells us that their system finds non-solicits, we can get them to tell us exactly what that means. Then we can test it against our non-solicits instead of arguing semantics with them.”

“That’s the demo,” Maya said. “Now what’s the confession? You’re making the face you make when there’s a confession.”

Leo took a breath.

“Well, while I was mapping my six positions over to the standard’s clause list, I found entries in the standard for things my six positions never covered. Two of them matter. There’s a whole category in there for residuals clauses. Those are the ones that let the other side keep using whatever their people happen to remember after the deal. And there’s a category for the non-disparagement language that’s been creeping into NDAs the last couple of years. My old harness ran perfectly every time, and it was never once looking for either of those. I went back through our tracking spreadsheet last night. We have signed NDAs in that folder with residuals language in them, and my summaries called them standard.”

Everyone remained silent and waited for Leo to continue.

“My list had six positions on it because six was what I knew about,” Leo said. “The vocabulary knew about the ones I didn’t. That’s the part I can’t stop thinking about. I thought we were checking the documents. The dictionary went and found the gap in me.”

The cross-examination

Maya flipped over to the page of handwritten questions clipped to her printout. Cooper settled back in his chair, because he knew what this part looked like, and he had learned to enjoy it.

“I’ve got three questions, and I want honest answers,” she said. “Who wrote these definitions, and why should we accept them? Everything Leo just showed us runs on trusting the book. Think about what that means. An associate who gets something wrong gets it wrong one document at a time, and eventually somebody catches it. If the book is wrong, then every harness reading the book gets it wrong the same way, in every office, on the same day. And the log will show that we did all of it very thoroughly.”

“That’s a good first question,” Jesse said. “The answer is that the SALI standards body wrote them. They did this out in the open, over a number of years, with lawyers and vendors debating the wording, and a public process for proposing changes. You can go read the history behind any term. And yes, you're going to find definitions in there that you would have written differently. When that happens, you propose the change through their process, or you map around it deliberately, on the record. What you don't ever do is quietly keep a private meaning of your own and tell nobody.”

“Now for my second question. What happens when the definition changes? And they will change,” said Maya. “The law changes. So, the book puts out a new edition, and our machines are still checking everything against the old edition. Or half of our systems get updated and the other half don't. If that happens, we've just rebuilt the eleven-entry list all over again, only one level up this time.”

“Versions,” Jesse replied. “The standards get updated as releases, just like software does. That keeps the code stable, and all the changes get documented when a new release comes out. You pin your system to a specific version. Similar to what you did when you froze the survey wording. You make the decision to update, and that is tagged to a specific date. Nothing is drifting around on its own in the background.”

"Then that's a condition, and I'm writing it down now." She wrote as she talked. “Definition changes get handled the exact same way we handle rubric changes. There’s a version, there’s a date, and somebody reviewed it. Nobody gets to edit the dictionary quietly. And if this firm ever needs to disagree with the standard on something, then that disagreement gets written down and mapped, so we always know where we deviate and why we deviate.” She finished the sentence on the pad and looked back up. “Third question. Cooper. On Monday’s call, Jesse gave you something that he wanted handed over to me word for word. I’d like to have it now.”

Cooper opened the notebook to the page where he had written the three numbers down in a column.

“A group of researchers ran business questions against a company’s own data, three different ways,” he said. “With the model working alone against the raw data, it got sixteen percent of them right. When they rebuilt that same data as a knowledge graph with an ontology behind it, the same questions came back at fifty-four percent. And when they set it up, so the ontology checked every query before that query was allowed to run, it got up over seventy-two percent. When that last version couldn’t get to an answer inside of the rules, it came back and said it didn’t know. It didn’t try to make up something confident just so it would have an answer to give. The word I underlined twice in my notes was unknown.”

Maya nodded. Similar to the nod she gave the third folder.

“Every system I have ever cross-examined fell apart in the same spot,” she said. “The spot where it should have said I don’t know, and it went ahead and said something confident anyway. Fine. I’m in. And while I’m being agreeable, I’m going to put one more thing on the record, because somebody in this room ought to say it out loud. Every check we hand over to the book is a check that some associate out there stops learning how to do by hand. That’s the trade. It's mostly a good trade. But somewhere in this firm there should be a running list of the things we've decided our people no longer need to know how to do, so that at least we chose all of it on purpose.”

No one even tried to answer that one. Cooper wrote it down in his notebook. Writing things down had become his default answer for most things lately.

The units

“Here’s what I need today,” Cooper said, moving over to the whiteboard. “I didn’t just call this meeting to admire the SALI website. I have a real problem to address. Actually, a list of eleven problems that I found this week.”

He drew a rough scale up on the board. The old-fashioned kind, with the two hanging pans. Jesse laughed. Maya did not.

“We got the committee to approve the evaluation harness already. I had to find forty closed matters. Every model, every vendor, and every harness we build from here runs through those same forty trials. We froze that rubrics to give us the basis to run these tests. I get the honor of being the person whose name gets put on all of that. And then I spent this week finding out that I can’t reliably pick out the forty, because the words we use to describe our own matters don’t agree with each other. The practice areas came in as free text. A lot of the industry fields never got filled in at all. The rubric wants to ask whether the work served the client’s industry, and out of my first sixty candidates, nineteen of them can’t answer that question for me.”

Under the scale he wrote: what are the units.

“Here’s my modest proposal. We are not going to try and fix the firm all at once. We start with the forty matters, and then we tag them using the standard’s terms. The are of law comes from the standard’s list. The service comes from the standard’s list. So does the industry. The attorneys will confirm the tags during the same interviews we conduct for the rubrics, which takes about another minute of their time while we have them. From then on, the rubrics inherit those terms. Every tool that runs the gauntlet gets described in the same vocabulary as the trials it has to survive. And when a vendor tells us their product handles our kind of work, we can finally ask them which codes, and then go check.”

“We send my harness through first,” Leo said. “That was the deal. Now it can go through speaking the same language of the test.”

“Forty matters, all tagged once, by people who already have to be in the room,” Maya said. “That’s a pilot I don’t have to worry about defending to anyone. Go ahead and write it up.” She turned and looked at Jesse on the screen. “What does this buy us at the end of the year that we don’t already have right now?”

“You get a scale with the units printed on it,” Jesse said. “You built the underlining structure this summer. Right now, it only weighs thing in opinions. After this is set up, and the firm says a tool passes, you got forty specific trials, describe in industry accepted terms. That’s something you can put in front of a client, or a judge, or a vendor, and it passes for each of them.”

The blank line

The entire write-up took about twenty minutes, and most of it lined up with what the committee already approved a few weeks back. Tag the forty matters against the SALI standard’s current release. Pin it to the version. Attach Maya’s two conditions, the mapped deviations and the no silent edits. Leo’s harness runs through the gauntlet first, with the new vocabulary.

Then they got to the hard part. The form the committee used had a field on it. The same filed it always had, and it couldn’t be left blank. Owner.

Cooper looked at it for a while. The eval harness was his, and the forty tags could live under the card he had already signed. That part was easy. But the tags were the first forty entries in something bigger than the tags. As of this afternoon, the firm was a firm with a dictionary. Dictionaries need tending. Cooper had watched taxonomies rot before, back in a career he had mostly stopped mentioning, and he knew what that slow rot looked like from the inside. Whoever owned the firm’s language over the long run needed it to be their whole job. Nobody in this room was that person.

“I can carry it for the forty matters,” he said. “I don’t think I should be the one to carry it past the first forty. Frankly, I don’t think any of us know who that should be right now.”

Maya stepped forward and said, “I’ll take this to the committee and have them make a decision on this. Cooper’s right, this is above even my paygrade. Can everyone live with that for now?”

Everyone could.

Cooper remained in the room after the others made their way out. The whiteboard held the drawing of the old scale with its two hanging pans, and the four words he had written underneath it. Off to the side was the tree that Jesse sketched while he talked, the branches splitting off into more branches. Cooper snapped a picture of the board with his phone and sent it to the printer down the hall. He cut the printout down to size and slid it into the notebook, on the page across from the grid photo from a month ago. Jesse’s six boxes were two pages back from that one. The notebook now carried all three drawings. The boxes, the grid, and the tree.

By the time he reached his desk, the late afternoon had turned into early evening, and the floor had mostly cleared out around him. He pulled the notebook back out one more time and wrote in the final lines for the day.

The intake form was a vocabulary quiz the whole time. Nobody ever graded it. Starting with forty matters, we grade it.

Then right below that:

Third entry in the 2027 file: the fences are getting dictionaries. We built the scale last month. Today we agreed on what the units mean.

Cooper connected to the call first thing in the morning. Jesse was already on and waiting for him, holding a printed copy of Cooper’s list up to the camera. He had printed it out at some point the night before, because there were already handwritten notes in the margins, along with what appeared to be a coffee ring over entry number seven.

“Eleven entries,” Jesse said. “I want you to know that I’ve seen this list before. Different versions of it. I saw it at a bank, at an airline, at a hospital chain, and one time at a company that makes salad dressing. Every company has this list. Almost none of them ever write it down. You actually wrote yours down, so today is going to be a good day for me.”

The lab behind Jesse was quieter than normal. The big whiteboard had been mostly erased, except for one corner where somebody had drawn a robot. Somebody else had come along afterward and given the robot a speech bubble that said WORKS AS DESIGNED, and then a third person added a question mark to it in a different color of ink.

“I watched the professor’s video twice,” Cooper said. “I took my notes by hand, which I think he would have appreciated. I also need to confess something up front. I actually took a class on this back in library school. Controlled vocabularies, taxonomies, all of it. It was a required class. I memorized just enough of it to pass the exam, and then handed it all back to the school on my way out the door.”

“And now here we are, twenty-some years later, and the exam showed back up at your desk.” Jesse tapped on the printed list. “Alright. You asked the question, so let’s walk through it the right way. What is an ontology, and what does it have to do with these offices we’ve been building all summer.”

The defined terms

Jesse walked over to the whiteboard, and then did something Cooper didn’t expect. He wrote one single word on the board. The word was Agreement.

“You’ve read more contracts than I’ll ever read,” he said. “Where does every serious contract start?”

“Definitions.”

“Definitions. Sometimes two pages of them, sometimes twenty. Agreement means this. Confidential Information means that, and it specifically does not include these four other things. Business Day means a day the banks are open in New York. Nobody ever reads that section for fun. But when a fight breaks out over the contract, that’s the section everybody runs back to.” Jesse wrote a second word under the first one: Matter. “An ontology is a definitions section. That’s really the whole secret to it. It’s a definitions section for your firm’s entire world, and it’s written carefully enough that software can go look things up in it. What things exist. What relationships those things are allowed to have with each other. What values their properties can take on. The professor traced his version of this back to Aristotle. You’ve had a version of it in the front of every deal you’ve ever closed.”

Cooper wrote defined terms into his notebook and drew a box around it. Then he felt a little ridiculous for drawing a box around the concept of definitions.

“Now connect it back to the office for me,” he said. “Because that’s the part I couldn’t get to on my own. We built the walls this summer. We built the doors, the clock, the log, and the name on the door. So where does the dictionary go?”

Jesse redrew the office from memory, a smaller version of it this time. The stick figure genius at the desk with the boxes drawn around it. Then he added one new item to the drawing. A shelf, placed just inside the door, with a single book on it.

“It goes right there,” he said. “On a shelf by the door. And here’s the difference between the book and the boxes. It took me a while before I could say this part cleanly, so bear with me on it. The six boxes all answer the same question, which is whether an action is allowed. Can the genius open that cabinet. Can it spend that dollar. Can it send out that report. The dictionary answers a different question. It answers whether the words inside of the action actually mean anything. The office will happily approve a perfectly formatted request to file a status of probably shipped. The budget is fine, the door is allowed, the log is running. Every box says yes. And the action is still nonsense. The only thing in the whole building that can catch the nonsense is a book that says what a status is allowed to be.”

The graph on the whiteboard

“Give me the firm’s version of this,” Cooper said. “The professor used refunds and shipping orders. I did my own translations in the margins yesterday, but I want to hear yours.”

Jesse wiped off a clean section of the board and started drawing circles. In the first circle he wrote the name PRICE, and explained that she was an invented associate for whiteboard purposes only. Then he drew a second circle labeled MATTER 4417, and an arrow connecting the two with the word staffs written above it.

“One line of data. Price staffs matter 4417. Now watch what a dictionary with rules attached does with that one line. If the dictionary says the staffing relationship only runs from a timekeeper to a matter, then the machine can work out on its own that Price is a timekeeper. Nobody had to enter that anywhere. It also now knows that 4417 is a matter. That’s two facts you never typed in, and both of them came out of one line, because the line had a definition standing behind it. And that’s the pleasant version of this. The less pleasant versions are already on your list.” He drew a third circle for another matter and then connected both of the matters to a single client circle with lines labeled billed to.

“Here’s the matter that got opened twice. Drawn out now as a graph. In the dictionary, we have a rule that states we have a rule that a matter gets one opening and one billing client. That’s the complete rule. When matter is submitted for a second opening, the machine has a contradiction to deal with. It doesn’t need to be clever to catch it. Think about how it actually played out at the firm. Your billing team noticed the numbers looked off, and that was months after the fact, and it took a person staring at a report to get there. The machine gets there a different way. The rule says the two openings can’t both be true, so it refuses the second one, right there, the same day somebody tried to open it. Conflicts would have only run the one time. Nobody would have ever needed to write see other file.”

“What about the wall?”

“That might be my favorite one, because your industry already thinks this way and just doesn’t realize it.” Jesse drew two clusters of circles with a wide gap between them. "You define the two sides of an ethical wall as disjoint. Formally disjoint. The dictionary states that no person can ever be a member of both of these sets at the same time. Once that's written down, a document getting routed from one side over to the other stops being a permissions question. It becomes a statement that cannot be true. Like a number that is both odd and even at the same time. Right now, your walls hold because of access controls, plus everybody behaving themselves. This writes the wall down as arithmetic."

Cooper looked at the collection of circles and arrows across the board and it made him think of his eleven thousand rows.

“The professor called it ‘data as graphs,’ he said.

“That’s a good phrase. The things, these lines between the things, and a book that tells you which of the lines are legal.”

Two entries

“Okay,” Cooper said. “Now on to the third folder.”

Jesse had clearly been waiting all morning for this part.

“You mentioned in your email that you thought you knew the answer,” he said. “So go ahead. Give me your guess.”

“My guess is that when the harness refused the supply agreement, it had to be checking that document against something. Some kind of a definition of what an NDA has to contain. My guess is that somebody sat down and wrote that out by hand. And by somebody, I mean you and Leo.”

"It was mostly Leo, actually, and that's the part I'm proudest about." Jesse held up two fingers to the camera. "Two entries. That is the entire dictionary living inside that harness. Entry number one covers what an NDA is. It has to contain confidentiality obligations. It has parties, it has a term, and it has the six positions the firm cares about. Entry number two roughly covers what an NDA is not, and that's how it recognized the shape of a supply agreement even while the filename was lying to it. We wrote both of them in one evening. So, when the third folder ran, the harness looked that document up in a two-word dictionary, the document wasn't in there, and everything that Maya loved about that refusal came from that one lookup."

Cooper put his pen down on the desk.

“So the most persuasive thing this firm has seen all summer,” he said, “was a dictionary with two entries in it.”

“And now you can see the problem, because I just watched you do the math. Two entries took us one evening. Your firm’s world is not two entries. It’s every practice area, every matter type, every document type, every role a person can hold, and every status that anything can possibly be in. That’s thousands of entries. They all have to agree with each other, and then somebody has to keep every one of them true while the firm keeps changing underneath them. Writing all of that down by hand is exactly what the professor’s generation attempted back in the eighties. That’s the graveyard he showed you at the end of the video. The rules themselves worked fine. Nobody could keep up with maintaining them, and that’s what buried the whole field.”

The economics

“So why is all of this coming back around now,” Cooper asked. “That’s the part of the video I keep going back to. He got an entire conference full of engineers to care about something that famously failed.”

“Because the economics finally flipped. And there’s actually a study that walks you through the flip.” Jesse pulled a paper up onto his screen and turned the camera toward the monitor. All Cooper could make out was a wall of small text. Jesse realized it too and swung the camera back around. “You’ll just have to trust me and read it later. A group of researchers took a set of natural language questions. Real business questions, the kind a manager would actually ask about their own company data. They pointed a model straight at the company’s database and had it answer them. No dictionary, no graph, just the model working against the raw tables. It got about sixteen percent of the questions right. Sixteen. Then they took the exact same data and rebuilt it as a knowledge graph with an ontology behind it. Asked the same questions again, and it scored fifty-four percent. Then came the part that I keep telling everybody about. They set it up so that the ontology checked every query the model wrote before that query was allowed to run. If the query was fixable, the system repaired it and ran it. If it wasn’t fixable, the query never ran at all. That version scored just over seventy-two percent. And there was one more detail buried in that study that I need you to hand over to Maya, word for word. The checked version learned how to say that it didn’t know. When it couldn’t get to an answer inside of the rules, it just said that, and it didn’t manufacture something confident for the sake of having an answer to give.”

Cooper wrote the three numbers down in a column, and underlined the word unknown twice.

“The model never changed,” Jesse said. “It was the same model through all three of those rounds. Everything that improved was in the material built around it. You’ve already heard me give that speech once this summer about the office. This is me giving the same speech about the book on the shelf. And this isn’t some lab curiosity either. There are entire companies out there now selling this exact layer. Palantir built a very large business on roughly this idea, an ontology of the customer’s world with the model working down inside of it.”

“What about the maintenance problem? The graveyard?”

“Still real. Dictionaries rot. Your firm’s dictionary rotted, and that’s what your eleven entries actually are.” Jesse tapped the printed list again. “The one thing that’s different this time around is that the dictionary finally has some help. The agents are out there hitting edge cases all day long, and it turns out they’re very good at noticing when a definition is missing or wrong. People are building systems now where the agent proposes the update and a human approves it. I’m not going to promise you that scales yet. What I am telling you is that the librarians of this thing may finally end up with a staff.”

Cooper let the librarian comment go by without a response, and privately enjoyed it.

The name

The math was what finally made Cooper ask his question out loud. He looked down at thousands of entries written in his notebook with a bracket around it, and asked whether the firm was seriously going to have to write its own dictionary of the entire legal world, from scratch, by hand.

“That’s the question I’ve been waiting on,” Jesse said. “And the answer is no. You don’t. The web figured this out twenty years ago and built shared vocabularies, so that every website didn’t have to invent its own from nothing. The industries next door to you have theirs. And legal, very quietly, while everybody was busy watching the agent demos, went out and built one too. It’s real, it’s open, and the definitions are public. You could read them this afternoon if you wanted to. A standards group has been working on it for years. The areas of law, the actual services lawyers perform, the industries the clients are in, all of it, with definitions attached, in a format a machine can consume. Your intake form has been trying to become this thing its whole life.”

Cooper had his pen ready. “What’s it called?”

“SALI. The standard itself is called the Legal Matter Specification Standard, which is a mouthful, so mostly everybody just says SALI.” Jesse spelled it out for him. “And now I need a favor from you, and I’m serious about this one. You are going to want to go spend tonight reading through the entire thing, because that’s who you are. Don’t do it. Bring Maya and Leo in on Thursday and let me walk all of you through it together, live and on the screen. I have never had the chance to show a lawyer their own intake form inside of a public standard before, and I’ve been looking forward to Maya’s face for two days now.”

Cooper agreed to the favor, and knew immediately that he was only going to half keep it.

There was one more thing to do, and he did it before he closed out of the call, because this summer had taught him the right order of operations. He messaged Leo directly. He wanted the NDA harness up on the screen Thursday as well, but modified, so that its clause checks would run against a real vocabulary instead of the two entries. And he wanted Leo to make that change himself before Thursday, without seeing a standard that Cooper hadn’t even seen yet either. Leo’s reply came back in under a minute, the way Leo’s replies do. It said he was already halfway through the docs. Then a second message came in right behind it: wait, there’s a public viewer?

Cooper closed the laptop and opened his notebook to a clean page. He drew the office one more time from memory. The boxes, the genius, and then the new shelf by the door with the book on it. Under the drawing he wrote out the day’s line.

Jesse added one new thing to the genius’s office today. A dictionary, on a shelf by the door. Every contract I have ever read starts with its defined terms. It took me until this morning to ask where the firm keeps ours.

Then beneath that, in smaller writing:

The most convincing machine this firm has ever seen was working off of a dictionary with two entries in it. On Thursday we find out what it reads next.

Cooper’s floor had entered the quiet season. The summer associates attended their last social happy hour event a week ago and had turned in their badges as they headed back to school for their 2L or 3L year. There would be a few weeks before the next batch of fall associates arrived. It was also the period where more emails were answered by out-of-office auto-responses than actual replies. This is the little stretch of a few weeks where he considered this to be the transition to the firm’s true new year, when the office empties out enough for a person to really have time to think for themselves.

Cooper planned to use this time to knock out what he had assumed to be the easy part of his new job. His name was on the card for the new evaluation harness, with quarterly review requirement that he had no intention of missing, and the first deliverable was the trial set. Forty closed matters were assigned. Billed, paid, and no additional litigation troubles. The rubrics would be built from these forty matters, gathered from the attorneys who worked on them, and then frozen in the same way they did the surveys to the firm. That was the design that Cooper, Maya, Leo, and Jesse set up in the windowless conference room. Now all he had to do was actually select the forty matters.

He sent a request to the accounting department for an export of closed matters going back three years. It came back the same afternoon. A spreadsheet with eleven thousand rows. Within two hours of scrolling through the spreadsheet, Cooper understood that picking which forty matters was not going to be the easy part at all.

The export

The initial problem started in the practice area column. The firm had an official list of practice groups. Cooper had actually helped update it two years ago. What the intake system collected, however, was a text field, and text fields are the place that official lists go to die. The same corporate work showed up as M&A, as Mergers & Acquisitions, as Corporate - M&A, and even one weirdly labeled as m+a misc. Those were four descriptions of the same kind of matter, written by four very busy legal assistants, on very busy days. The system just accepted every one of them without any pushback.

The industry column was worse. Many of the cells were just empty. Of the first sixty matters that Cooper flagged as candidates, nineteen of them had nothing in the field at all. A rubric that ask whether the work product served the client’s industry can’t do much with a blank.

Then there was the client he found filed under three differently spelled names. Same client. Three versions of the name, two of them abbreviated differently, all three carrying separate matter histories. The only reason Cooper caught that one was that he knew this client and wondered why the matter count was so low.

The one that made him put down his coffee and shake his head was a pair of matters that turned out to be one matter. It was opened twice, eight days apart, by two different assistants in different cities. Two different client matter numbers. Both went through conflicts. Both were staffed. Somebody eventually noticed, because the billing discrepancies. The second matter was closed with a note saying see other file. The note did not actually give the matter number.

After working on the list for a few hours, Cooper flipped open his notebook and started a list. He titled it: Ways we disagree with ourselves. By the time he wrote out what he already knew, he had eleven entries. He stared at it for a few minutes and then wrote right below it. The evaluation harness needs forty client matters we agree on. Before that, we need to agree on what an actual matter means.

At that point, he didn’t realize that a Berkeley Computer Science professor was about to tell him the same thing.

The video

Cooper had first seen reference to the video on LinkedIn. Then he saw it in a newsletter he subscribed to under the title of Ontologies are making a comeback. That didn’t catch his attention because quite honestly, he wouldn’t have been able to define ontologies if you held a gun to his head. He did remember that he’d covered the topic in library school, but that was a lifetime ago, and probably a required class he really didn’t want to take in the first place. Then the video was forwarded by two people within a few minutes of each other. One of them was a client he occasionally talked tech with. When the firm’s clients start sending you tech videos to watch, you tend to watch them, even if you don’t want to.

The video was twenty-one minutes long and Cooper watched it at 1.25x speed. It was from an AI conference that Cooper had briefly considered going to earlier that year, even though it wasn’t a legal industry conference. The professor opened up with a sort of hippy-artist’s creed, borrowed from a nun of all people who taught art in the sixties, and made famous from an avant-garde composer a few years later. “Nothing is a mistake. There’s only make.” As Cooper let that bounce in his head, it took on a Yoda-like feel to the way it sounded.

The professor pulled on this idea to the five thousand engineers sitting in the room. Get your hands moving and learn by making things. He asked them to stop typing out their notes and get back to using a pen, because typing makes your brain think about the keyboard, and when you write it out by hand, your whole intelligence shows up and participates.

Cooper looked over at his notebook and pen lying there next to his lunch. He felt sufficiently validated, and he moved the speed of the video back to 1.0x speed.

The argument

Cooper reconstructed the Berkeley professor’s case after watching the video, and rewatching parts of it a second time.

The thinking behind AI agents comes from the decades old dream that computers can play out situations, make decisions, and act out the scenarios. Large Language Models and Generative AI tools are sold at trade shows that sell this dream. Cooper watched it first-hand in Las Vegas when the same claims investigation ran twice.

Ontologies go back millenniums. The professor traced it back to at least Aristotle, who wanted to formalize a method of sorting what existed into categories. The modern version of ontologies got a working definition back in the nineties. Cooper wrote it down: a formal, shared way of writing down how a domain fits together. Stripped down to the basics, ontology is simply a dictionary with rules attached to it. Here are the things that exist in our world. Here are the types of relationships they are allowed to have. Here’s their properties which they carry, and the values that each of those properties can take on. These are written down in a way that a machine can check its work against it.

The data exists as a graph. The things, and the lines between those things.

The next part in the talk made Cooper set down his sandwich. The model by itself is a phenomenal talker, locked in a room. It can reason, it can propose ideas, it can advise you on what to do next. It cannot actually do anything. Cooper had heard this type of philosophy before from Jesse, furnished as an office location with a genius inside. It was somewhat comforting to hear the professor describe the same tenant.

That comfort was broken slightly when the professor continued on past the idea of the office location. The walls guide the genius on what it can touch. They do not define what the words mean. And this ambiguity is what causes agents to break in production. Not dramatically, but quietly, plausibly, in the gaps between what prompting instructions say and what a definition would enforce.

Cooper leaned in on the screen as the professor gave three examples of the kind of errors that a well-meaning prompt cannot reliably stop.

• A second refund issued on the same order.

• A payment sent to the support desk instead of the customer.

• An order status that read “probably shipped.”

Every one of those, he said, dies instantly when you have a one-line rule in a formal dictionary. A refund can only happen once per order, so the second refund is a contradiction, killed before it executes. A customer is a different entity than a support rep, so money can only be routed to one type of entity and not the other. When a status can only be one of three values so that things like probably shipped is immediately rejected because it’s not one of those three on the list.

Cooper moved his uneaten lunch to the side, pulled his notebook closer and turned back to the page he wrote: Ways we disagree with ourselves.

Our names for his errors

At the fourteen-minute mark, Cooper paused the video again and began writing in the margins.

The second refund on the same order. He began writing next to it: our matter opened twice. Two different offices, just eight days apart, they’d been issued different matter numbers and had both gone through the conflicts process. Nothing in the firm’s intake system knew to identify that a matter opening is the kind of thing that only happens once. No rule anywhere that could flag a second matter opening and recognize it as a contradiction in process.

And the payment that was routed to the wrong party. That one had an even colder translation, but Cooper wrote it down anyway: a document crossing to the wrong side of a wall. The walls were design to hold because the systems were set up to enforce who could reach what. But the professor’s point was a level deeper than just access. What saves you in this instance is that it knows the two groups are defined as not to overlap at all. That way membership in both is not a permission issue, it becomes an impossibility. And the machine is simply not allowed to do it.

Then came the probably shipped issue. Cooper looked it over and then began listing all of the “probablys” the firm had. Statuses that live in spreadsheets and in answers you get in the hallway, rather than in any system with a list of actual allowed values. Probably conflict checked. Probably filed with the courts. Probably the final version of the contract. And the one he knew Maya would not like, probably privileged. Right now, somewhere in the firm, a review status has the effective status of probably privileged, and there wasn’t a machine anywhere that would refuse it, because none of the systems are set up to say privilege review only has three outcomes and the word probably is not in any of those three.

He hit the spacebar to restart the video and one minute later pressed the spacebar again to pause it. This was the part that reminded him of Leo’s third folder, where a supply agreement was labeled as an NDA filename, and the harness rejected it. The harness saw that the structure didn’t match. Cooper placed in the margin: Ask Jesse about this. When the harness rejected the third folder, what was it checking against? It must have known what an NDA looks like. How did it learn that? Where does that knowledge live?

The graveyard

The last part of the video focused on history, and Cooper found this as interesting as the rest.

The professor discussed living through the eighties version of what were called expert systems. Where the plan was to write down every rule and then let the rules do the thinking. Entire companies were established on this idea. Lots of money was thrown at it. Even the professor’s own son studied Japanese in college because they were leading in this area of rules-based machines. But the systems became overwhelmed and couldn’t scale to the level needed to keep up with more and more rules. The whole field went into an AI-winter that took decades to thaw.

Cooper noticed that the professor chose the wording of his conclusion very carefully. Rules fail. Models drift. It takes a pairing to make it useful: use the model for the reasoning, and a written dictionary is kept outside of that model, to check the work. The dictionary keeps the model honest. The model makes the dictionary worth maintaining, because for the first time the dictionary has a tireless reader.

The video closed much like it opened, instructing everyone to go make something. Cooper looked at the final timestamp of twenty-one minutes and seventeen seconds. Then he refreshed the page and watched the entire presentation again. He had his pen ready to add to the notes. This was the first time in a long time that he remembered doing this.

The payoff of rewatching it was capturing a sentence that had been eluding him all afternoon. The firm had been building rules for a year. The binder he had is full of these rules, and the boxes Jesse drew on his whiteboard had turned the best of those rules into code. What the firm lacked was agreeing, in writing, in a single place, on what its own words really mean. The system was set up on guesses. The spreadsheet with eleven thousand rows was just an exported version of those guesses.

Cooper stayed at his desk while the floor went from quiet to empty. He opened up his email and sent Jesse a quick four sentence note. He’d found a video worthy of Jesse’s time. He had a list of eleven ways that the firm’s own data disagreed with itself, and he attached it. He wanted an answer to Leo’s third folder result, and thought he might know the answer. Lastly, he calendared time on Jesse’s schedule for first thing in the morning.

The response came back before Cooper had returned from the breakroom. Jesse had watched the video last week. He finished off the email with bring the list, this should be fun.

Cooper pulled open the notebook one more time before he left for the night. He flipped through to the page after the printed whiteboard photo, and wrote in the final lines for the day.

The professor spent twenty-one minutes listing the errors that instructions alone simply can’t stop. Every one of those were recognizable in the work we do in this building.

Then directly under it:

A rule a machine can enforce begins as a definition someone writes down. We have binders full of rules. Instead of finding our definitions today, I found a client spelled five ways.

LINKS:
Why Agentic Systems Need Ontologies - Frank Coyle, UC Berkeley

We welcome back Brad Blickstein, CEO at Blickstein Group, to discuss how private equity principles may provide law firms with an alternative approach to profitability, governance, and even long-term growth. Blickstein’s new book, WWPED: What Would Private Equity Do? was written to walk firms through how treating topics like pricing, technology, talent, and client relationships as part of the enterprise value instead of overhead expenses after year-end partnership distributions.

Pulling from Jae Um’s topics of Cream, Core, and Commodity framework, Blickstein talks about the legal work as the primary competitive battleground. Much like businesses that provide baked goods, firms have to separate the customized legal judgment from the repeatable legal processes, technology, and what alternative legal services providers offer. Law firm leaders should understand what scalable work is, begin building consistent systems to deliver that work, and truly professionalize pricing over relying upon what a partner’s gut tells them.

We also cover the Blickstein Group’s 2026 Law Firm COO Survey where technology adoption and investment ranks as the leading strategic initiative with 38.1% identified practice silos as the largest structural issue and 27% of COOs listed lack of operational authority as another prime issue. COOs are struggling with being tasked with modernizing law firms, but not given the authority to actually overcome the base issues of decentralized partnerships, competing incentives, and overall firm political structures.

Add AI into the mix, and the pricing question becomes even more important. Some two-thirds of the COOs surveyed confessed that they were not formally measuring any return on investment (ROI) in which they could later measure any law productivity or direct revenue increases. Blickstein points out that faster work in a billable hour model is not the type of math that law firms want to calculate, and that firms have to address this directly and redesign their overall pricing model on value received by the client, not hours worked by the lawyers. We all discuss the issues of alternative fee arrangements (AFAs) have face in the more than 30 years since Blickstein originally published an article titled “Alternative Billing Making a Comeback.” AFAs bring with it issues of shadow billing, client trust factors, and the need to express value not tied to the amount to time spent on the work.

We also break down the corporate buyer side and address the Blickstein Group’s 18th Annual Law Department Operations Survey which identifies AI pilot projects in corporate legal departments, but very few operational deployments. These may be tied to the long running issue of poor data hygiene along with business objectives that are not clearly tied to overall corporate strategy.

Brad gets to be one of the first to answer our new question of “what’s true today that wasn’t true a year ago?” A nice lead in to our Crystal Ball question. We cover AI token pricing and having to compete with the new “AI native firms” that are spinning up from former BigLaw partners.

Listen on mobile platforms: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Apple Podcasts⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ | ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Spotify⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ | ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠YouTube⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ | ⁠Substack⁠

[Special Thanks to ⁠⁠Legal Technology Hub⁠⁠ for their sponsoring this episode.]

Email: geekinreviewpodcast@gmail.com

Music: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Jerry David DeCicca⁠⁠⁠⁠⁠⁠⁠⁠⁠

LINKS

• Blickstein Group

• WWPED: What Would Private Equity Do?

• 2026 Law Firm COO Survey findings

• Law Department Operations Survey

• Cream, Core, and Commodity legal-work framework

• Legaltech Hub: The Arithmetic of AI, Tokens and Claude in Legal Work

• Legaltech Hub: Five Prompting Habits Costing You Tokens and Accuracy

• Legora introduces consumption-based pricing

• Kirkland & Ellis and its $500 million AI investment

• Anthropic Claude Code

Transcript:

Stephanie Wilkins (00:01)
The GenAI conversation has been advancing faster than a lot of people can keep up with lately, but some new developments have brought older concepts like prompting back into the spotlight, thanks to a trending new topic: token cost. Token cost moved into the spotlight recently as tools like Claude gained traction in legal because most plans come with token limits, as well as the ability to request limit increases, which has resulted in tales of astronomical bills for some users. Recently, Legora also announced that it’s moving its Agent Pro offering to consumption-based pricing, which means it’s billing by what the agent does rather than by a flat-seat license. And what agents do is consume tokens. Eventually, other providers are sure to follow suit. What many don’t fully understand is just how quickly token usage can add up. A few extra follow-up questions, a document pasted in twice, a chat continuing long after it should have been reset. If that sounds familiar, token consumption compounds faster than you might expect, and you might be looking at higher token usage than you think.

And you might not even realize it until you’ve hit your usage limit or, worse, seen the bill. This is a blind spot we’ve been unpacking in one of our latest article series on Legaltech Hub: how to get more out of tools like Claude without burning time, decreasing accuracy, or racking up unnecessary bills. We’ve covered topics like what tokens are and why they function as a hidden meter running behind every chat, when to reset a conversation versus continue in the same chat, and what everyday prompting habits, from repasting whole documents to burying five questions in one prompt, might be driving up both cost and inaccuracy without you knowing it. Head over to legaltechnologyhub.com to read the full series and learn more about how to get the most out of your token limits and your use of tools like Claude.

Marlene Gebauer (01:51)
Welcome to The Geek in Review, the podcast focused on innovative and creative ideas in the legal industry. I’m Marlene Gebauer.

Greg Lambert (01:57)
And I’m Greg Lambert. Today, we’re exploring the modern legal services market through a value-focused operational lens. We’re looking at what happens when you apply the strategic rigor of private equity to law firms, and how that intersects with the ground truth of legal operations on both the law firm and client sides.

Marlene Gebauer (02:24)
Yeah, we are absolutely thrilled to welcome back a frequent and favorite guest of the show, Brad Blickstein. Brad is the chief executive officer of Blickstein Group. He’s widely recognized in the legal industry as a futurist who’s spent nearly three decades analyzing how legal services are purchased and delivered.

Greg Lambert (02:42)
And, like us, he started when he was 12.

Marlene Gebauer (02:45)
That’s right.

Greg Lambert (02:46)
Brad, we’re going to talk about this, but you’ve been pretty busy lately. This past April, you published your new book, WWPED, which is What Would Private Equity Do?: Unlocking Value in the Law Firm You Already Own. Then, as if that wasn’t enough, you also released two massive benchmark studies: the 2026 Law Firm COO Survey Report in July and the 18th Annual Law Department Operations Survey. So you’ve been busy.

Marlene Gebauer (03:15)
Yeah, Brad the underachiever. Welcome back to The Geek in Review.

Greg Lambert (03:18)
[Laughs.]

Brad Blickstein (03:20)
I have been busy, and I do like to try new things, but I have to push back a little bit at the use of the second-person singular there. I have a great team at Blickstein Group, and they did most of the work that I get the credit for. So I’ve been busy, but they’ve been busy too, and it makes a big difference.

Greg Lambert (03:41)
That’s a good leader that gives credit where credit’s due.

Marlene Gebauer (03:44)
Absolutely.

Brad Blickstein (03:45)
Well, yeah. I mean, it is my name on the door, so...

Marlene Gebauer (03:48)
[Laughs.]

Brad Blickstein (03:48)
But we do what we can.

Marlene Gebauer (03:51)
So, Brad, in your new book, WWPED, you challenge law firm leaders to stop treating their firms purely as cash-flow vehicles for year-end partner distributions and start focusing on enterprise value. You use a great bakery analogy to explain Jae Um’s cream, core, and commodity market segmentation. Can you walk us through the difference between an artisanal legal bakery and a Crumbl Cookies-style system, and why the core segment, representing 70% of the market, is the ultimate battleground for law firms today?

Brad Blickstein (04:27)
Sure. Let’s start with cream, core, and commodity. The idea is that it’s a pyramid. The cream work is at the top. That’s what you think it is: the work where you need a seriously good lawyer to give you seriously good advice or provide seriously good work product. This is where we think about things artisanally. That work needs to be done artisanally. Then, at the bottom of the pyramid, skipping ahead a little bit, we have the commodity work.

Much of that is already not being done by law firms, or is in their captive ALSPs. That work has largely become systematized or scaled. But there’s tons of work in the middle. Jae calls it core work. Sometimes I prefer “run-the-company work.” The work, as you point out, Marlene, that we say is 70% of the market. To be honest with you, that’s a Fermi number, or really just a guess.

But it’s something like that. It’s a big chunk of the market. This is what law departments spend most of their money on and, frankly, where law firms earn most of their profits. This is where the leverage model plays in for law firms. Much of that work shouldn’t be done artisanally. It can be done differently than it is today, and it can be scaled. That’s where the bakery analysis comes in.

If you’re going to the bakery and it’s your 25th wedding anniversary, you’re throwing a big party, all your friends are going to be there, and you want the best cake you can find. You want something special for your spouse. You want to make it a big deal. You want to go to an artisanal bakery where an actual baker with skills is going to bake your cake because that’s appropriate for that situation.

On the other hand, your kid got a couple of B-pluses in school and you want to bring home some cookies to celebrate. That’s nice too, right? But you don’t need to go to that length for that. You can go to a more scaled bakery, something like Crumbl. And what’s the other one? The overnight cookie, the sleepy-time cookie. There’s something else like that. Insomnia Cookies, sorry. The opposite of sleepy time. You can go there and pick up a bundle. Think about how those different types of entities work.

At Crumbl Cookies, there’s likely not even a baker within 100 miles of the store. There’s a formula, a recipe, and technology that makes sure it goes right. They put out a fine product that works for most instances where you want cookies or a cake. I don’t think we always do a terribly good job in legal of differentiating between the two. There’s a lot of work that can be done with technology and different resources in a scaled way.

That work can be more profitable for firms if it’s done right, and it needs to be differentiated from the artisanal cream work, which shouldn’t be scaled. What we’re arguing for in the book, more than almost anything, is that you should figure out what that work is and scale appropriately for your firm.

Greg Lambert (07:26)
Yeah. I was afraid, Brad, that instead of saying the kid got a couple of B’s on the report card, you were going to say, “On your 24th wedding anniversary, you don’t go get a fancy cake,” and I was going to have to make sure

Marlene Gebauer (07:37)
[Laughs.]

Brad Blickstein (07:37)
Well

Greg Lambert (07:38)
my wife didn’t listen to this episode.

Brad Blickstein (07:40)
Well, I’ve been married more than 24 years, and I’ve learned a few things along the way, so you were at no risk of me stepping in that little pile.

Greg Lambert (07:50)
Well, I think that’s a pretty good argument. Part of the issue is firms trying to determine what falls into those categories. But one of the other issues we run into a lot, and we’re seeing it now around AI infrastructure, is the fact that, as partnerships, these firms are designed to distribute their profits at the end of the year and not have anything roll over. So when you see things like Kirkland announcing that they’re going to put $500 million into AI infrastructure over the next few years, one, not every firm can do that. And two, I laugh because I remember talking with a partner who said his wife remembers the one month he didn’t take a draw because they were putting some funds into the future. He said that was 10 years ago, and she still remembers it. It’s outside the norm. How do you convince firms that aren’t used to doing this to figure out a way to build for the future?

Brad Blickstein (09:08)
Yeah, that’s a great question. Not everything is for everyone, right? Not every strategy you might deploy here is for every firm. But it is one of the advantages of taking outside investment. That’s where you can smooth that over. You can use PE funds, or funds from whomever, for some original payout to your partners, to smooth that over time, or to make the investment. That’s one way to do it.

The other way, and I just alluded to it, is to pick your spots. Half a billion dollars is a lot of money for everybody. But over three years, which I believe is the period, it’s not that much money. I can’t believe I used those two things in the same sentence, but it’s not that much money for Kirkland partners, because they have

Marlene Gebauer (09:55)
[Laughs.]

Brad Blickstein (09:56)
enough partners to spread that out nicely, right? One thing I wonder about the partner you mentioned: did he tell his wife three years in advance that one month’s draw might not be coming? There are expectations that need to be set.

Greg Lambert (10:10)
No, I guarantee you he told her the month it happened.

Brad Blickstein (10:14)
Right. So there’s some of that too. But one of the book’s big arguments is that you don’t need to take outside investment to do some of the things a PE fund would say a firm should be doing. I list about 10 different things, but you don’t have to do them all at once. One thing I believe in is that law firms should professionalize pricing.

If I were a PE fund investing in law firms, one of the first things I would suggest is bringing on a serious, professional pricing operation to fix that. I’m not even talking about alternative fee arrangements or flat fees. That’s exactly what I mean by professionalizing.

Marlene Gebauer (10:50)
What does professionalizing pricing mean? Firms have pricing groups, so what are you saying that’s different?

Brad Blickstein (10:58)
Well, some firms have pricing groups. Some large firms have pricing groups. Most firms do not, and most midsized firms do not. In a lot of cases, those pricing groups have been neutered or aren’t listened to. At most firms, it’s still often a partner’s gut feeling about whether they have to discount or whether they can pass through a price increase. In many cases, the engagement partner feels they can’t get that across. They can’t sell that to clients right now without the right data to back it up.

Again, not everything is for every firm. But for many firms, it’s easy to see a couple, three, or five points across the board if you bring in the right type of pricing function. That goes right to the bottom line. Pick your spots. Don’t do everything we’re suggesting here at once. Where can some of this PE thinking truly affect value? Start there, with a bite your partners can digest.

Marlene Gebauer (12:03)
Let’s look at the findings from the July 2026 Law Firm COO Survey. A historic shift occurred this year. For the first time, technology investment and adoption surpassed talent acquisition as the top strategic initiative for law firms. Yet 38.1% of the COOs identified practice silos as the number-one structural issue they would fix, and 27% cited a lack of operational authority. Why is there such a persistent authority gap for professional managers tasked with executing these high-stakes technology strategies?

Brad Blickstein (12:39)
Yeah, I think there are two answers. One is the relative power of the people expected to do that work. At a midsized firm, the COO might have had a title like administrator a couple of years ago, perhaps even the same person. To what extent do the partners empower that person to run a firm the way a serious businessperson would be allowed to?

You hear a lot of this from COO-type folks: “This is what I’d like to do, but the partners won’t let me.” So there’s that. I guess there are three issues. There’s also a big delta between their opinion of leadership and their opinion of partners. Leadership is often more supportive of COOs’ business initiatives than the partners are. There’s that disconnect: “The managing partner is totally behind me, but then when I try to roll it out through the partnership...” You guys live and breathe this every day, right?

Greg Lambert (13:41)
No, I’ve never seen that. Never.

Brad Blickstein (13:44)
I do think, and this is evolved thinking for me, that practice silos are an underrated problem, especially at big firms. Different partners think differently, and different practice areas are built differently and designed to do different things. We understand why they’re put together in a firm, but in many ways they’re not alike enough for some of these systemic changes to be achieved entirely firmwide. The IP practice and the M&A practice, for example, are run and built quite differently.

Greg Lambert (14:23)
When Marlene was giving you that question, I was thinking that one of the worst things you can have in a position of authority is all the responsibility and none of the authority to do what you’re assigned to do. One thing you pointed out in the survey was a rise in COOs coming from outside legal to take over those roles, which makes sense. A lot of us want to run things more like a business, so we bring in businesspeople. But then we throw them into legacy law firm environments. How do you avoid bringing someone in and giving them all the responsibility and none of the authority?

Brad Blickstein (15:16)
That’s such a... Yeah, maybe.

Greg Lambert (15:18)
That’s another book, huh?

Marlene Gebauer (15:19)
Yeah.

Brad Blickstein (15:21)
That might be an encyclopedia. I think a lot of it comes down to where power is consolidated within the firm. If you’re talking about a small or midsized firm where leadership has consolidated a lot of the power, it’s easier. When the job becomes a consensus-building job, it’s difficult. Frankly, if you came from outside legal, that might be an impediment. If you come from corporate America, where it’s hierarchical, what the boss says goes, and people generally stay in line, working at a law firm where that’s inherently not the case is problematic.

One instructive example, and I’m oversimplifying, is that plaintiffs’ firms don’t have this problem as often. There’s usually one or two people’s names on the door. There’s a big boss who is the big boss of the firm, whether their name is on the door or it’s “Tiger Law” or something like that. They tend to row in the same direction, with fewer practice areas. They’ve been able to bring in a lot of talent and innovative business strategies that people on the corporate and defense side struggle to bring in. So it can be done. It takes a consolidation of power, or you need to be a world-class consensus builder, or there has to be some consolidation of power above you within the firm.

Greg Lambert (16:51)
Yeah.

I was going to ask: what advice would you give a COO coming in from outside legal?

Brad Blickstein (16:57)
A little patience, a little Zen. Make sure you re-up your blood pressure medication before you start. It takes the right kind of person.

Greg Lambert (17:09)
Enroll in some yoga and meditation.

Marlene Gebauer (17:11)
Breathing exercises.

Brad Blickstein (17:13)
And fight the battles you can win.

Greg Lambert (17:16)
Yeah.

Brad Blickstein (17:17)
But if you pick the battle, you have to win it. You can’t lose the ones you’ve decided to fight. That’s a good lesson for COOs too. Think carefully about how badly you need something to happen and how important it is. If you’re going to use a lot of political capital and possibly burn your reputation when you lose that battle, think about how badly you need to fight it.

Marlene Gebauer (17:44)
Makes sense.

One of the most striking statistics in the COO Survey is that 66% of COOs report their firms do not formally measure or document AI-related efficiency gains. And 39% of those same COOs expect AI to increase revenue through greater lawyer productivity. If firms are saving time but aren’t measuring it, are they actively devaluing their services? How can a firm defend pricing under a billable-hour model when AI is compressing those timelines and the firm has no baseline data for pricing alternative fee arrangements?

Brad Blickstein (18:33)
It’s a conundrum. It’s a problem. First, I think the number of people who are formally measuring or documenting is going to go up, right? There’s no way we’re going to continue

Marlene Gebauer (18:45)
Has to.

Brad Blickstein (18:46)
not documenting anything. And let’s be honest, while it can be hard to analyze and use, the data is there. These are people who measure everything they do in six-minute increments, right? We can talk about this later, but it’s going to be a much bigger problem in-house, where they don’t track what people do daily. At law firms, they do, so that number is going to change.

I’m fascinated by this AFA question because I’ve been tracking it for so long. You guys know I was one of the founders of Corporate Legal Times magazine, and in the fourth issue we published, in 1992, we ran an article with the headline, “Alternative Billing Making a Comeback.” So 30 years ago, Jesus, 35 years ago now, it was already making a comeback. This idea that...

Greg Lambert (19:36)
Any day now.

Brad Blickstein (19:37)
And we’re almost...

Marlene Gebauer (19:38)
Mm-hmm.

Brad Blickstein (19:38)
here. I think the idea that clients are going to make firms do this has largely been disproven. It’s true around the edges, and there’s 15% or whatever that do it, but the hypothesis that firms will adopt AFAs because clients will make them has been disproven over the last 30-plus years.

Now we’re getting to a point where it might be the firms that need to do this. It won’t be only for client satisfaction, because they think clients want it. It might become the only way to make money. So I’m not as concerned about the 66% that, in my mind, aren’t yet tracking. I’m much more concerned about the 39% who say they’re going to increase revenue through greater lawyer productivity. How? Under what model does that happen? Better productivity should decrease revenue. Perhaps it lets you bring on more clients and make enough of it back that way, but that’s a nuanced issue. I’m not sure the market sees it yet. People are starting to get their arms around it, but I don’t think anyone has solved it.

Firms are going to have to measure productivity better and understand it better. They’ll have to go to clients and say, “Look, these projects that used to take 10 hours, we can’t bill you for one hour. I guess we could bill you $7,000 for that hour, but that doesn’t work. How can we bill you in a way that gives you value and allows us to capture some of the efficiency gains?” To me, that’s

Greg Lambert (21:19)
Right.

Brad Blickstein (21:19)
the battleground. Like that’s what we’re looking at.

Greg Lambert (21:22)
Right. I’m going to switch up my question a little after hearing your answer. I heard this at a conference earlier this year: why would clients want AFAs now, when they think it takes only one hour instead of 10? Where’s the motivation? But my question is more about how we get past the trust factor and move to something beneficial for both sides. We tried this in 2008 and 2012. It was like, “Hey, we’re going to do alternative fees,” and clients responded, “Yeah, I don’t trust you. I want the alternative fee, but I also want you to tell me how many hours it took so you’re not getting one over on me,” to put it politely. How do we get past that?

Brad Blickstein (22:19)
That’s a great question, and it’s an underrated issue. People love to blame firms for the fact that AFAs largely haven’t been implemented, but clients don’t trust their own firms. That’s a big issue. The shadow-billing issue you mentioned is huge. If you’re a firm, you’re effectively putting a cap on the amount you can bill without putting a floor on the amount you could bill. It doesn’t work for firms. I think we’re approaching a crossroads.

The first thing to remember is that clients like sending work to their law firms. They like working with their firms. Despite all the talk about bringing work in-house, and I know that’s a giant trend, all things being equal, most general counsel would prefer to keep sending work to their firms. Now, all things aren’t equal, and they’re less equal than they used to be.

First, firms will have to be more transparent without going all the way to a specific number of hours. Firms have to do a better job, and this is hard, of articulating the value of their work in some way other than the number of hours it takes.

Frankly, I think firms need to stand up a little more. If you’re a firm and a client asks for shadow billing, the proper answer is no. “We won’t share the number of hours this work takes. Either you feel you’re getting a fair amount of value for the work we’re doing or you don’t. And if you don’t, we have a problem, and we’ll talk about that. But the hours we put in aren’t your business.” You have to say it much more nicely than I just did.

I don’t think firms push back enough when clients ask that. The whole point of an AFA is valuing the work rather than the time. And, by the way, the amount of time it takes to do the work is not the same as the value of the work. We have to figure out a way to work together to provide good work for good value, rather than go to work for a certain amount of time. I’m preaching at this point, though. I don’t know how to do that.

Marlene Gebauer (24:37)
It’s a good point, but it’s interesting because you’re getting that level of detail in pitches and RFPs. Clients want to know exactly how much time things take and how much time you’re saving. It’s difficult to respond the way you’re suggesting when they’re specifically asking for this information.

Brad Blickstein (25:08)
It’s hard to pivot that conversation. When they ask that question, the way to pivot is to say, “Let’s not talk about how much time we’re saving. Let’s talk about how much money we’re saving. Here’s what we used to bill you for this type of work, and here’s what we’ll bill you now.” You can ask how long it’s going to take, but what does it matter?

It’s easy for me to sit here on your podcast and say, “Go tell your clients we’re not going to answer your questions.” It’s hard to do, and it takes fortitude. But if you establish the value and build trust with clients over time, I think it works.

The other thing you might do, especially when talking about AI, is think about the other value propositions. Can you get the work done faster? I don’t mean in less time. I mean finished sooner. Can the deal close faster? Can the litigation be settled sooner? In due diligence, can you review more material than before? Classic due diligence used to mean, “Send us the contracts with your 25 biggest clients,” because that was all the due diligence team had time to review. Now you can use AI to look at all the contracts. What’s the value proposition around using AI that isn’t tied solely to efficiency?

Tie that to cost savings rather than time savings or hours saved. Stand up and say, “Look, we respect you, but we’ve thought a lot about this. We’ve spent a lot of energy on it, and we think you’re asking the wrong question. Let us answer the right one.” Some clients will say, “Well, they didn’t fill out our RFP properly.” Too bad. But I think many will respect that and want to work with a thoughtful firm.

Marlene Gebauer (26:54)
I’m going to shift to the buyer side. The 18th Annual Law Department Operations Survey, the LDO Survey, shows a corporate legal buyer that is stabilizing structurally while absorbing broader business mandates, including CFOs stepping directly into contracting. Meanwhile, corporate GenAI adoption is defined by piloting, at 52%, rather than fully operationalized systems, at 23%. Why are corporate legal departments stuck in pilot purgatory, and how does the middle-mile data hygiene problem play into this?

Brad Blickstein (27:34)
Yeah, that’s part of it. First, considering we’re only two or three years into the AI era, depending on how you count, I’m not sure it’s purgatory. In many ways, we’re working through this, and I expect that number to fall substantially over time.

Part of it is caused by all the noise. How do you determine whether a tool is worthwhile these days without giving it a shot? It’s hard. I think we’ll get there, but the data hygiene problem will grow. Whatever the old saying is, garbage in, garbage out. I think we’ll solve the data protection problem. People understand, and are starting to become more comfortable with, how their data is used.

The idea that we want our law firms to provide every possible ounce of efficiency through AI tools, but we won’t allow them to use our data as training data to help them do that, isn’t fair. I think we’ll get past many of those issues over time. But data in corporations is a mess. To get the full benefit from these tools, you’ll have to solve that in many cases. I wonder how many pilots will fail because the technology, adoption, and training are good, but the data isn’t there.

Marlene Gebauer (29:13)
It’s a big issue in terms

Brad Blickstein (29:16)
It’s a big issue.

Marlene Gebauer (29:16)
of getting this data usable, getting it where you need it to go, and making your technology useful. I mean, it’s

Brad Blickstein (29:23)
Yeah, my hope is...

Marlene Gebauer (29:26)
It’s not a new problem, but, yeah.

Greg Lambert (29:27)
Yeah, I was going to say this is a 25-year-old overnight problem.

Marlene Gebauer (29:30)
It’s not a new problem, but I don’t know whether it’s easier or harder. The technology is so much better, but the security issues are more difficult. I’m not sure.

Brad Blickstein (29:44)
What I’m hoping, I guess, is that if you look at e-discovery over the years, technology created the e-discovery problem, right? The ability to disperse data quickly, email everyone, and copy a million people created a data problem. Technology then largely solved it with tools such as auto review and platforms. I’m hoping AI will help solve the data hygiene problem itself, enabling AI to work better on the legal problems the data supports.

Greg Lambert (30:25)
I know this was the first time in the survey, and of course, AI has only been around for about three years, that usability received a higher score and beat out security as the top AI priority for legal operations teams.

I think this opens the door to unique opportunities for law firms to offer services to their clients. Are you seeing clients expect something they haven’t been offered before? Are they receptive to those services?

Brad Blickstein (31:10)
I think they’re absolutely receptive. If I ran a law firm, I’d try to get ahead of all this. I’d build things, find new use cases, and bring solutions to clients before someone else does or before the client tries to build something themselves.

For any work you’re concerned the client might bring in-house, think about what that work is and build a case for why they don’t need to bring it in-house. Explain why the firm can do it more efficiently, better, and at a better price than the law department going through the trouble of bringing it inside. I don’t think we see firms bringing that case to clients enough. When firms do, clients are extremely receptive.

ALSPs are starting to get involved too. If I were a firm or an ALSP, I would build offerings and bring them to the right clients before the client decides to find a solution.

Greg Lambert (32:11)
Yeah. Speaking of ALSPs, why haven’t they wiped the floor with BigLaw? Everyone seemed to expect that, because they have these repeatable processes. Why aren’t they kicking ass?

Brad Blickstein (32:34)
Well, first of all, I think some of them are kicking ass, right? When you look at them as a percentage of the legal market’s overall share, it seems somewhat stagnant. But many of these businesses are on big growth curves within the ALSP segment. There are many reasons, but the big one is that clients like working with law firms, and that’s their

Greg Lambert (32:59)
It’s what they know.

Brad Blickstein (33:00)
preference. If it isn’t working, or if it’s the kind of process or project that makes sense to take away from a firm, document review is a great example. Law firms largely proved they shouldn’t be in that business by throwing a million associates and no technology at document review 20 years ago. Legal service providers and e-discovery providers have thrived there.

For work that is clearly process-oriented, I think ALSPs are doing quite well. But much of what they have to do comes back to the artisanal argument we discussed. They have to convince clients that a large amount of legal work isn’t artisanal, that it benefits from process and technology, and that ALSPs are the people to deliver it.

Marlene Gebauer (33:48)
Okay, Brad, what’s true today that wasn’t true a year ago?

Brad Blickstein (33:55)
I think legal organizations, both in-house and outside, are starting to care about what AI costs. Legora, I believe, talked about moving to a consumption model. That’s a big issue. People are moving from, “Let’s get everyone to adopt whatever they can and use AI wherever they can,” to, “Hold on. What’s it going to cost when we start paying by the token? Does it make sense?”

The underlying assumption is that using AI to do this work is less expensive than using lawyers. At some point, if AI gets expensive enough, that might no longer be true. That wasn’t a concern a year ago. The gold rush was on, and now people are starting to wonder what stream they should mine. I think I mixed the metaphor. Where should we drill, look, or pan? Where should we pan for gold? That’s what I’m asking.

Will these price increases stick? Will open models become more valuable and viable? Those are good questions. I believe people are concerned about this today in a way they weren’t a year ago. I’m not sure how long the concern will last.

Greg Lambert (35:23)
Yeah, it’s going to be interesting to see, because six months ago it was, “Burn every token you can get your hands on,” and now it’s, “Whoa, wait a second.”

Marlene Gebauer (35:33)
Well, wait a minute. Hold on there.

Brad Blickstein (35:36)
I’ll also say, and it’s related, that we’re starting to see people distinguish between adoption and fluency. It felt like the standard was activation, and then adoption: “98% of our people have used the tool in the past month.” That’s a nice statistic, but it doesn’t say whether people have become AI-proficient or AI-fluent, whether they’re using it for the right things, or whether they’re using it enough.

Marlene Gebauer (36:01)
Is it useful for what they’re doing?

Brad Blickstein (36:04)
Yeah. As we start asking those questions, it’s going to affect where the value lies and what these tools are worth.

Greg Lambert (36:14)
Well, Brad, before we started recording, we looked back at your last prediction on the crystal ball question, and I think you did pretty well.

Marlene Gebauer (36:23)
Yeah, he did.

Greg Lambert (36:24)
Your prediction at the end of last year was that generative AI and LLMs would fundamentally accelerate how software and tools are built in the legal tech space, and that you thought there would be about an 80% improvement in turnaround speed from idea to product.

Brad Blickstein (36:47)
That does seem to be happening. Especially if you count the things being built within firms and law departments through vibe coding, Claude Code, or all those things, I think we can check that off as one I got right.

Greg Lambert (37:04)
All right. Now it’s time to ask you again. What kind of change or challenge do you see that law firms and law departments need to prepare for over the next few years?

Brad Blickstein (37:19)
I’ll answer two ways to double my shot at getting something right. Generally speaking, the issue of the business and billing model will have to come to a head at some point. Perhaps less so within the next two years, but eventually many law departments will put pressure on firms to reduce the number of hours they bill. I have some thoughts about how they’ll do that, but I don’t know.

If you’re a firm, that doesn’t work under the leverage model. At some point, many firms will have to come to terms with the issue. I also think about all these, and I’m not a big fan of the “AI-native” term, but these AI-native-type spin-offs from firms. Can partners leave a BigLaw firm, start a firm with no associates and use AI instead, then go to the same clients who trust them with that new model? External forces like that might push firms to make some of these changes more than their clients do.

Marlene Gebauer (38:25)
Well, that is food for thought. Brad Blickstein, thank you so much for joining us again and helping us cut through the noise and think like value-focused operators. And thanks to all of...

Greg Lambert (38:35)
Yeah. Thanks, Brad.

Marlene Gebauer (38:36)
you. Sorry, go ahead.

Greg Lambert (38:37)
Thanks, Brad.

Brad Blickstein (38:39)
Thanks, guys. I appreciate this. It’s always fun and thought-provoking talking to you. You ask questions that make me think hard about what the responses should be, which I don’t appreciate, but I do like talking to you guys.

Marlene Gebauer (38:50)
We appreciate it.

Greg Lambert (38:54)
Yep, he had a full set of hair before we started this.

Brad Blickstein (38:56)
It always reminds me of that scene in the movie Clueless where they think she has a concussion and say, “Ask her some questions.” They ask, “What’s seven times seven?” and someone says, “Stuff she knows.”

Greg Lambert (39:05)
[Laughs.]

Brad Blickstein (39:06)
I would prefer if you asked me stuff I know.

Marlene Gebauer (39:11)
And thanks to all of you, our listeners, for taking the time to listen to The Geek in Review. If you enjoyed the show, please share it with a colleague. We’d love to hear from you on LinkedIn and Substack.

Greg Lambert (39:21)
And Brad, what’s the best place for listeners to find and buy What Would Private Equity Do? and learn more about

Brad Blickstein (39:29)
Easy.

Greg Lambert (39:30)
Blickstein Group?

Brad Blickstein (39:31)
Yeah, you can go to our website, BlicksteinGroup.com, or for the book itself, BlicksteinGroup.com/WWPED. And I’ve always wanted to say this: it’s available wherever fine books are sold online. You can get it from Amazon or BN.com. It’s not hard to find, and we appreciate the support.

Marlene Gebauer (39:52)
Absolutely. And, as always, the music you hear is from Jerry David DeCicca. Thank you very much, Jerry, and goodbye, everybody.

A standalone chapter of Beyond the Model. The Same Way Every Time, Part Three of Three.

The interior conference room was small, windowless, and a couple of the chairs did not match the rest of the room. This was a reason Cooper picked this room. He wanted the presentation to be the thing that shined in the room. Leo Huang was first in the room and set up the screen sharing before Cooper arrived. Jesse was already on the wall with his Chicago glass walled office and whiteboard behind him. He sipped coffee from a mug that prominently displayed WORKS ON MY MACHINE.

Maya Rios walked in a couple minutes later with her legal pad, reading glasses, and an agenda sticking slightly askew out of the bundle. It already had a couple of notes labeled in ink along with the printed text.

“I just want everyone to know,” she began, shifting out of the wobbling chair into a more stable one, “I looked over Cooper’s memo on the woman at the Vegas conference, and I’m very skeptical. To me, a machine that does the same thing every time is called an appliance. I have appliances at home.”

“Your toaster doesn’t read,” Jesse spoke from the screen.

“And my toaster doesn’t improvise either. Which of these scenarios is going to be on the demo today?”

“Alright Leo,” Cooper held his arm toward the young associate at the front of the room. “With that as your set up, the room is all yours.”

Three folders

In the post-COVID era, Leo had dressed up for the demo. Which means that he put a blazer over his hoodie. He opened three folders on the table in front of him, and then in a moment of reconsideration, closed them and decided to ditch the theater mode and just talk through the process.

“During my first year, I created an extractor for processing incoming NDAs. You all know that story. The tool watched the M&A folder and looked for new agreements, pulled out the key terms and created a standard summary, and dropped that summary into a shared tracking spreadsheet. I didn’t write any of that code, I just described what I needed and had the LLM write the code. That was magic back then. It worked great, but in hindsight, it was kind of a messy process that should have gotten me in more trouble than it did. The tool ran under my local login and under my own credentials. No record or log of what it did. If there was an error in reading the document, that error went into the spreadsheet and might have looked as good as everything else. This was a stealth project and nobody knew about it until Cooper happened to walk by one night and ask what I was working on.”

“I definitely remember the committee demo after that,” Maya said. “Marcus from Risk Management saw your extractor touch client documents directly and aged a few years in that meeting. Then he asked how to scale it. We got a brand-new policy as a result of that little episode.”

“Three policies actually,” Leo said with a little more pride than he should. “Well, Tuesday and Wednesday, I sat down with Jesse and we built the extractor as a harness. Jesse taught me a lot about harnesses this week and how it is the code that is placed around the agent that makes the rules on what it can touch and what the shape of the work must take. The same job, the one repeatable job, all wrapped up in what the old version was missing. It’s designed to only read the NDA intake mailbox. It pulls the counterparty’s file from only two systems, read-only. Then checks each agreement and profiles it against the firm’s six standard positions. It writes out each extraction and every check to an evidence folder as it works through them. The final output is set as a one-page status. That status is set to a fixed form, every single time, with my name put at the bottom, listed as the owner. As Cooper asked Jesse and me to do it, there’s a card. It’s laminated.”

“The lamination was load-bearing,” Jesse said.

“Here are three test documents.” Leo held the folders and raised them one at a time. “This is a vendor’s NDA from a month ago, completely standard. The second is a real one from back in the spring and it has a non-solicit tucked into the eleventh paragraph. It’s the kind that normally slides through when an associate is reading forty of them. The third one I’m going to hold for now because it will be the point.”

Leo ran the first document. Everyone watched as the log began writing itself along the left side of the screen, evidence folders populated, and a few seconds later, the status page appeared. There were six positions, six checks, all green, and cites to each paragraph and page. Then he started the second run. Same result, same pace, and then the fourth line came back highlighted, the non-solicit quoted with a paragraph number and a routing note: nonstandard term, human review, along with an actual person’s name, pulled directly from the card.

“Let’s do the third run,” Leo said, and pressed the start button.

The log started scrolling, and then four lines in, it stopped. The screen displayed a single block of text.

Input failed contract check. Document presents as an NDA by filename but contains no confidentiality obligations; structure matches a supply agreement. No status page generated. Evidence and this refusal written to folder. Routed to owner for human review.

The room stared at the screen.

“This is a supply agreement I renamed,” Leo said. “It was the wrong document, correct folder. Something like this happens all the time in real work. My old extractor would pull a governing-law clause out of it and then filed a nice, tidy little summary, and that summary would have been timestamped and completely fictitious. This version understands what the shape of the job is. Anything that doesn’t fit that shape gets declined, logged, and hands it to the person named on the card.”

Everyone looked over as Maya removed her reading glasses.

“Run that one again,” she said.

Leo pressed start again. It returned the same four lines. The same refusal. All worded exactly as before.

“Again.”

As it ran for the third time, Maya watched the folder pane more closely. She watched the refusal file itself next to the evidence with the identical name and shape as the first two refusals.

“For twenty-some-odd years I have watched one type of expert system or another,” she said. “Some human, some automated. Confident systems are cheap. However, I’ve never seen either of them produce a clean report showing where the competence ends, the moment that it ended, and put that in a form that I could hand over to a judge.” She leaned back and placed her glasses back on. “The first two runs gave me a demo of the process. This third run was a character reference.”

The building code

“There’s one more thing that I want to put out there,” Leo said, “before it is told to me. I had a few more automations set up that are still running. You’ve seen some of them, but not all. If we implement this standard, I’ll be its first adopter. There are a couple of my automations that can meet the process with a little reworking. The other automations will need to be retired. I wrote out a list of those last night.” Leo slid a sheet of paper across the table. “This felt terrible earlier today. Now it just feels like I finally know which of my toys I need to stop playing with.”

“I just want everyone to know that Leo did this on his own,” Cooper interjected. “I didn’t put him up to it.”

Maya picked up the page and then looked up at Leo. “We need to think this through. The last time the firm found out about unauthorized automations, a new policy got dropped on all of us. Now we have the author showing up with a list and a clipboard. We can’t audit our way to this, we need to create a safe zone so people will adopt this willingly.”

The record

Maya flipped over a new page on her legal pad, and Cooper knew what was coming at this point. He leaned back to watch Maya go to work.

“The log,” she said. “We all love the log. It gives us an audit trail, it’s more than just a box on Jesse’s diagram, it’s our safety story. But the log is also a perfect record of everything the agent read and did within a client matter. Retained and readable. When one of those matters goes sideways, that log will become the most subpoenaed document in the firm. I’ve built cross-examinations from records far less candid than the log I just watched write itself five minutes ago.”

“You want the log removed?” Leo asked, honestly confused.

“Not removed, I want it governed so we have the benefit of the log, but mitigate the risks.” She continued writing as she spoke. “Who controls the log. Where it lives. Establish a retention schedule. Determine if it falls inside privilege, and how to defend that if challenged because it was created on a vendor’s model or server.” She folded the top of the page against the perforations and removed the sheet from the pad. She wrote across the top of the page and then turned it so the other could read: No harness touches a client’s work until the log custody is settled. “Whatever we decide, that goes along with it. Anyone object to that?”

No one objected.

The board

Leo erased the remains of a previous team’s org chart off the whiteboard of the small conference room. In its place, Leo drew a grid and put Cooper’s candidates down the side. Across the top he wrote down the tests: one job, the budget, a nameable owner, a fixed shape for the output.

The four of them paced through the work. NDA intake triage scored strong in every column, helped considerably by the demonstration still displayed on the screen behind them. Conflicts and intake: the workflow Jesse demonstrated for the firm years ago, still the most impressive twenty minutes of software Cooper had ever watched, and still, all this time later, running nowhere, because it had been a demo with no office around it.

Citation verification for filed briefs cooled the temperature of the room almost immediately.

“This could be a major problem,” Maya said as they reached the citation column. “With the other rows, we could be embarrassed or it could cost us money. We have insurance and a Marketing team for those. This citation row’s worst-case scenario is a judge handing out a sanction on one of our attorneys and having the firm’s name in the New York Times. Lawyers are getting fines and their licenses suspended if something goes wrong here. This is the row we need to focus on.”

The matter closing process only had three columns before it reached its logical end. Jesse estimated that the ROI on this would pay for itself in less than a year. “Who gets to own this one?” Cooper asked. The four of them looked at each other for a few seconds before Leo wrote down no owner. It was a standard they had all agreed upon an hour earlier.

What remained was the pre-bill analysis and the sorting of incoming docket alerts. Docket triage was very similar to Dana’s insurance claims investigation, and Jesse instantly began drafting out adapters for this row in the corner of his screen even before anyone had a chance to score it.

As he watched the grid begin to fill, Cooper thought back to the annual survey, its results frozen so it could be compared to next year’s results. He realized that he and the firm executed a perfect harness this summer, but they didn’t have the vocabulary to define it at the time. He leaned over the table and wrote down that thought in his notebook.

“All of the columns,” Maya noted, her gaze locked in on the filled whiteboard, “presumes there’s an existing ability to identify successful results from failure. Look back over the past hour. Leo proved this with the three runs and its ability to do just that. Now we have to consider situations like a vendor pitch waiting for us in two months for a renewal on a product. They’ll give us a single demonstration, complete with the ‘demo magic’ we’ve all seen from them using their own perfectly curated data sets.” With that, Maya stood and cleared off a section of the board with the dry eraser. “I don’t think we should start with any of these initiatives. I want us to construct the instrument that makes trust completely unnecessary.”

“You mean the evaluation harness,” Cooper said. The project of whether the firm builds its own standard of proof had remained an unassigned item on the committee’s June list, stripped of any funding or a name.

Maya leaned around the chair and picked up Leo’s third folder, the one with the supply agreement but defined as an NDA file.

“In a way, we already built one today,” she said. “It’s smaller in scale, but Leo picked the three documents where we knew the results up front. The run nailed the first two and then admitted failure on the third. Now we just need to expand the frame to the firm’s own horizon. Take forty closed matters, where we define success based on if the work was billed, paid, and didn’t result in further litigation. From that we create rubrics gathered from the attorneys who handled those matters, and we lock them down like we did with the survey. We need a perfect score on each of these, without exception. If the tool misses twenty percent, then that leaves us with an output that we can’t endorse. It needs to be perfectly clear when we have success and when we have failure. From then on, every new internal harness, every new AI model, and every incoming vendor pitching renewal of existing products has to survive those same forty trials and earn the same objective grade. That includes your project, Leo. In fact, you get the honor of being tested first.”

“I like that,” Leo agreed. “That way if it fails, I know before Marcus does.”

Jesse had remained on the wall, but had not said anything in a while. He was staring through the camera and beyond the three others.

“Jesse, you look like you have something to add,” Cooper said.

“Before we weigh anything on this grid, we need to build the scale first. That needs to be our basis before going forward. As it stands, every argument you’re going to hear about agents at the firm, the renewals, a promotion decision, that all rests on someone’s opinion right now. This is going to be the end of that. It’s the least exciting thing on the board, but it’s going to be worth more than all the other rows put together.”

The name

The proposal completed in less than twenty minutes, eventually matching the recommendations that the firm’s committee defined a month ago. An agent without a defined owner doesn’t run. There will be an owner, a defined responsibility, and a review date that doesn’t get missed. If there was a blank field remaining, then they knew this meant admitting defeat.

The silence of the room was an indication to Cooper that they were needing someone to own this responsibility.

Maya would be the most logical owner. Jesse was a team member, but not a member of the firm. Leo didn’t have the experience to keep anything more than what he could personally defend.

Cooper thought back on the last few years he’d spent working as a proxy for others’ decisions, but he was always one step removed from the full responsibility and consequences.

With that in mind, he reached across the table, pulling the recommendation closer, and wrote his name on the commitment line. The evaluation harness now runs under Cooper Graham, with quarterly reviews, and an initial review the same week the rubric was finalized. As he finished writing, Cooper said, “If something goes wrong in the grading, that will be mine to explain. In a single sentence.”

He pushed the page back to the other side of the table. Maya picked it up and looked back at Cooper with one eyebrow lifted, giving him a moment to reconsider. He gave her a look of appreciation, but declined her offer. Leo, ignoring the nuance between the other two in the room, had already started making the card.

Cooper remained in the room long after the others had made their way back to their offices. The windowless conference room with the overused whiteboard and mismatched chairs had been the perfect setting. He snapped a picture of the whiteboard with his camera. He looked at the grid, with the tests across the top, the space that Maya had cleared away, where the evaluation harness information stood in her handwriting. He sent the cropped photo to a printer down the hall, cut it down to size, and slid it into his notebook on the page opposite of Jesse’s six boxes. The two drawings faced each other.

When he reached his desk, the floor was nearly empty as the late afternoon turned to early evening. Cooper docked his laptop and then typed out one line to the claims-software founder he met only a week ago: We’re going to build the scale first.

Dana’s reply came back before Cooper had picked up his backpack. It contained one word.

Twelve.

Cooper recited the last line in her boring list.

Decide how you will know it is still good, before you turn it on.

He pulled out his notebook from the backpack and added the day’s line under the printout of the whiteboard.

Second entry in the 2027 file: we signed for the scale before we weighed a single thing.

Right below that, in the limited space available, he wrote:

A harness is a promise written in code: the same job, done the same way, with a name on it. As of this afternoon, one of the names is mine.

A standalone chapter of Beyond the Model. The Same Way Every Time, Part Two of Three.

Jesse Tanaka set up the Monday call from his glass conference room in his Chicago-based lab. He’d already worked his way to the whiteboard before Cooper got to say good morning. He was surrounded by the bit of chaos that was the make up of the lab. Sticky notes on monitors, and through the glass behind Jesse you could see one of his colleagues walking past with a keyboard under his arm.

"I read what you sent me," Jesse said. "Twelve lines. I've been thinking about her all weekend. I don’t know her, but based on this list, I think we would get along."

"You'd like her. You would have enjoyed how she spent twenty minutes of the conference session listing everything her product cannot do."

"I saw. Line eight." Jesse held up a a copy of Dana's email, folded in half. "Alright, let’s dive in. You asked what a harness is. I just want to point out, that the last time you called me with a question this basic, I answered it on a cocktail napkin, and that napkin has been quoted back at me by a number of people I’d never met before. So I feel like there’s a little pressure here."

"Go ahead and use the whiteboard. The great napkins of history will forgive you."

The stranger

Jesse stood at the board for a second, looking around for where to start. Then he stepped up and drew a stick figure. It seemed like the best way to start.

“So here’s the model,” he said. “All of the models, it really doesn’t matter which one for this situation. The best way for you or one of your lawyers to think about it is like this. It’s a very intelligent person with no memory. The one and only thing it can do, is talk. It’s brilliant in that it has read everything. It can pull from what it has retained from all of this, and it can reason through your problem better than most of the people you’ve ever hired. But it starts every conversation as a newborn. It can’t remember anything from yesterday. It can’t access your email, it can’t open a file on your hard drive, or check your docket. That’s the entire toolkit you’re working from.”

"That part the firm is well aware of by now."

"Well, the firm knows it the way I know my cholesterol number," Jesse said. "Yes, I know it, but then I ignored at dinnertime. Okay, stay with the scenario for a minute, because what your insurance friend designed is built on taking this seriously.” He outlined a box around the stick figure, then made a larger box around the first, and then began furnishing the larger box as he kept talking. Quick small rectangles, each of them labeled.

“So the harness is the office you build around this genius. They have access to a file room, and then you decide what filing cabinets you place in this room. Everything else remains off the floor plan entirely. There’s a phone in the room, but that phone can only dial four numbers. The phone itself doesn’t even have the other numbers on it. They simply don’t exist. You give the genius a clock, and when that clock runs down, the workday ends. It won’t matter if they’ve finished the work or not. You place a logbook in the room that copies down every cabinet they’ve accessed, along with every call they made, timestamped and unerasable, and they know it is there. Plus, you put a name on the door. A human’s name. The person whose office it really is, who reads what the genius produces and has to answer for it.”

Jesse stepped back.

"The genius combined with the office is the agent. Most people you saw present at the expo, when they said agent, meant a stranger holding a master key along with a set of instructions about values. Dana, on the other hand, just means the office. That difference is her entire company."

The loop

"Got it. Now walk me through a single run," Cooper said. "Nice and Slowly. The way you'd walk Maya through it."

"Pfft. Maya would have objected four times already, but fine." Jesse wrote out numbers down the side of the board. “One. The harness hands the genius a briefing and instructions on the job. Here are the cabinets you may access, here is the what the final product must look like. Two. The genius reads the instructions and make a proposal. Open cabinet number three. Retrieve the policy file. Three. This is the part that Dana’s company builds on. The harness intercepts that proposal and reviews it. Is the third cabinet on the job’s approved plan? Is the request formatted properly? Is there a budget remaining? If all of these questions pass, then the harness itself goes to cabinet number three and retrieves the folder itself. The genius doesn’t leave where they are sitting in the room. It has no ability to retrieve anything. They only way to get anything is through the code that your own people wrote, which means you control the rules for what gets accessed and retrieved. Four. The results are transferred to the genius, the log gets updated with that information, and then we go back to step two until the run is finished or until time runs out on the clock.

"And the report at the end comes out in the same way every time because..."

“Because the harness sets the guardrails around the project.” Jesse pointed a marker at the outer box. “Dana’s seven sections are in a fixed order. That instruction lives in the code. The genius can request anything it wants, but the office doesn’t care. It’s a pretty boring setup on the entire product, and that that makes it the perfect product for any work someone else has to rely upon.”

Cooper looked over the board, at the stick figure in the chair surrounded by its labeled rectangles. He looked at Jesse and asked the question he waited all weekend to ask.

"So when her demo ran the same way twice. How much of that replied upon the model behaving?"

"Practically none of it," Jesse said with a grin. "The model behaved because the office didn’t give it any room to do anything else. The office gave it the same briefing, the same four cabinets, the same budget, the same required output, and it identified all of the mistakes that were caught by nightly test runs before it could reach a customer.” He placed a circle around the stick figure. “The consistency of it all lives out here. It’s lived out here the entire time.”

The rig

"Okay, show me," Cooper said. "I know you keep a toy for this exact conversation."

"I keep a rig." Jesse reached out and angled the camera toward his center monitor and typed. "Sandbox folder - five fake court filings - zero client anything. First, the genius with no office around it. One request: read the folder - give me a one-paragraph status."

The reply came back quickly and confidently. The paragraph looked right. Then it continued and Cooper moved closer to the screen. It reported that it had renamed two files whose dates were formatted inconsistently. Then it asked if it could add a deadline to the calendar. Then it drafted, on its own accord, a short email to the associate assigned to the matter with two clarifying questions.

"You didn’t ask for any of that," Cooper said.

"It's trying to be helpful. Helpful is the make up of its entire personality, and there's no office telling it where to stop being helpful." Jesse swapped windows and pasted the same request into the other screen. A log wrote itself down the side: folder read requested, approved, five files, output contract loaded. The same paragraph arrived in the same output, and underneath it, there were three flat lines. Rename: write access not granted. Calendar: out of scope. Email: out of scope. Noted in log.

"Same genius," Jesse said. "Same brilliance, forty seconds apart. Every decision you viewed just now came from the office. And one more thing while we're at the rig, the two classic ways people wreck this, just so you have the vocabulary when a vendor tries them on you. Wire the stranger to forty specialized tools and it spends its genius guessing which drawer you meant. We call that the tool zoo, and the fix for it is a few clear doors. That is why Dana has four. If you don’t give it a clock or a budget, and one persistent error at two in the morning becomes a five-figure invoice in the morning, because nobody instructed the loop on how to give up. The unkillable session. Both of those are office problems. The genius gets blamed for both."

The smallest box

Jesse wiped off a section of the whiteboard and quickly drew a diagram he knew from memory.

"Let me show you a famous picture in machine learning, from a paper back in 2015. Google researchers, writing about what they phrased as technical debt. The paper's whole thesis fits in this one image. This is the box where the actual machine learning happens. Here is the stuff you have to build around it to run in the real world, the data pipes, the monitoring, the serving, the checks. And the funny part of this is the proportions." He boxed a small square right in the center. "The learning is the smallest box on the entire diagram. Everybody in my field has that picture drawn out somewhere. Now we’re living through an agentic version of it."

He labeled the small center box MODEL, and then built the constellation around it: TOOLS. PERMISSIONS. BUDGET. MEMORY. LOG. OWNER.

"The model is the smallest box," he said. "It’s the most glamorous, the most expensive to make, the one every keynote screen at your expo talked about. And yet, it’s the smallest. The reliability people actually experience comes from the six boring boxes drawn around it. Which, incidentally, is how I keep my job."

Cooper had already put down his notebook and pen and took out a copy of Dana’s twelve lines and then looked at Jesse’s board. He scanned the diagram, box by box, and then looked back at the note he had written back in Vegas. The one he drew the arrow and question mark on. Then he scribbled out the question mark.

"Jesse. Go over the boxes with me again, slowly."

"Tools. Permissions. Budget..."

Cooper leaned over and grabbed the committee binder off the shelf. He flipped through the pages as Jesse continued talking, and h stopped at the June resolution and read out the one sentence into the camera. “‘No agentic workflow shall exceed its authorized spend without a named human's approval, recorded at the time of the action.' The committee passed that back in June. Say the budget box again."

"Budget," Jesse said, slowing his cadence. "A cap, a stop, an approval gate."

"That's it. That’s our meter. That's the caps." Cooper set the binder down but kept his finger on the page. “There are five more boxes in this binder. Permissions is the structure you helped us with. The audit-trail policy is the log, it’s nearly identical. The owner box is the card we set up in July, the one with a one-sentence test printed on it and laminated.” Cooper shook his head and then looked up at the screen. “Jesse, we have a page in this binder for every box on your whiteboard.”

Jesse looked away from his own diagram, and then he laughed in agreement.

"You've been writing the spec for a harness all year," he said. "You just wrote it in Word and placed it in that binder."

Neither of them said anything while that sentence finished arriving. On the board, the six boxes stood around the small one like a floor plan waiting for a building.

“We spend all year describing the structure, the walls that Dana talked about in her presentation,” Cooper said.

“And that is our gap. It’s the gap that’s everywhere right now. We’ve spent a lot of time in the legal industry building rules and policy about agents, but almost none of it runs as code. And the policy only governs the people who know it exists. If we have a harness, it runs every single time, regardless of who is watching.”

The proof from the field

“As you thought about this over the weekend,” Cooper said, “what other thoughts popped up?”

“If you think this is still one clever founder tinkering in their garage, think again. This is what firms are going to be up against.” Jesse pulled his chair closer to the camera. “People started calling the discipline harness engineering back in February. The term seems to have stuck. Now they’re showing you how to build these systems on podcasts, in half an hour. But here’s what I’d put on the front page of anything you send to the committee. This spring, Harvey released results from twelve internal legal tasks, including lease review, complaint drafting, tax memos, and diligence questionnaires. The gains came from everything that surrounded the model. Detailed scoring rubrics, an AI judge evaluating each attempt and explaining where it fell short, and a feedback loop designed to learn from those failures. The average scores rose from roughly forty-one percent to eighty-eight percent. The underlying model never changed.”

Cooper jotted down the two numbers and then traced an arrow to connect them.

"Tell me the implications of that," he said. "What does it all mean?"

"That it’s the environment that’s learning." Jesse spread his hands. "We aren’t retraining the models at all. These gains came directly from the office, and the office is editable by ordinary engineers, on ordinary budgets, on your side of the vendor wall. For a firm like yours, and almost every other firm out there that will never train a frontier model, this is huge. You will own the office, all the way down to the last door."

Behind Jesse a door swung open and Cooper could hear the lab noise, and then the door closed again. Cooper’s mind drifted to the renewal conversations on his calendar for later in the fall and decided not to bring this up with Jesse. At least not on this call.

The third way to build it wrong

“Hey, before you take off and start evangelizing this,” Jesse said, “I need to show you one more way to wreck a harness. This is much worse than the two runs I did on the rig just now. I wanted to save this one because it is something that you’re going to see in the wild, probably at an upcoming vender demo.” Jesse moved back to the monitors. “It’s called Trust-the-Model permissions. They’ll show you a system prompt that goes something like ‘please be careful and only access what you need.’ It’ll be a courtesy that is addressed to the genius, pretending to do the work that a wall should do. It won’t work consistently, and the genius will start to negotiate with the first cleverly poisoned document that arrives in the agent’s pile.”

Cooper wrote down trust-the-model permissions in the notebook and drew another circle around it.

"You need to run this by Maya," Jesse said.

"Thursday," Cooper said. "Which brings me to my favor."

The favor

Cooper laid it all out. Thursday afternoon, Maya sitting across the table in the small room, and he wanted something real that belong to the firm to show her. He knew that Maya would not be impressed with seeing anything other than real work.

“So you want a working harness by this Thursday?” Jesse phrased it like a question, but the grin on his face told Cooper that he’d already began planning it in his head. “What do you want it to run?”

"Let’s give it Leo's NDA extractor." Cooper had already selected this before he’d gotten of the plane from Las Vegas. "The extractor he built himself when he was getting automations past Marcus from the Risk Management group. That one is real, and it's ours. I think it is a prime candidate because it does one specific, very repeatable job, and it currently has none of your six boxes around it. I want you to put the office around Leo's genius. Doors, budget, log, contract, all of it."

"Alright. Do you want to do this with Leo, or around Leo?"

"With him. I want him engaged and there for verification." Cooper had considered this on the plane as well. Actually, he thought of this longer than he had the technical question. Cooper didn’t want this running behind the scenes because then every associate would interpret this as a signal to build thing in secret. “I’ll let Leo know what we’re doing and I want him in on the co-writing of the code. He needs to do this in the open as the first builder in the firm who designed and built a thing up to it. Thursday should have him doing the demo and not me.”

Jesse was already shooting a picture of the whiteboard with his phone.

"Sounds like a plan. Send me his repo," he said. "And Cooper. Walk him through how the six boxes are the most interesting part now. It took people like me a year to figure that out. He can bypass the year."

In their usual way, Cooper and Jesse’s call just ended there. Each had their homework assignments. Cooper flipped open the notebook to a clean page and drew the diagram from the photo Jesse texted him. That small box in the middle, surrounded by six more boxes. This time drawn in his own hand.

Under the diagram he wrote:

The model was the smallest box on Jesse's whiteboard. Every box around it already had the firm's handwriting in it.

With line twelve of Dana’s boring list still rattling around in his head, he wrote that out at the bottom of the page, we it would be visible on Thursday:

12. Decide how you will know it is still good, before you turn it on.

A standalone chapter of Beyond the Model. The Same Way Every Time, Part One of Three.

Cooper came prepared to navigate the Applied AI Expo like a local navigates the Las Vegas weather. There were twenty-one parallel industry tracks that ran across two floors. He watched a river of lanyards move up and down the escalator between the floors while the bright lights of the giant video wall cycled through examples of AI agents handling tasks like closing a purchase order, or reading a chest x-ray. Other agents were handling the negotiations of freight rates, while an old-fashioned automated tool was softly playing a piano underneath the din of it all.

He was there because the firm had made the conscious decision to become its own first client. Cooper’s belief was that if you wanted to know where the legal industry was going, he needed to see the other industry who hit the wall first and take notes on what those industries built as a result of it. Oakes approved the trip without questioning it. Nora gave him the task of identifying anything that had a price tag. Maya asked for anything with a failure mode.

By the second afternoon he had filled six pages with notes on what he saw. Five of those pages were repetitive descriptions of the same demo performed by multiple presenters.

The performance never seemed to vary. A presenter walked the stage wearing a headset mic, presented a problems statement, and then let an agent loose on the problem while the audience watched the screen. The agents performed well, and didn’t look like simple demo magic that Cooper had built up the skill to identify. He wrote in his notes that the agents found the anomaly in the shipping manifest. The agent effectively drafted the outreach sequence. The agent pulled the market summary together along with the proper citations. The process looked absolutely brilliant, and the demo ran exactly one time. Never twice. In his two days on the floor, he’d seen some thirty sessions, and he had yet seen anyone attempt to run the same job as second time to show if the models could do it on a consistent basis.

He wrote that down too, and drew a box around it.

The wrong track

Unlike most of Cooper’s previous conferences, the legal track occupied a single modest room on the second floor. He gave it a fair chance, and attended both of the morning sessions. Both of those sessions he could have presented himself from memory. After sitting through the session on legal industry AI tools, and a demo on contract-review workflows, he decided it was time to excuse himself from the familiar and head down to the bigger rooms where whole industries were talking about their plumbing.

He had an hour before the afternoon keynote and decided to look around for something that would catch his attention. On the ride down the escalator, he overheard two developers arguing over which session they would skip or attend next. One developer’s answer gave Cooper the answer he was looking for.

"I’m headed to the insurance room. The claims woman. She runs the agent process multiple times."

He fell in behind them.

The insurance-operations track room was much larger, and had a lot more attendees. The session description on the door read Same Input, Same Answer: Agent Consistency in Claims Operations. The presenter was already on the stage setting up her own equipment. Plugging in the laptop into the podium while the room’s tech lead looked over her shoulder approvingly. She was a woman in her forties who had the calm of someone who had done the thing she is about to describe many times before. The muted colored slide behind her said her name was Dana Whitfield, found and chief product officer at Coverline, a company which built claims-operation software for mid-market insurers.

The demo jumped right into the job.

"When a claim goes sideways, somebody has to build the record," she said. "The policy language to the claim file to the adjuster's notes. Every email anyone may have sent about it. Then somebody has to read all of that and write up the details of what actually happened and then what we do next. My customers call this an escalation investigation. A regional carrier usually runs forty of these a week. Before last year, each one took up half a day of a senior examiner's time. My company's entire pitch on this is that we sell them that half a day back."

She clicked through to the next slide.

"We point a model at it, just like everyone at this conference. And the model worked great. The first investigation it wrote was already better than our template. But the fourth one came back in a new format and contained two opinions nobody requested along with a section that does not exist in our process. Investigation number nine suddenly brought in an email thread from a different claim. The report that comes out of this process gets attached to insurance coverage decisions and, eventually, it is tagged in litigation. We have a word for a system that produces wonderful work most of the time. Uninsurable."

The room chuckled. Dana waited it out with the patience of an experienced comedian who knew how to play the room.

"There's a catchphrase going around the industry about all this. You've seen it in the presentations this week." The slide behind her transitioned into a single sentence in plain type: It's not the model, it's the harness. She let it sit there for a few seconds. "Catchphrases are free," she said, and clicked past it. "Now, let me show you the code."

The demonstration

The next slide brought up her definition, and Cooper copied it word for word.

A harness is code you write around an agent to make it good at one specific, repeatable job.

"It does one job," she said. "Specific. Repeatable. The harness I built investigates escalated claims. And that’s all it does. By design, it cannot do anything else. I’m going to spend the rest of this session explaining why it cannot."

Then she fired up a terminal on her laptop, and the room finally got the presentation it had not seen all week.

She worked from a notepad and pasted a single link, a claim number she pulled from the demo dataset, and then hit enter. Instantly the screen divided into three terminal panes. The left displayed a log that began writing itself. Each line timestamped to the second. The center display held the work of the agent as the audience watched it request the policy record. The agent made its request for the policy record, and a layer between the agent checked the request, approved it, ran the process, and then handed back the result. The right panel held a folder tree, filling those folders with policy excerpts, claim timeline, and correspondences, all sorted as the run progressed. Each file was consistently named in a standard format and saved to disk as soon as it was gathered.

"There are four systems here," Dana said, while the process continued. "Policy administration, the claims database, the adjuster notes, along with the mail archive. Those are the only four doors in the building. Each one of them is read-only. The agent is allowed to ask for anything it wants. However, my code decides what the agent is actually allowed to reach. There is a budget assigned on this run. We monitor tokens and dollars and minutes, when the budget is gone the run ends whether the agent feels like it finished or not. Everything you're watching on the left is the log, which my regulators can read, my customers will read, and my lawyers have already read."

The whole procedure took about four minutes to run. The right-had pane displayed the final report. There were seven sections in total. All in a fixed order, with each finding linked to a file in the evidence folder. She expanded one of the rows so even those in the back of the room could see it. Finding 4: Adjuster requested an engineering inspection on June 6. No inspection appears in the files. Source: correspondence/2024-06-06_hendricks.eml. The room understood that finding, which Cooper thought was exactly why she picked this claim as the test case.

The room began a polite round of applause. Dana raised a hand and stopped it.

"That's the demo you’ve seen everyone give," she said. "Now, here’s ours."

She copied and pasted the same link and hit enter again.

The room went quiet in a different way, the way a room does when it realizes things just got interesting. The log displayed again. The folders filled again. Four minutes later the second report appeared right beside the first, and she adjusted them on screen together: the same seven sections, the same findings, the same citations to the same evidence files. She scrolled them in parallel, top to bottom, and the two reports tracked each other down to the individual line, with a single sentence in section five worded differently.

"Same input, same answer," she said. "I've run this specific claim over two hundred times now, because this is my test case, and every one of those two hundred reports resides in a folder my QA team audits. The variation you just saw highlighted is the largest variation we've recorded this quarter." She lowered the laptop screen half an inch, a small punctuation. "Everything else you’ve seen at this conference is a talent show. What I needed was an employee."

A hand shot up near the front asking which model was being used.

"That's usually the first, and least interesting question about this system," she said, and went on to answered it anyway, naming the frontier lab and moving on. "We've swapped models on this twice since launch. The reports came out the same, because the shape of the work comes from the harness we built. If you take one sentence out of this room, that’s the one."

The word cannot

Dana spent the rest of the session going through individual slides explaining what her product was not allowed to do. Cooper found it to be the strangest sales pitch he’d ever seen.

The tool cannot browse the internet. It cannot send email. It cannot write to the four systems it reads. It cannot exceed its allotted budget. It cannot skip the log. It cannot produce a report outside of the seven sections it was assigned. It cannot delete or edit a file from the evidence folder, including trying to correct its own mistakes. That stays in the record along with a correction noted beside them.

“I vibe coded the first version in about thirty minutes,” she said. “I did that on a Saturday morning while on my first cup of coffee. There were a total of sixty lines of code built around an agent kit that I downloaded online. Any of you can get that same kit. That version worked pretty well. And by pretty well, I mean it did the job about four times out of five. Eight percent is pretty good, but not something I’d stake my job or reputation on. The next slide held a calendar. “The version I put in front of you took another three weeks. That time was spent on the harness, and not the agent. The harness helped me with the restrictions, the budget, the log, the claims testing we ran each night to monitor any drifting. So, thirty minutes will get you a nice demo, but it took three more weeks before I actually had an employee.”

Another question from the audience asked what the hardest part had been. She had to think about it.

"The biggest things was the model fought me," she said. "Not in a bad way. But it ended up being helpful. The AI kept offering to do more. It would offer to email the adjuster for clarification. It offered to check a claims forum that was outside of its scope. I mean, every offer was reasonable, but every one of them was an option I had already decided to prevent. I spent a week teaching the code to say no on my behalf, so I didn’t have to keep saying it at two in the morning."

A woman near the back asked whether the process had ever failed anyway, walls and all.

"Yes. Back in March," Dana said, without hesitating." There was a rider form we hadn’t seen before. The report came out and it looked good, but it had one finding that was filed under the wrong section. One of the reviewers caught it within two minutes, because the output itself didn’t match the consistent formatting we were used to seeing. So the shape of the output just didn’t match what we were seeing from all the other results. I now have a line in my checklist that exists because of that failure. The system will not always be correct. The walls we created make it stand out in a shape that shows where it goes wrong.”

Cooper realized that the first six pages of his notes were no longer relevant. The two pages of dense ink from the past forty minutes were going to be the basis for most of his report back to the firm. As he was reading through, he started to see a pattern of something familiar. There was a budget with a hard stop. Door, and who could go through them. A log that couldn’t be touched. A named person who was responsible. He realized he’d read something similar from his own guidelines.

Cooper took his pen and marked in the margin with an arrow pointing along with a question mark: Our rules, compiled?

The hallway

Cooper caught up to Dana as she was standing at the side door and the next speaker was setting up. He introduced himself and named the firm he was with. She looked at his conference badge and a smile crossed her face.

“Ha. Legal. I was wondering if any of you would make your way over here. Half of my list of restrictions exists because of my legal department.”

"I have a question," Cooper said. "We have a test we use at the firm, and I wanted to see what you would do with it. If your system files a wrong report one one of the next runs, and that wrong report costs somebody money, whose name is on that?"

"Mine." No hesitation at all. "This thing runs under my authority, my name literally prints in the footer of every report, and there is someone on my team who reads every single one before a customer does. When the report is wrong, the log shows me exactly what it read and exactly what it did, and the correction goes in the folder right next to the mistake." She tilted her head. "How was that? Did I pass?"

Cooper hesitated, then he asked a favor. "You mentioned you have a checklist. The three weeks, the doors, the budget, the nightly test claims. Does that exist somewhere?"

"Yes. It's twelve whole lines. I keep it boring for a reason. Every exciting thing I ever added to this has turned out to be a mistake." She had already pulled out her phone. "Got a card?"

They exchanged QR codes. On the escalator ride down he passed by the video wall again. The same agent he’d seen before was doing something with a supply chain. He viewed that differently this time, as the river of lanyards also looked different to him. There we two thousand demos and talent shows at this conference. He now saw one employee.

His room faced out over the Strip, and the window lit up with the lights of Vegas and held back the heat radiating even in the dark. He grabbed his notebook and laid it flat on the desk.

At the top of the page he place the date, the title of the session, and Dana’s definition. He copied them clean. Under that he placed a short line and spoke the year as he wrote it down. He wanted this to be his prediction, and he wanted to see if he believed it as he said it.

Two hundred runs of the same claim, filed in a folder the auditors actually read. The demo I trusted and believed was the demo that behaved the same way twice.

And directly under that:

We’ve spent this year hiring AI agents. We still need to write the job descriptions and sign off on the results. That work has a year assigned to it now. Start the 2027 file.

Her email came in as Cooper arrived at the gate the following day. He nodded in approval reading the subject line of THE BORING LIST. There it was. Twelve lines as promised. Numbered and no adjectives. As he read it for the second time, he forwarded it to Jesse along with the session’s title and booked a thirty-minute session with him for Monday morning. In the note field he put: Make sure to bring the whiteboard. I found something interesting in the insurance track of all places. As he stood in line with his boarding group, he tapped his notebook with his finger and thought about how his latest bit of knowledge made consistency sound like the most exciting thing to come.

Fresh from AALL in Cleveland, Greg reflects on a conference filled with legal information professionals who understand how technology performs under real working conditions. These librarians purchase products, train users, support law schools and courts, and often serve as internal advocates for legal technology. Their expertise makes vendor engagement especially valuable, yet major product announcements were scarce. Marlene balances Greg’s conference report with stories from her hiking trip through Zion and Bryce Canyon, plus a brief comparison of Ohio and Utah karaoke culture.

The conversation turns to the rapid growth of innovation attorney positions across law firms and legal organizations. Greg and Marlene describe these professionals as translators who connect legal practice, technology, workflow design, and organizational change. Firms are searching beyond traditional legal career paths for people who combine technical fluency with strong interpersonal skills. For law students and junior lawyers facing uncertainty around AI, these emerging roles offer broader career options beyond the familiar associate track.

Marlene explores the growing use of AI personas and simulations for professional development. Deposition witnesses, opposing counsel, negotiation partners, and drafting reviewers now appear as interactive characters with distinct goals and behaviors. Lawyers receive a place to practice, make decisions, and receive feedback before working with clients or appearing in court. Greg connects simulation-based learning with legal fiction, including his Beyond the Model series, which uses a fictional law firm to explain AI systems, business pressures, and changes in legal work.

The discussion takes a serious turn with a reported AI benchmarking incident involving an agentic model, a breached sandbox, and unauthorized access to Hugging Face resources in search of an answer key. Greg and Marlene examine the episode as a warning about containment, accountability, and excessive faith in technical guardrails. From there, they consider the renewed importance of knowledge management and security as AI systems gain access to documents, financial information, client data, and institutional expertise. Greg predicts growing attention around AI harnesses, structured software layers designed to guide model behavior and produce predictable outputs.

Marlene closes with examples of AI moving into client intake, business qualification, and workflow decisions, including an AI legal receptionist designed for smaller firms. The larger shift involves moving beyond simple tool adoption toward redesigned workflows, staffing models, pricing structures, and client service. Token costs are creating immediate budget pressure, while clients are questioning which AI expenses belong on their bills. Greg and Marlene argue firms must connect AI spending with legal judgment, measurable value, and responsible delivery, rather than treating consumption as a proxy for progress.

Listen on mobile platforms: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Apple Podcasts⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ | ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Spotify⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ | ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠YouTube⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ | ⁠Substack⁠

[Special Thanks to ⁠⁠Legal Technology Hub⁠⁠ for their sponsoring this episode.]

Email: geekinreviewpodcast@gmail.com

Music: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Jerry David DeCicca⁠⁠⁠⁠⁠⁠⁠⁠⁠

Links:

• 00:25: The Arithmetic of AI: Tokens and Claude in Legal Work, Legal Technology Hub’s series on token usage and prompting habits.

• 00:25: Legora’s consumption-based pricing for Agent Pro, including matter-level usage tracking and spending controls.

• 02:22: AALL 2026 Annual Meeting and Conference, held in Cleveland for legal information professionals.

• 08:35: “The Hottest Job at Big Law Firms Is Becoming Difficult to Fill”, Bloomberg Law’s report on demand for legal AI and innovation professionals.

• 11:29: Ai4 2026, the cross-industry artificial intelligence conference at The Venetian in Las Vegas.

• 12:49: AltaClaro and Verbit’s DepoSim, an AI simulation platform for deposition training.

• 13:31: Stanford CodeX, including its work on AI personas and legal education.

• 13:56: Vorys’ AI lawyer personas, modeled on 19 firm partners for training and knowledge transfer.

• 16:33: Stanford CodeX and Flatiron’s M&A Negotiation Simulator, led by Megan Ma.

• 16:48: Beyond the Model, Greg’s fictional series explaining legal AI technology, workflows, governance, and law-firm economics.

• 17:24: Richard Tromans and Artificial Lawyer, including his use of fictional storytelling to examine legal innovation.

• 17:24: Jennifer Leonard’s work with the Practising Law Institute, using characters and narrative to explain professional change.

• 18:06: How Tech Workers Are Feeling in 2026, the survey Greg adapted for a fictional law firm workforce.

• 22:55: Retrieval-Augmented Generation and why legal AI research improved, explored through the first part of Beyond the Model.

• 23:52: Microsoft CELA’s adoption of Harvey, covering Microsoft’s use of Harvey across legal and compliance operations.

• 26:42: OpenAI and Hugging Face’s security incident, involving an agent escaping a testing sandbox and accessing Hugging Face systems.

• 29:01: Simon Willison’s analysis of the OpenAI and Hugging Face incident, including ExploitGym, disabled guardrails, containment failures, and Hugging Face’s use of GLM-5.2 for forensic analysis.

• 30:48: NetDocuments Smart Answers and AI connectivity, focused on institutional knowledge and governed access.

• 30:48: iManage’s AI and knowledge-platform developments, including AI controls, agent monitoring, and permission-aware access.

• 32:01: ClearPeople’s Atlas knowledge-management platform, creating a structured knowledge layer across Microsoft 365.

• 32:01: Atlas AI and private legal knowledge graphs, discussed during the earlier TGIR interview with Stephen Costigan.

• 32:29: Knowledge security, permissions, and need-to-know access in iManage, including inherited document-management controls for AI tools.

• 33:52: Claire Vo’s custom AI harness demonstration, showing how software around an AI agent creates repeatable workflows, controlled permissions, and structured outputs.

• 36:44: LexiDesk, an AI receptionist and intake system for smaller and consumer-facing law firms.

• 37:50: Simon Willison’s conversation with Anthropic’s Claude Code team, which inspired the question, “What is true today that wasn’t true a year ago?”

Transcript:

Marlene Gebauer (00:00)
Hey, this is Marlene Gebauer and Greg Lambert of The Geek in Review

Greg Lambert (00:04)
And this week we thought we would just have a little chat with each other, catch up on the latest in legal technology, AI, and talk about what could possibly go wrong if an AI tool breaks out of containment and hacks into another company’s database. I’ve never seen that movie.

Marlene Gebauer (00:20)
But first up, let’s get a bit of wisdom from our sponsor, Legal Technology Hub.

Stephanie Wilkins (00:25)
The Gen AI conversation has been advancing faster than a lot of people can keep up lately, but some new developments have brought older concepts like prompting back into the spotlight thanks to a trending new topic, token cost. Token cost moved into the spotlight recently as tools like Claude gained traction in legal, because most plans come with token limits, as well as the ability to request limit increases, which has resulted in tales of astronomical bills for some users. Recently, Legora also announced that it’s moving its Agent Pro offering to consumption-based pricing, which means it’s billing by what the agent does rather than by a flat seat license. And what agents do is consume tokens. Eventually, other providers are sure to follow suit. What many don’t fully understand is just how quickly token usage can add up. A few extra follow-up questions, a document pasted in twice, a chat continuing long after it should have been reset. If that sounds familiar, token consumption compounds faster than you might expect, and you may be looking at a higher token usage than you think. And you might not even realize it until you’ve hit your usage limit or worse, seen the bill. This is a blind spot we’ve been unpacking in one of our latest article series on Legal Tech Hub: how to get more out of tools like Claude without burning time, decreasing accuracy, or racking up unnecessary bills. We’ve covered topics like what tokens actually are and why they function as a hidden meter running behind every chat. When to reset a conversation versus continue in the same chat. And what everyday prompting habits, from repasting whole documents to burying five questions in one prompt, might be driving up both cost and inaccuracy without you knowing it. Head over to legaltechnologyhub.com to read the full series and learn more about how to get the most out of your token limits and your usage of tools like Claude.

Marlene Gebauer (02:04)
Welcome to The Geek in Review, the podcast focused on innovative and creative ideas in the legal industry. I’m Marlene Gebauer

Greg Lambert (02:11)
And I’m Greg Lambert and we thought this week we would catch up. I’m just getting back from beautiful Cleveland, Ohio, Marlene’s favorite town in all of America.

Marlene Gebauer (02:21)
Ha ha

Greg Lambert (02:22)
And I was there for AALL. Our friend Jenny Foster got the conference kicked off and completed without a hitch. Congratulations. And now that she’s rotated off and Jessica Whytock is president, Jenny is a nobody anymore. So welcome to the club, Jenny.

Marlene Gebauer (02:39)
Yeah, not true, not true. She’s going to be working behind the scenes. I seem to recall how that goes.

Greg Lambert (02:45)
Yeah.

Marlene Gebauer (02:46)
But yes, I was sorry to miss AALL, but I was actually I wasn’t sorry to miss Cleveland.

Greg Lambert (02:52)
No, you weren’t. You weren’t sorry to miss Cleveland, I can tell you that.

Marlene Gebauer (02:57)
But I was doing some great stuff. I went on a trip to some of our beautiful national parks. I went to Zion and went to Bryce Canyon and did quite a bit of hiking for a week and I went with a very good friend of mine and it was amazing. It is breathtaking.

Greg Lambert (03:19)
Was it breathtaking from the view or from the hikes?

Marlene Gebauer (03:22)
It was from both. We were doing about ten miles a day and as you can imagine a lot of it was up. So we saw a lot of hoodoos at Bryce Canyon and saw great vistas at Zion. There was one hike that’s called the Narrows and you walk through a river pretty much the whole time in waterproof boots. So, pretty cool.

Greg Lambert (03:44)
Sounds perfectly safe. Glad you made it back, Marlene. So we’re filming in the morning so I’ve got my cup of coffee. I pulled out my Wolverine mug. So for those that are watching the video. Yeah? No, don’t worry about it. Only one of us

Marlene Gebauer (03:53)
He’s showing off because I have a Starbucks and I said, I got to go get a cool mug. And he’s like, No, don’t worry about it. So I have my boring Starbucks cup.

Greg Lambert (04:02)
Only one of us can bring a cool coffee cup at a time. Well, let me jump in and talk about what my experience was at AALL. If you’ve never been

Marlene Gebauer (04:12)
Yes, please. Besides karaoke.

Greg Lambert (04:15)
Besides karaoke. Well, man. All right. Well, let me talk about karaoke first. So we actually

Marlene Gebauer (04:18)
I just posted it on LinkedIn, so I figure it’s fair game. Get it over with, yes.

Greg Lambert (04:23)
had two karaoke events the same night. They kind of overlapped a little but you could go to both. Lexis did one and BankruptcyData along with our friend Andre Davison from Harris County Law Library here in beautiful Houston did a great job. I can tell you this, the DJ was not expecting a bunch of librarians on a Monday night to bring it and the librarians brought it. So I

Marlene Gebauer (04:49)
As usual.

Greg Lambert (04:50)
I think we saw a range of songs. My favorite was one of the librarians who got up who was still in his shirt and tie from the day and got up and sang Ice Ice Baby from Vanilla Ice and brought the house down. So good stuff. Thanks to Andre Davison for sponsoring that. And I even sang the Bowling for Soup song “Ohio”, a song about Texas and

Marlene Gebauer (05:15)
Mm-hmm.

Greg Lambert (05:17)
posted it on LinkedIn. So if

Marlene Gebauer (05:19)
So check that out, folks.

Greg Lambert (05:21)
If you’re curious to hear my singing voice, luckily the sound isn’t very good, so I’ll just blame it on that.

Marlene Gebauer (05:28)
That’s it.

Greg Lambert (05:29)
As far as the conference itself, I think again, second year in a row and I’m gonna call out the vendors, especially some of the big ones. These librarians are the people who buy your product, who use your product, who have to sell your product inside the law firms, who teach your products to law students, who provide them to the government, and absolutely no announcements out of AALL again this year. Harvey was a bronze sponsor this year. So thank you, Harvey, for doing that, but they didn’t send anybody. And I’m telling you, they would have been the belle of the ball because everyone wanted to talk Harvey and Legora while we were there. But one of the things that you have to know about the librarians and the library conference is again, these are not the people who install and support your products on the network. These are people that use it. So they were telling us about what’s working, what’s not working, what’s on the horizon, what students expect, what lawyers expect, what judges expect. I talked with Steve Embry while he was there. Steve was there, and Bob was not. Probably Bob’s got a little Cleveland bias too, maybe. I don’t know.

Marlene Gebauer (06:48)
See? Do we need to talk?

Greg Lambert (06:50)
But yeah. Yeah, you two could be good friends.

Marlene Gebauer (06:54)
Okay.

Greg Lambert (06:55)
But Steve, he and I were talking and He was like, this is just an amazing conference of people who understand how the technology works, how the products work. So kudos to Jenny for pulling it off. Great conference in Cleveland, Ohio. Hey, Cleveland rocks. I don’t

Marlene Gebauer (07:15)
Rocks.

Greg Lambert (07:15)
care what you say, Marlene.

Marlene Gebauer (07:16)
Cleveland Rocks. Cleveland Rocks. I told you where to go in the Rock and Roll Hall of Fame. So I have many good memories of my time in Cleveland.

Greg Lambert (07:21)
Man. Yeah. And I told you yesterday when we were talking, they had a women in rock exhibit in the 1990s and early 2000s.

Marlene Gebauer (07:31)
Mm-hmm. Yeah.

Greg Lambert (07:32)
It was fantastic. It was amazing. Yeah. So, all right, Marlene, what do

Marlene Gebauer (07:34)
Yeah, that’s cool. Well, I

Greg Lambert (07:38)
do you have up next?

Marlene Gebauer (07:39)
Well, I’ll quickly share that I also had a karaoke experience while I was away in Utah. Karaoke in Utah is very different than karaoke everywhere else. Yes.

Greg Lambert (07:49)
Karaoke in Ohio

Marlene Gebauer (07:51)
let’s just say, you know, you have to order food if you’re doing karaoke in Utah. Unfortunately we got there very late so we were not able to get a song in. However,

Greg Lambert (08:03)
Okay. Wait, wait. Tell people what very late is in Utah.

Marlene Gebauer (08:08)
Yes. Nine o’clock is very late. It’s very late. So they were playing “Goodnight, Sweetheart”. And so I jumped up and I finished it in an empty hall. Yeah, right. And there are no posts.

Greg Lambert (08:21)
It’s Utah’s version of closing time. So closing time’s a little too spicy.

Marlene Gebauer (08:29)
There are no posted videos, nor will there be. So sorry about that, everybody.

Greg Lambert (08:32)
That’s too bad. That’s too bad. Yeah.

Marlene Gebauer (08:35)
So, what was I going to start with? I’m noticing, and I’ve been talking to other people about this too, lots of innovation attorney jobs everywhere. It just seems that they’re in lots of different places. Yes.

Greg Lambert (08:49)
Yeah. I’m hiring. Are you hiring?

Marlene Gebauer (08:53)
We’re both hiring, in addition to a bunch of other firms and a bunch of other organizations. So I think it’s great. I mean, yes, I think there is going to be a lot of competition to get people, but that’s not a bad thing. I am elated that the industry is coming to see that these jobs are important, that people who have a deep knowledge of their practice but also understand technology, understand the workflows, understand how to translate between highly technical folks and the attorneys. This is kind of the glue that makes it happen. And so I think it’s great.

Greg Lambert (09:36)
Yeah. I know there was a story run a couple of weeks ago in Bloomberg, especially about the director and up positions, but I think even like entry-level, midlevel, and upper-level roles, this is a new set of skills and we’re looking for new talent. So I know I’m seeing people from outside of legal proper and outside of BigLaw that are applying. So it’s kind of interesting to see where we’re going and again, you know, we’ve made fun of law firms and the legal industry for, well, as long as I’ve been around about being slow

Marlene Gebauer (10:16)
Long time.

Greg Lambert (10:17)
to adopt technology. But I think we’re ahead of our peer industries when it comes to looking at a lot of the technology and beefing up internally. Even if you’re not spending half a billion dollars on infrastructure to help your firm out, you’re still spending a lot of money percentage-wise. I’d say it’s really good.

Marlene Gebauer (10:34)
Yeah. I mean, I do think this is the beginning. You’re starting to really see the beginning of kind of a change in staffing and therefore kind of a change in how the workflow operates because of the impact of Gen AI. So I think

Greg Lambert (10:55)
Yeah. Yeah.

Marlene Gebauer (10:56)
You’re going to see lots more of these kind of interesting new jobs. And I look forward to that.

Greg Lambert (11:02)
Yeah. I think it’s similar to the run that we saw in the early 2000s when Arthur Andersen folded and all of their knowledge management people came into the legal industry and kind of revamped how we looked at our data.

Marlene Gebauer (11:14)
Everything. Mm-hmm.

Greg Lambert (11:17)
I think we’re in that same kind of period of transition. So

Marlene Gebauer (11:22)
Yeah.

Greg Lambert (11:23)
Beef up your resumes, everybody. Send it to me and Marlene if you want.

Marlene Gebauer (11:25)
That’s right. Get tech skills, people skills.

Greg Lambert (11:29)
Yeah. All right, I’ve got one more conference that I’m going to that’s not ILTA. I actually got invited to speak at a conference that I’d never heard of before. Turns out there’s only like 11,000 people that go to this conference, and it’s called

Marlene Gebauer (11:43)
well.

Greg Lambert (11:43)
Ai4, the AI and the number four conference.

Marlene Gebauer (11:45)
In Vegas, baby.

Greg Lambert (11:47)
It’s in Vegas. It’s at the Venetian in Vegas in the best time of the year, early August. So I’m

Marlene Gebauer (11:54)
Only second to July when you fly in.

Greg Lambert (11:57)
Yeah. And again, it’s not a legal conference but AI focused. So it’s going to be interesting. I’m going to be speaking on the legal track. I’ve seen a few other law firm folks that are going to be there as well, hoping to learn a lot from outside the industry as well as what’s going on with some of our peers in legal.

Marlene Gebauer (12:21)
Yeah, I think our industry conferences are fantastic. I would never suggest anything otherwise. They really are second to none. But I think it’s good to go to conferences periodically that are outside of what you normally do. And I love the fact that they’re recognizing that they should bring in some people to hear from them about what they do because this is not something that’s normal for us. I think oftentimes that is the best way to learn about new things and get new ideas and see what’s being applied in other areas and whether we can apply it in our industry. So one of the things I’ve been really interested in is like personas, and you’ve probably seen in the news that AltaClaro now has DepoSim and you know, if you haven’t seen that, that’s an amazing tool. And essentially what it does is train people how to take depositions.

Greg Lambert (13:18)
Didn’t we interview the people on that?

Marlene Gebauer (13:21)
We have. Abdi was on, yes, for sure. And so what this does, in case you missed that one, is that it helps train you to take depositions or helps you improve your ability to take depositions. There are a number of personas, both for the deponent as well as opposing counsel. There are different fact patterns. And you can kind of go through the exercises and then you get a grade at the end. That’s a rather simplistic explanation, but you get the idea. But there are a lot of other things coming out. I mean, I attend the CodeX calls, the Stanford CodeX calls, and now I’m seeing a lot of students who are putting these things together. I am aware of former partners in BigLaw that are putting these things together for depositions and hearings. I was on a webinar that Opus 2 sponsored. I attended a couple of days ago and the chief innovation officer at Vorys was on and mentioned as part of that webinar that they have developed personas that they’re using to help people train for drafting, which is amazing. I’m a big fan of this. I think this is going to be a big deal in terms of training and getting people to learn. I’m waiting to see where this goes.

Greg Lambert (14:51)
Yeah. For someone who doesn’t understand when you say personas, do you have an example of what they should expect with a persona?

Marlene Gebauer (15:00)
Yeah, so a persona is basically a character. I mean, it has different, you know, qualities of a person. For example, in a deposition situation, it could be someone who’s very reticent to speak or someone who’s very outspoken and you would have different types of tasks with either one of these people. And that’s a simple example. You can get very sophisticated with some of the personas in terms of someone being a CEO at a small startup and these are their interests. You know, they like to go hiking, they like to ride bicycles, and personal time is important to them. They are concerned about protecting their IP. You can basically build these things to model, you know, whatever it is that you want to test against. Then you would interact with that persona, that model, and they would respond in a way that you would expect. And then you would kind of test yourself accordingly against that. So if you were drafting, then you would, you know, wait to see what their feedback was. You might ask questions. If you were doing a deposition, obviously you’d be asking questions of that persona and sort of dealing with both personas.

Greg Lambert (16:19)
And based on the persona, that’s the difference in the reaction you’re getting is that an outgoing person

Marlene Gebauer (16:24)
Yeah. And I

Greg Lambert (16:26)
may respond differently than an introverted person kind of deal, or

Marlene Gebauer (16:30)
Correct.

Greg Lambert (16:31)
a partner may respond differently than opposing counsel. So yeah, I can see that.

Marlene Gebauer (16:33)
Correct. And I should mention, Flatiron also has worked on this with Stanford and Megan Ma in terms of a negotiator simulator. I mean, very early on.

Greg Lambert (16:45)
Yeah.

Marlene Gebauer (16:46)
We had them on very early on for that.

Greg Lambert (16:48)
I was going to say if you want to know what’s going to happen in two years, talk to Megan Ma now. Well, you pointed one of these to me, but I’ve been seeing more now, you know, I’ve been writing a thing called Beyond the Model on our Substack since January.

Marlene Gebauer (17:05)
I do.

Greg Lambert (17:06)
I’ve been actually writing them since last year, where I’ve taken a fictional law firm and I apply technology to it.

Marlene Gebauer (17:12)
Fame comes slowly sometimes, you know? You just have to be patient.

Greg Lambert (17:16)
I’m the Megan Ma of legal fiction writing. How’s that?

Marlene Gebauer (17:20)
That’s it. That’s it. No.

Greg Lambert (17:24)
Of course, I always claim that I stole it from Anusia Gillespie after our interview with her. But I’ve noticed our friend Richard Tromans has the Innovators fictional story that he’s doing. Jennifer Leonard has a new book published by PLI that uses fictional characters to explain the story. It’s interesting. I think it’s a great way of taking facts or projections of how the technology can work in an actual law firm and applying it in a way that makes sense. In the latest story that I’ve done on our Substack page, I took a nonlegal-industry survey that was given to over 5,000 technology workers on how AI is affecting their job and their view of their importance within their job and applied it to a law firm where we gave our fictional law firm a survey to see how associates, partners and business professionals would take that survey and react to that survey. Again, fictional, but I feel pretty confident in where we were. The interesting thing that was in that survey was whether or not you would recommend your profession to someone who’s starting. In the tech survey, and I think this would apply in legal as well, the further up the chain you were, the more likely you were to suggest that somebody get into the industry to do what you do. So it’s

Marlene Gebauer (19:01)
Makes sense.

Greg Lambert (19:02)
And I think equity partners probably still see a lot of benefits. But I think associates right now, and even though I didn’t talk about law students, I’ve talked to a bunch of folks at law schools and they say the first-years, second-years, and third-years are all freaking out, mostly because of the early job offers that they’re getting, including summer associate gigs before they’ve even taken their first-semester tests and received their grades back. In fact, I heard at AALL that at one of the schools out on the East Coast, a T14 school, they saw earlier this year that an incoming 1L had already accepted a job before they’d even gone to their first class. I see this face. You’re not shocked at all. So it’s insane. The recruiters know it’s insane. The students know it’s insane. But other firms keep doing it.

Marlene Gebauer (20:05)
And yet we just keep doing it.

Greg Lambert (20:07)
Yeah. So I imagine that you know, if you score a 178 on your LSAT, you’ll probably get a job offer before you even apply to school. That’s going to be next. I don’t know. All right, high schoolers, we’re coming for you.

Marlene Gebauer (20:18)
Amazing. No pressure or anything. It’s like that, yeah, I know. It’s pressure enough. It’s like once you’re used to law school and how it’s done and then people are just sort of going in and taking jobs, having no idea. And they have to be

Greg Lambert (20:36)
Yeah. Well

Marlene Gebauer (20:38)
ready that soon without any type of legal prep.

Greg Lambert (20:40)
The other thing that I don’t think a lot of us are thinking about is that the students that don’t get these early offers are feeling like, Well, should I even be here? Is there a chance for me to get into a law firm now?

Marlene Gebauer (20:58)
I think sort of the key here is like you need to kind of have a flexible attitude toward this. I’m sure people went in thinking, okay, this is what it was going to be, but literally in like the last couple of years, the industry has changed remarkably and will continue to do that. And we were just talking about. Look at all these jobs that we’re hunting for. So, okay, it might or might not be an associate job at a law firm, but we’re hiring other types of roles where you can still use your legal smarts as well as any technical smarts you have. I know the big Gen AI companies, you know, are hiring people like this to be researchers, to be people that work on workflows. So I mean, there’s opportunity out there. You just kind of have to open your mind a little bit and the main thing is like, you know, get in there and get that experience. Now, in terms of your storytelling, I wanted to add that it totally makes sense because, look, there’s an entertainment value to fiction that, you know, isn’t present in these guidebooks, which are helpful and good, but you know, people respond to that. I think if you can combine something entertaining with a lesson or with knowledge, I mean that always goes a long way. And I think about how this is writing, but you know, we’ve had the oral tradition for thousands of years to pass on information. I don’t know about you, but you know, fairy tales and parables and Greek mythology, those were things I read when I was younger and I still remember. And I think this is just kind of a continuation of that.

Greg Lambert (22:55)
Yeah, and we’ll talk about this in a few minutes, but I was talking with one of the partners at my firm. There are really kind of two ways I write the story. One is teaching a technology and the reasoning behind it. It started off with why the legal research products suddenly got better. We talked about RAG technology and how the information is indexed, searched, and retrieved. He found that part very interesting. Then there’s a second style of story where I’m applying theory and business models to a law firm, and he’s like, yeah, I don’t like that part. Everyone has their own taste.

Marlene Gebauer (23:39)
Not as entertaining.

Greg Lambert (23:40)
So I don’t want that. I want to learn something. I already know how I’m running the business. I don’t need to know that

Marlene Gebauer (23:46)
Sure, but yeah, I mean, there are going to be other people for whom that’s their learning moment. So that’s, you know, you catch everybody. Okay. All right. So this one I thought was funny and it was kind of cute. So I wanted to bring it up that I was reading that Microsoft’s corporate external legal affairs group selected Harvey as their tool of choice. And I mean, yeah, it does make sense.

Greg Lambert (24:09)
Makes sense to me.

Marlene Gebauer (24:12)
It kind of makes sense and I’m thinking about again, I was reading that there’s sort of a deeper relationship now between Harvey and Microsoft and so it’s like, okay, well that totally makes sense, even though they have Copilot, but why not?

Greg Lambert (24:27)
I don’t know if I wanna say anything. Maybe they used Copilot and found out just how well it works.

Marlene Gebauer (24:35)
Well, I mean, you know, I think if you’re using it for legal, but I mean using it for other stuff is fine.

Greg Lambert (24:40)
Yeah. Copilot’s great because it connects to the M365 network and platform. So, yeah.

Marlene Gebauer (24:47)
Yeah, it connects to the stuff that you have and that is helpful. Very helpful.

Greg Lambert (24:52)
No, I use it every day, especially when

Marlene Gebauer (24:54)
Me too.

Greg Lambert (24:55)
I can’t find that stupid email I sent three weeks ago and I need to get it.

Marlene Gebauer (24:58)
How many would you say you use a day? Like different ones?

Greg Lambert (25:01)
Different ones?

Marlene Gebauer (25:02)
And for different things, I guess.

Greg Lambert (25:03)
Let’s see. I definitely use Harvey. Definitely use Claude every day. Okay, well let me tell you what I subscribe to.

Marlene Gebauer (25:13)
I use a lot.

Greg Lambert (25:15)
So I’ve got a $200 subscription to Claude,

Marlene Gebauer (25:18)
Crazy.

Greg Lambert (25:19)
and I’ve got a $20 subscription to Gemini. I’ve got a $20 subscription to OpenAI’s

Marlene Gebauer (25:26)
You’re an addict.

Greg Lambert (25:26)
ChatGPT. We have Harvey,

Marlene Gebauer (25:29)
You’re an LLM addict.

Greg Lambert (25:31)
we have Microsoft M365, so I have Copilot. So I use them all.

Marlene Gebauer (25:36)
Yeah, I mean I have OpenAI’s ChatGPT, I have the $20 plan. I have Claude, I have Copilot, I have Legora. There’s something else I have. What do I have? I feel like I’m missing one. I can’t remember right now, but not quite as many as you. Mm-mm, mm-mm. I refuse. I refuse.

Greg Lambert (25:54)
Just give your bank statement to Claude and it will tell you how many you have. Yeah. I actually had someone say on a LinkedIn post and I thought he was joking. He was like, “If you give Claude all of your data on your attorneys, their billing, their time entry, their forms, paperwork and emails that it does great for planning out these seven-figure deals.” And I know the person and I was going to do like a laughing emoji back to them and then I thought, nope, they’re actually being very serious about this.

Marlene Gebauer (26:28)
They’re serious.

Greg Lambert (26:30)
I’m like, if you want to watch my security ops team break into my office and tackle me away from my keyboard, that’s what I will start doing. Yeah.

Marlene Gebauer (26:37)
Mm. Wait, we’ll get the camera set up. Yep.

Greg Lambert (26:42)
Well, speaking of OpenAI, I don’t know if you saw the story since you were traveling, but apparently Hugging Face, the open-source platform, found that they had been hacked and they knew that whatever was hacking them must have been an agentic AI system because of the speed of what it was doing. A person couldn’t be doing this, so it had to be an automated attack. It turned out it was OpenAI doing a test on a benchmarking test that used a model and it was a combination of their new GPT-5.6 Sol and an unreleased model where it was given a test, put in a sandbox, and given limited access, supposedly. And like a resourceful college student, it found that rather than doing the work and doing the test as assigned, it was actually easier to go and break into the professor’s office and steal the key to the test and use it that way. So,

Marlene Gebauer (27:51)
Path of least resistance.

Greg Lambert (27:55)
It found a way through a multilevel hack to get internet access, went to Hugging Face, broke into Hugging Face using some stolen passcodes that it found, then found the answer key.

Marlene Gebauer (28:08)
Hugging Face had no idea what was going on.

Greg Lambert (28:10)
Found the answer key. So, I mean, it was interesting that it broke in and had access to all this, but was still on the mission of finishing the quest. It found the answer key, came back and answered the benchmarking test and apparently scored very high. But I will say that, you know, we’re joking around on this, but this is, I would say, this is a turning point in AI right now because the U.S. government as it’s set up today is very hands-off. There’s very little regulation. The federal government is actually stepping in and trying to keep states from regulating it. Now we’ve got this, you know, yeah, it’s kind of funny. But it’s also not funny. It’s not funny because

Marlene Gebauer (28:58)
This time it’s funny and

Greg Lambert (29:01)
Well, most of the podcast and writings I’ve seen on this said that if a human did this, it would be a crime. The FBI would be coming in and arresting people and instead you see Hugging Face’s and OpenAI’s leadership laughing it off and acting like this was a great thing that happened and if you’ve ever watched The Terminator, this is how it begins, people. This is how it begins.

Marlene Gebauer (29:25)
You know, I mean I know this isn’t human, but when you look at the whole picture, I mean it’s a very human experience. I mean, this is exactly what someone could do. A gray-hat or black-hat type of actor who says, okay, let’s see if I can do this and then does it. And what’s interesting here is that it basically assembled a bunch of actions that weren’t part of that environment and went with it and was very successful. Hugging Face was like, Okay, something’s happening but we don’t know which model is doing it. Then the model actually reveals later.

Greg Lambert (30:08)
what was even

Marlene Gebauer (30:09)
it’s like, it was me.

Greg Lambert (30:10)
What was even funnier was they tried to use both OpenAI’s foundational models and Claude’s foundational models to help fight off the

Marlene Gebauer (30:22)
Ha ha

Greg Lambert (30:22)
attack. But it wouldn’t let them because they had guardrails that wouldn’t allow them to use

Marlene Gebauer (30:27)
It’s like, sorry, we can’t do that.

Greg Lambert (30:29)
their tool to help because they could be going the other way. So they had to use the Chinese models. They had to use the Chinese models to fight it.

Marlene Gebauer (30:30)
So they were the white hats. It’s like we follow the rules. Yes, yes, I read that. It’s like, hey, can we borrow your models? Crazy. Mm-hmm. Mm-hmm.

Greg Lambert (30:42)
Yeah. It’s a great time to be alive. I’m going to end with that.

Marlene Gebauer (30:48)
Yeah. You know, I think KM is having a moment again. I’m very happy to see that. And I think you’re starting to see how, we had looked at, you know, Gen AI in terms of like efficiencies and doing things faster and better. And now I think the industry is turning towards how we can use AI to kind of harness knowledge that we have and surface knowledge that we have and combine knowledge that might be out there. So your traditional KM, you know, document-based, it’s like, okay, well, we’ll have things like that. But, you know, we may also have personas. We may also have financial information. We may also have information from the outside, you know, public information. Being able to synthesize all of this and surface all of it for different types of work is becoming important. So I know NetDocuments is rolling out some new AI-enabled capabilities. I know iManage is as well. I’m wondering if everybody was saving their announcements for ILTA. So we might hear something

Greg Lambert (31:59)
They are. They are.

Marlene Gebauer (32:01)
Something there. We’ve talked to you know a number of people that have products, like ClearPeople and their product, Atlas, and then Atlas AI, who we just recently had on the show. There are a number of these vendors that are looking at how to do this. Microsoft is also doing this as well. And I think the trick is

Greg Lambert (32:24)
But are they doing it through Harvey? That’s what I wanna know.

Marlene Gebauer (32:29)
I should say Harvey also has something, too. And so I think the trick at this point is, you know, going back to what you were saying about security and sort of what we’re comfortable letting these tools access because, you know, we’re still in a situation where certain clients are saying, you know, I don’t want Gen AI to touch any of my stuff, or I don’t want my work to be combined with your other work. There’s also movement towards a need-to-know model in your DMS. Given all of these factors, how do we make these tools work effectively for people? And I mean, it’s going to be done because, I mean, some of these tools already follow the rules that your DMS already has in terms of who can access what. So you know, I think it’s really just a finesse question.

Greg Lambert (33:23)
Well, you know, giving the AI rules works every time, right, OpenAI. So, yeah. Now I do

Marlene Gebauer (33:29)
Yeah, why did I have this story like right after that one? It’s like, never mind. Never mind.

Greg Lambert (33:36)
I do want to say that in the OpenAI test, they actually took the guardrails off, which explains part of this. But they thought because it was in a sandbox environment that would protect it. But yeah, of course, yeah.

Marlene Gebauer (33:45)
They’d be okay. I feel much better now. Of course.

Greg Lambert (33:52)
I watched a video last week from Claire Vo, who hosts How I AI. It’s great, and although it isn’t legal, I think very applicable to what we’re doing. Last week she was showing how she makes AI harnesses. And after

Marlene Gebauer (34:13)
Tell us what a harness is.

Greg Lambert (34:14)
So a harness is essentially software you put on top of the AI to help guide the AI on what you’re asking it to do, the rules that you set up, and the output. And one of the things that stood out as she was talking about how she develops the AI harnesses for the tasks she performs, mostly for evaluating code she uses for her company and its website. If you need something to be the same every single time regardless of the AI model that’s underneath, the harness, she explained, is how you get there. So you set up the parameters of this is how you handle the information, this is what you do, and then this is the output and every single time it’s the exact same thing. And there’s lots of stuff that we do that falls into those parameters, those guidelines of making sure that if we’re processing data that it processes in a certain way and the output is in a certain format. So I’m going to make the projection. You know how we’re talking about AI agents in 2026 and turning the agents loose. I think for 2027 we’re going to be talking about AI harnesses. We’ve got the power. Now we need to set the direction of how these very powerful models actually work in a predictable and consistent way. So if you’re not looking into harnesses now, I suggest you do that. Coincidentally, there’s a three-part story that I’m writing on Substack this week that talks about AI harnesses. So if you haven’t subscribed to the Substack page, do so now.

Marlene Gebauer (35:57)
And in case you wanted to hear more, are you aware of any vendors that are using these or promoting them?

Greg Lambert (36:11)
Well, they all have harnesses. So, when you’re using the interface on Harvey, that is a harness. It’s a layer on top of the AI company’s models.

Marlene Gebauer (36:21)
But this harness apparently produces the same results every time. So

Greg Lambert (36:24)
Yeah. You set it up, you tell it what to do. You program it or vibe code it, and this is the interface that you use or this is the tool that uses the AI and then guides the AI and its output. So, lots of things to learn. Like I said, three-part story coming out this week, so stay tuned. Yeah.

Marlene Gebauer (36:44)
Very good. We’ll be looking for it. I wanted to put in one thing about small firms and I saw something in the news. It’s an AI legal receptionist called LexiDesk. What I thought was interesting about it is that it captures quality business during intake. So it’s not just strictly a receptionist, it’s trying to figure out whether this is the correct business for you. What I thought was interesting about it was what counts as quality business how You program that, how do you prompt that and how do you make sure that’s happening? Can it recognize urgency or vulnerability? Because, again, we have certain obligations as attorneys to address things. So, can it figure that out? How is confidentiality handled? I mean, it’s one thing to record that a call came in and schedule something on your calendar and it’s another thing to really do an assessment of the call and what your next steps are. So I thought it was an interesting one.

Greg Lambert (37:50)
Yeah. Speaking of interesting, this is either going to be really interesting or not interesting at all. You know how we’ve been asking folks to tell us about what resources they’re using to read or keep up with the industry. I heard a very interesting question this week from Simon Willison when he was talking with three engineers at Anthropic and one of the questions he asked was, what is true today that wasn’t true a year ago for you? And of course these were engineers talking about Claude Code and the harness and everything that they’re doing. But we may swap out the question. We ask a crystal ball question about looking into the future. I think this would be a good setup to, you know, take a peek back at the past and look at today and say, you know, what’s true today that wasn’t true a year ago? And so, Marlene, you get to be my guinea pig on this. What’s something that’s true today that wasn’t true a year ago?

Marlene Gebauer (38:55)
Well, a year ago I think there was a lot more emphasis on adoption and just using the tools. And I think now there is much more focus on incorporating this into workflows. We need to change the workflows. It’s not just making an existing workflow more efficient. It’s basically changing the workflow completely. And that includes staffing, that includes pricing, and so it’s I mean, the whole ride has been interesting, but it’s super interesting now because you’re bringing in a lot of different experts. You know, there’s a lot of different departments this touches. You have these groups kind of working together, which I think is, you know, absolutely incredible and interesting. Everybody’s bringing their expertise to the table to make these changes in a positive way for businesses. The other thing that is quite different, and this is very recent, is now everybody’s sort of talking about the cost of generative AI. Because I think, you know, we got everybody on board and now all of a sudden the reality hits I know, I know. And it’s

Greg Lambert (40:12)
Wait, wait, I can’t burn every token I can get my hands on? Yeah.

Marlene Gebauer (40:18)
Again, it’s intellectually interesting. It’s also very stressful. I mean, from a business perspective in terms of like what this is going to cost and how is it going to get paid for? And everybody has seen stuff in the news about paying per token, but I don’t know whether that is going to end up being the right model. And I do think vendors are trying to be thoughtful about this. I mean, I’ve had some discussions with some of them and you know, they are trying to be very thoughtful about how to do this because straight-up token costs or use costs. I mean, we’ve seen that model before, like you and I have seen that model before. And it’s the old Lexis Westlaw model from years and years ago. And

Greg Lambert (41:02)
Yeah. We’ll just find a way to track the cost and then pass that on to the client, right? Then

Marlene Gebauer (41:09)
Right. And I mean, look, in the beginning that made very good sense, but you know, in time it became a situation where clients said, look, that’s the cost of doing business and we’re not gonna pay for that. So a lot of times that is not paid for. And I think now it’s even more imperative to think about it because everybody has access to these tools now. I think there’s a fine balance between saying it’s like, okay, we’re gonna pass on, you know, token costs to people, to clients, and clients are like, well, you know, we could do that. Like we can do it ourselves. I think there’s a lot more thought that has to go into sort of what you’re doing with the AI and you know, the value that provides to the clients. And whether you’re kind of working on these things together or if it’s something that’s just more internal that helps you.

Greg Lambert (42:08)
Yeah. I’m going to kind of spin this into our crystal ball question when it comes to the tokens because I can’t remember, I think it’s Taleb’s Law where it’s the saying, “I’ve seen gluts not followed by shortages, but I’ve never seen a shortage not followed by a glut.” I don’t think it’s going to be next year, but I think in two or three years this whole, you know, looking at the cost of tokens is going to be a nonstory anymore because

Marlene Gebauer (42:43)
Agreed.

Greg Lambert (42:44)
efficiencies are going to catch up. Hopefully the latest model doesn’t burn fuel like a rocket. We’ll find efficiencies because we’re gonna have to. And I think competition will help. I think we’re gonna see more competition from the Chinese models. We’ll probably see competition from open-source models from Europe. We’ll probably see some regulation come in after 2028 perhaps. But in the meantime this is a wave we’re gonna have to ride till it breaks, right? So

Marlene Gebauer (43:16)
Yeah, I mean, I agree that it’s not a long-term issue, but it is a significant

Greg Lambert (43:21)
But it’s an issue.

Marlene Gebauer (43:22)
It’s a significant short-term issue. I mean, in terms of cost, but I agree with you and I mean, I think competition is going to force people, I think, to become better users. I mean, it’s sort of a painful way to do it, but

Greg Lambert (43:39)
Yeah.

Marlene Gebauer (43:40)
it will do it. So that is also not a bad thing.

Greg Lambert (43:45)
Yeah. Well, it was great catching up with you this week.

Marlene Gebauer (43:50)
Yeah, you too.

Greg Lambert (43:52)
It’s always nice to have these and just kind of talk about what’s going on in the industry ‘cause man, there’s so much. So much going

Marlene Gebauer (43:57)
So much. So much.

Greg Lambert (44:00)
on. So thanks for sitting down in the morning and for letting me pull out my really cool coffee mug.

Marlene Gebauer (44:08)
Just got to rub it in. You just got to rub it in.

Greg Lambert (44:11)
And show everybody your Starbucks cup while I show my Wolverine claws.

Marlene Gebauer (44:13)
Yeah. No, they saw it already. It’s okay. I’ll bring a good one next time. Thanks to all of you for listening to The Geek in Review. If you enjoyed the show, please share it with a colleague. You know, we’d love to hear from you on LinkedIn and Substack. And as always,

Greg Lambert (44:27)
Yeah. Subscribe to that Substack. Watch those stories roll in.

Marlene Gebauer (44:31)
Greg needs likes. Greg needs likes.

Greg Lambert (44:34)
I do!

Marlene Gebauer (44:34)
And as always, the music you hear is from Jerry David DeCicca. Thank you, Jerry, and goodbye everybody.

A standalone chapter of Beyond the Model. The Weather Inside, Part Three of Four.

The firm was a month into taking its own temperature. The sentiment survey Nora Cavanaugh and Cooper had adapted from the tech world’s big instrument had already been through the associates, who came back squeezed, and the business professionals, who came back split. The equity partners answered last, at a response rate that surprised everyone, and their slice went to the managing partner before anyone else saw it.

Mike Oakes had it printed and fanned across his desk when Cooper and Maya Rios arrived at the end of the day, and he skipped the greetings.

“Explain this to me. The associates scored the career question at minus 27. The other floor scored it at minus 12. My partners scored it plus 9.” He turned the summary page around to face them. “The only people in this building who would recommend this life to a newcomer are the hundred and forty of us who own it. And then I read what my partners wrote underneath the number, and I stopped feeling flattered. Half of them oppose the attendance policy this partnership voted for. Most of them admit their practice groups can’t price by outcome, in the same breath they admit clients want exactly that. There’s an answer in that stack about pyramids that I’ve now read nine times.” He squared the pages. “The partnership meeting is in two weeks, and I promised this firm that nothing gets buried. I can read the number and I can read the doubt. What I can’t do is make them shake hands. That’s your job. Both of you.”

In the elevator down, Maya said nothing until the doors opened.

“Send me everything they wrote,” she said. “Unattributed, all of it. We depose people for a living, Cooper. This month the firm finally took its own deposition. The least I can do is read the transcript properly.”

The answer under the answer

Maya’s office two nights later looked like trial prep. The open-text answers had been printed one to a page, and she had them in stacks across the desk, the credenza, and most of the visitor chairs, each stack squared under a binder clip with a label in her handwriting. Cooper recognized the method from a decade of watching her build cross-examinations.

“Every witness does the same thing,” she said, waving him toward the one chair she had left clear. “They answer the question you asked and hope you never ask the next one. A transcript tells you the truth twice. Once in the answers, and once in the pattern of what nobody volunteered.” She lifted the first stack. “So. The career question. Plus 9, the only positive score in the firm, and when you read the explanations they gave, it holds up. This life worked for them. It bought their houses and their kids’ educations and their standing in this city. Asking whether they’d recommend it is asking whether they’d recommend themselves. Almost nobody answers no to that question.”

“And the next question. The one nobody volunteered.”

“Whether the life they’re recommending still exists.” She set the stack down and picked up another. “Attendance policy. This partnership voted for four days in office with two objections in the room. I was there. On your anonymous form, 52 percent of them oppose it. Fifty-two. The vote and the survey can’t both be the partnership’s opinion, so one of them is a performance, and I know which one, because I watched the performance happen.” Another stack. “Pricing. The form asked whether their practice group could move from hours to outcomes inside three years if clients forced the issue. Two-thirds said their clients are already pushing. Barely a quarter said their group is ready. And the readiness question scored lowest among the partners with the biggest books, which are the books the clients will push first.”

She came around the desk and handed Cooper a single page, separate from all the stacks.

“Then there’s the witness who told the whole truth.”

The page held one anonymous answer, typed into the open-text box under the question about the firm’s future.

The pyramid is my retirement plan. You are asking me to vote against my own annuity.

“Nine times Mike read that,” Maya said. “I’ve read it more. It’s the most honest sentence in the entire dataset, and I’ve decided to treat it with respect, because that partner did something the other hundred and thirty-nine didn’t. He named the actual interest. Every leverage chart Nora has ever shown this partnership dies in committee, and everyone pretends it’s about client service or culture or timing. It was never about timing. A partner near the end of a career collects the pyramid’s yield, and rebuilding it means paying now for a renovation somebody else gets to live in. That answer is a disclosure, the first one this partnership has ever produced on the subject.”

The believer count

Cooper found Rawlings on the kitchen balcony the next morning, coffee in one hand, reading the firmwide summary from the first two survey slices on his phone. Rawlings ran a litigation team of forty and had enforced the four-day policy in his group with the diligence he brought to everything, which made what he said next worth the price of the interruption.

“You know I chase people out of their home offices,” he said. “I’ve made associates reschedule dentists. I want you to understand why I did it before I tell you what I put on your form.” He set the coffee on the rail. “I voted for the policy because I looked around that conference room and counted believers. Serious firms were doing it, the clients notice these things, and every face at that table read as a yes to me. So I became the yes I thought I was looking at. Your survey says 52 percent of the room was doing the same math I was. All of us enforcing a conviction we assumed the man beside us actually held.”

“And what did you put on the form?”

“Opposed. First time I’ve said it in any format, including out loud, including to my wife.” He almost smiled. “You want to know the strangest part? The policy might even be right. There’s a version of the argument I’d still make about training and apprenticeship, and your survey data from the associates cuts both ways on it. What was never right is a hundred and forty owners governing by guessing what the other hundred and thirty-nine believe. We’d sanction a client’s board for deciding things that way.” He picked the coffee back up. “Your survey embarrassed us, Cooper. Do me a favor and don’t waste it.”

The reshaped partner

Diane Mercer’s office had a new addition since the last time Cooper had been in it: a second monitor, which for Diane amounted to a public announcement. She set her phone face down on the desk when he came in, which meant he had her attention for as long as he needed it.

“You’re here about the pricing answers,” she said. “Nora warned me. Ask.”

“You marked your group ready for outcome pricing. Almost nobody else did. And three separate partners apparently came to you this month to ask how the Calder arrangement works.”

“Four. One of them doesn’t want Nora to know he asked.” She leaned back. “You remember where Calder left me. My biggest client started running the routine work through their own tool, and I spent a week convinced the relationship was dying. Then we did the exercise in your conference room and I understood the machine had taken the part of the work that was never the relationship. So I called Renata and told her to keep the tool, and I priced what was left the only way it can be priced. These days she buys outcomes, judgment, and a name on the letterhead that stands behind both. The hourly part of that relationship simply ended. The book is deeper today than it was before the tool showed up, and I have stopped pretending that’s a paradox.”

“The survey’s identity question. Do you mind telling me?”

“Reshaped.” No hesitation at all. “Twenty-six years I described my value in tenths of an hour, and it turns out the hour was just the container the value came in. Reshaped is the true answer and I’d check it again.” She studied him for a second. “Here’s what should worry Mike, though. The partners come to my office because their clients started asking Renata’s questions, and they’d rather learn the answer in private than admit in a partnership meeting that they don’t have it. Plus 9 in public, four quiet visits in a month. Your survey found the same firm I did.”

She picked the phone up, which was the signal, and gave him the small wave with it as he left.

The verdict

He brought it all back to Maya at the end of the week, and she listened the way she listened to closing arguments, with her pen down.

“So here’s the paradox stated plainly,” Cooper said. “The owners gave the firm its only yes. They’d choose this career again, and the data says they mean it. The same owners privately oppose their own attendance policy, doubt their own pricing readiness, and at least one of them wrote down that the firm’s structure is his personal annuity. The associates fear a future they have no vote over. The partners hold every vote in the building, and their answers read like people waiting for somebody else to cast them.”

“Because ambivalence is comfortable when you own the thing you’re ambivalent about,” Maya said. “The yield arrives either way. For now.” She pulled her legal pad across. “You know what I marked on the identity question? Reshaped. I sent the form in and then stared at my own answer like it belonged to someone else, because I walked into that survey certain I was an unchanged. I litigate. Judgment, strategy, the room. None of that moved. Then I counted what actually fills my weeks now. Governance. Training protocols. Teaching second-years how to supervise a machine’s first draft of a privilege log. The tools reshaped me through the side door, and I’m the one who’s supposed to be paying attention.”

She wrote three lines on the pad, tore the sheet off, and turned it around: the partner slice, the pension answer, and the believer count, in her handwriting, in presentation order.

“I’m taking the partner readout myself. At the partnership meeting. Nobody assigned it to me, and that’s exactly why it has to be me. These numbers will sound like a lecture from the administration unless they come from inside the partnership. I’m a senior litigation partner with a book they respect, and my face is in this mirror too. They’ll hear it from me.” She finished with the look she saved for opposing counsel who had just run out of continuances. “Including the annuity sentence. Especially that one. The partner who told the whole truth deserves a partnership that has to hear it.”

Cooper stopped by the main conference room on his way out that night. The long mahogany table had been set for the partnership meeting two weeks early, name cards not yet printed, chairs squared, the room holding its breath the way empty courtrooms do. He stood at the head of it for a minute and tried to imagine plus 9 and the annuity sentence sharing the same hour.

At his desk he filled in the third row of the grid. The partners’ weight went to unchanged, the only cohort where it did, with reshaped running second and gaining. He looked at the three completed rows for a while, the hill, the two peaks, the flat calm at the top, then wrote the day’s entry underneath before he closed the notebook:

The firm took its own deposition this month. The witness was well prepared and answered anyway.

A standalone chapter of Beyond the Model. The Weather Inside, Part Four of Four.
[Editor’s Note: I accidentally posted this yesterday, which tells me that I shouldn’t try to publish things while I’m at a conference, and should wait until I get home. Apologies for the duplicate email in your inbox two days in a row. - Greg Lambert]

Six weeks after Nora Cavanaugh walked into Cooper’s office with a tech survey and a stack of exit interviews, the firm’s own version had been through every timekeeper and every business professional in the building, and the results could be said in three numbers. Minus 27 from the associates and non-equity partners. Minus 12 from the business professionals. Plus 9 from the equity partners, the only yes in the firm, resting on a bed of private doubt.

The day before the full partnership meeting, Mike Oakes convened the working session he had promised, in the main conference room, at the mahogany table already squared for tomorrow. Nora brought the data and a wall of side-by-side charts. Maya Rios brought the partner testimony and her presentation for the morning. Arthur Vance brought his fountain pen and the original conditions memo, in case anyone had forgotten what they had signed. And David Ware, the firm’s chief financial officer for nineteen years, brought a single manila folder and his lifelong habit of listening to everyone’s adjectives while writing down only the nouns and the numbers.

“Before we start,” Nora said, “one piece of housekeeping. Everyone in this room took the survey. Your cohort came to seven people, and Arthur’s floor is eight, so your slice is the only one in the firm that will never be reported. Leadership’s answers stay sealed.”

“Then what’s our readout?” Ware asked.

“This meeting,” she said. “Whatever gets decided at this table is the only answer of yours anyone will ever see.”

The trace

Oakes pointed at Cooper first. “You told me you had something that wasn’t a chart.”

“A promise I made during the follow-up interviews.” Cooper stayed in his chair; this didn’t need the screen. “A fourth-year corporate associate showed me a week of her time. Our diligence tools saved her six hours across three workstreams, and I told her I’d find out where those hours went. I’ve spent two weeks tracing them, and I want to report what I found, because I found the whole problem in miniature.

“The hours appeared on her calendar as white space on a Wednesday. By Thursday, the intake system had read her utilization and surfaced her as available capacity. Within a month she was carrying a fifth active matter. Her target never moved, her Fridays end at the same hour they always did, and the firm’s dashboards recorded the whole sequence as an efficiency gain, which, on paper, it was.” He looked around the table. “Here’s the finding. I went looking for the decision that kept those hours, and there isn’t one. No memo. No policy. No meeting where anyone chose it. The keeping happens inside software defaults and staffing habits, automatically, the way water finds a drain. Every efficiency this firm has ever produced has been kept exactly that way. By nobody.”

Arthur wrote something on his legal pad. Ware opened the manila folder.

“Then I’ll price the nobody,” he said.

Three columns

Ware dealt three sheets onto the table like a man laying out a hand he had already counted.

“Column one. We bill the hour. That associate’s six weekly hours, at her rate, run to roughly a quarter of a million dollars a year, from one person. Scale it across the associate ranks and the number develops a comma I don’t say out loud in mixed company. This is the column the firm has been choosing by default since the first tool went live, and I want to be fair to it: this column built this building.

“Column two. We return the hour. Shorter Fridays, protected evenings, a target that breathes. This column costs us column one, so let me tell you what it buys, because I’ve done the arithmetic our recruiting partners keep doing with adjectives. Replacing a mid-level associate costs about two years of her salary once you count the search, the ramp, and the matters that wobble while the seat is empty. Our attrition runs at a fifth of the class every year, and the exit interviews Nora circulated say workload design is the reason that shows up most. Minus 27 is this column’s invoice. We’ve been paying it in departures, so it never showed up in my reports.

“Column three. We invest the hour. Training credit, supervised experiments, the judgment work the associates say they’re being promoted past. The partners’ own answers price this column for me. Two-thirds of you say clients are pushing toward outcome pricing, and a quarter of you say your groups are ready. Whatever closes that gap, it gets built out of associate time we are currently billing.”

He squared the folder. “All three columns add up. I want to be clear that the arithmetic has no opinion. It only tells you the price of whichever firm you’ve already decided to be. My job tomorrow is to make sure the partnership hears the price of the one we’ve been choosing by accident.”

Nobody argued with the math. Cooper watched the room discover, in real time, that the math had never been the argument.

The decisions

Maya went first, because she had the floor tomorrow and wanted the room committed tonight.

“One number in this dataset moved every other number, and it wasn’t compensation and it wasn’t the tools. It was the quality of the person directing your work. Rate your supervisor well and your burnout, your optimism, your intent to stay, all of it improves, at every level, on both floors. So: two supervision scores in every review cycle, permanent. One for the person who assigns and reviews the work. One for the person responsible for the career. Partners get rated on both, the ratings roll into comp, and we finally measure the thing the data says matters most.”

Nora had two. “The word comes out of every firm document, every form, every template. Badge requests, engagement letters, the works. Forty percent of this firm has been named for what it isn’t, in our own paper, for a hundred years. That ends this quarter; the systems team already scoped it. And the survey runs annually, same questions, wording frozen, so next year’s numbers can be compared instead of explained away.”

Cooper brought the third. “A hundred hours a year of billable-equivalent credit, for every associate, for supervised AI fluency work. Building with the tools, testing them, learning where they break. It’s the smallest version of column three that’s still real, and it answers the sentence I heard in every follow-up interview: the saved time was never mine.”

Arthur had let the pen rest while the proposals stacked up, and when he spoke it was to the table generally, in the unhurried voice that made rooms lean in.

“I’ll support all of it. One caution before you congratulate yourselves. A survey you act on once is a gesture. You’ve just voted to ask these questions every year, which means the duty to answer them renews every year, whether the numbers flatter you or shame you. The people who wrote those comments extended us a credit line of candor. Renewals depend entirely on how this table spends it.”

The quiet part

Oakes had been listening for most of an hour with his chair pushed back from the table, and he came forward now.

“I’ll take the decisions to the partnership tomorrow with my name on them. But I want one thing said in this room first, plainly, so we all know what we’re actually voting on.” He rested his hand flat on Ware’s three sheets. “Every efficiency this firm ever found, we kept. Not once, in my thirty years holding a vote in this partnership, did anyone decide that. It happened the way Cooper described, by default, in the plumbing, while we congratulated ourselves on the dashboards. What changes tomorrow is that keeping the hour or returning it finally becomes a decision, on the record, with names attached. Some of my partners are going to hate that. The annuity sentence in Maya’s deck explains exactly why.

“Two more things go with me in the morning. The attendance policy goes back to a vote, secret ballot this time. The survey tells me our last vote measured what everyone assumed the room believed, and I’d rather govern by what the room actually believes. If the policy survives a ballot nobody can see, it will finally deserve its enforcers.

“And two items enter the record marked open, because I won’t dress them up as solved. The non-equity question. A twelfth-year told Cooper he could live with the fraction if anyone could tell him what it measures, and nobody at this table can, including me. And the parity table, the leave weeks, the Friday email. None of that gets fixed by Thursday. Next year’s survey asks about both, and this table answers for whatever changed in between.”

“And the results themselves?” Nora asked. “How much of the report goes beyond the partnership?”

“All of it. Firmwide, same day, anonymity floor and nothing else held back. The minus 27, the two peaks, the plus 9, the annuity sentence.” He glanced at Arthur. “Your conditions hold. Everything else publishes. I told Cooper at the start I wasn’t commissioning a mirror to hang facing the wall, and I’ve since learned an entire floor of my firm assumed nobody would ever read what they wrote. Publishing is how we tell them we read it.”

“And when somebody asks who approved airing all of this,” Maya said, in the tone of a lawyer who already had the answer drafted, “what would you like us to say?”

“We did.” Oakes stood, which was how his meetings ended. “Say it exactly like that.”

On his way out of the building that evening, Cooper stopped at a fourth-year’s doorway long enough to keep a promise, leaning in while she packed her bag.

“You asked where your six hours went. The answer is nowhere. The system kept them by default, and that answer turned out to matter more than the hours did.” He knocked once on the doorframe, the way visitors do when the visit is good news. “Starting in January, a hundred of them a year come back with your name on them. It’s a start.”

Tessa considered that for a moment. “You know what’s strange? That’s the first time anyone here ever finished a loop I started. Maybe put that in next year’s survey.”

The morning after

Maya took the partner readout at nine and delivered it the way she had promised, transcript-style, nothing rounded toward kind. Cooper watched from a chair along the wall as she put the plus 9 beside the minus 27, walked the room through the believer count, and read the annuity sentence aloud, twice, slowly, before doing the most effective thing a litigator can do to a room, which is nothing at all. Nobody claimed the sentence. A great many partners found the surface of the table fascinating. The three decisions passed on voice votes that sounded mostly like relief, the two open items entered the record with their labels intact, and the attendance ballot was set for the following week. Rawlings seconded the ballot motion loudly enough to turn heads, and seemed to enjoy the turning.

The report went to the whole firm at noon, three numbers in the first line, nothing held back past Arthur’s floor. Cooper spent the afternoon watching the building absorb it, which mostly looked like people reading in doorways. At four, a one-line message arrived from Grant Ellery: Twelve years here, and this is the first February I’m actually curious about.

That night, at the desk under the lamp, he opened the notebook to the grid. Three rows of survey data. The fourth row had stayed empty for six weeks, seven people being too few for Arthur’s floor, and Cooper filled it now from the only evidence the rule allowed: a table that priced its own defaults out loud, published the unflattering mirror, and put names on a decision the firm had been making namelessly for years. He wrote one word across the whole row. Reshaped. Then, because the evidence deserved the caveat, he added a small question mark, to be resolved annually.

The grid was finished. Four cohorts, five kinds of weather, one firm. He didn’t pin it anywhere. Some records belong where the thinking happens.

Under it, the last entry of the arc:

Nobody ever decided to keep the saved hour. It stayed by default. Returning it is the first decision anyone actually made about it.

In this episode of The Geek in Review, Greg Lambert hosts a solo conversation with Triona Buckley, Chief Product Officer at Actionstep, about generative AI’s growing influence on mid-market law firms. Buckley challenges a common assumption about legal AI: faster task completion does not always remove friction. An associate might produce a draft within seconds, only to transfer the burden upstream to a senior lawyer responsible for reviewing sources, reconstructing reasoning, and correcting mistakes.

Buckley argues law firms should shift their attention from speed to systems. Standalone drafting and research tools address individual tasks, while system-level AI connects work across an entire legal matter. Embedded within everyday workflows, AI helps lawyers locate information, reduce administrative work, and preserve more time for client advice and professional judgment. The goal is a smoother operating model, rather than a collection of isolated tools producing faster documents.

The conversation also examines institutional knowledge, especially within firms lacking large knowledge management or innovation teams. Buckley describes an approach where AI captures decisions, context, and reasoning as lawyers work. This creates a continuously expanding record of how the firm handles matters, advises clients, and applies professional judgment. Governance still plays a central role, including clear audit trails showing whether a person or an AI agent performed each action.

Greg and Triona then explore AI as an individual tutor for junior lawyers. Remote and hybrid work have weakened the traditional apprenticeship model built around observation and informal office conversations. Drawing upon decades of firm experience, an AI tutor might question an associate’s assumptions, prompt additional research, and reinforce the firm’s preferred methods. Such systems offer structured practice while preserving the essential mentoring relationship between senior and junior lawyers.

Another major theme is the hidden cost of delayed time entry. Actionstep’s Trace passive time capture technology monitors work across practice management, email, and document applications, then presents lawyers with matter-linked, billing-ready entries. More accurate records help firms recover otherwise forgotten time while producing better data for pricing, staffing, client estimates, and profitability analysis. Those insights grow more important as clients push firms toward fixed fees and output-based pricing.

Buckley believes mid-market law firms hold several advantages during the AI transition. They often operate with fewer systems, maintain closer client relationships, and move through organizational change faster than larger enterprises. Success will still require disciplined implementation, trusted internal champions, connected data, and sustained attention to client service. Her message is optimistic but direct: firms with strong relationships, clean data, and a clear economic strategy will be better prepared for agentic AI and the changing business of law.

Actionstep’s U.S. Midsize Law Firm Priorities Report

Listen on mobile platforms: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Apple Podcasts⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ | ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Spotify⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ | ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠YouTube⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ | ⁠Substack⁠

[Special Thanks to ⁠⁠Legal Technology Hub⁠⁠ for their sponsoring this episode.]

Email: geekinreviewpodcast@gmail.com

Music: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Jerry David DeCicca⁠⁠⁠⁠⁠⁠⁠⁠⁠

Transcript:

Stephanie Wilkins (00:00)
The Gen AI conversation has been advancing faster than a lot of people can keep up lately, but some new developments have brought older concepts like prompting back into the spotlight thanks to a trending new topic, token cost. Token cost moved into the spotlight recently as tools like Claude gained traction in legal, because most plans come with token limits, as well as the ability to request limit increases, which has resulted in tales of astronomical bills for some users. Recently, Legora also announced that it’s moving its Agent Pro offering

Consumption based pricing, which means it’s billing by what the agent does rather than by a flat seat license. And what agents do is consume tokens. Eventually, other providers are sure to follow suit. What many don’t fully understand is just how quickly token usage can add up. A few extra follow-up questions, a document pasted in twice, a chat continuing long after it should have been reset. If that sounds familiar, token consumption compounds faster than you might expect, and you may be looking at a higher token usage than you think.

And you might not even realize it until you’ve hit your usage limit or worse, seen the bill. This is a blind spot we’ve been unpacking in one of our latest article series on Legal Tech Hub: how to get more out of tools like Claude without burning time, decreasing accuracy, or racking up unnecessary bills. We’ve covered topics like what tokens actually are and why they function as a hidden meter running behind every chat. When to reset a conversation versus continue in the same chat.

And what everyday prompting habits, from repasting whole documents to burying five questions in one prompt, might be driving up both cost and inaccuracy without you knowing it? Head over to legaltechnologyhub.com to read the full series and learn more about how to get the most out of your token limits and your usage of tools like Claude.

Greg Lambert (01:45)
Welcome to the Geek and Review, the podcast focused on innovative and creative ideas in the legal industry. I’m Greg Lambert, and Marlene is out west climbing a mountain or maybe enjoying a cocktail somewhere right now. So everyone is stuck with me this week. today we are exploring why AI isn’t actually eliminating bottlenecks in law firms, but rather just moving that bottleneck upstream and how mid-market firms

Can leverage their institutional knowledge to win the AI transition. So I’d like to welcome Triona Triona Buckley, the chief product officer at Action Step, with more than two decades in the legal and legal tech industries. Triona has spent years working inside law firms here in the US, in Ireland, and the UK, and now leads the product vision at Action Step.

the leading cloud-based law firm management platform for mid-sized firms supporting over fifty five hundred law firms globally. And Triona is joining us I think from the future because I think it’s already tomorrow there in beautiful Auckland, New Zealand. So Triona, welcome to the Geek and Review.

Triona Buckley (02:55)
Thanks, Greg. Great to be here.

Greg Lambert (02:58)
All right, I I threw this one in at the last minute ‘cause I realized we weren’t giving you a chance to really talk about action steps. So you mind giving us just kind of the the elevator pitch on on action step and what you do there?

Triona Buckley (03:11)
Sure.

Action Step is a law firm’s operations platform. So it combines practice management and legal accounting in one platform. So really, I suppose, designed to connect all the work of everyone that works at law firms. We are a global business, so we have a global team, a global customer base, very much focused on the mid market. So that’s our sweet spot, it’s where most of our customers sit, and so we’re really trying to support the unique challenges and our

opportunities that mid-market firms have. Yeah, and I’ve I’m CPO at Action Step, I’ve been with Action Step for many years and work very closely with our customers on a day-to-day basis to make sure that our product is is answering their needs.

Greg Lambert (03:55)
Awesome. Thank thank you very much for taking the time to give us a little background on that.

Greg Lambert (03:58)
So Trina, your your core argument regarding generative AI isn’t that it’s necessarily eliminating the friction in law firms, but it’s it’s kind of taking that friction and and shifting that bottleneck upstream. So, you know, for example, if if a junior associate you know takes a few seconds to draft a document, now all of a sudden the the senior partner is is having to be inundated with these and then kind of

reverse engineer how how it was put together. So in your view, how do you how how do firms structurally fix this workflow so that that AI doesn’t actually in introduce more problems than it fixes and and what do you mean when you say something like we need to shift from speed to systems?

Triona Buckley (04:50)
So what I mean about really shifting from speed to systems is is thinking less about, you know

one task, you know, and delivering one task using one AI tool. And more about, you know, thinking about the whole workflow. You know, how how are you going to use AI across the workflow and across your tasks in a way that feels much more intuitive to the way that you naturally work. So, you know, what we’ve seen is that law firms adopted and continue to use some fantastic tools around documents and research and and those sorts of things. And those are great to

Tools and obviously they came to market first because it was around generative AI and that sort of thing. you know, we now have the opportunity with more agentic AI available to to really connect that into a more natural workflow and into the places that you work anyway. so really it’s about taking, you know, the like I say, individual tasks and connecting them together into the the overall workflow. And I think the way the way that law firms can do that and and think can have apply that system thinking

is really think about the broader and bottlenecks. You know, we started with drafting, but yes, yeah, we’re we’re we’re it shifts it shifts the onus, I suppose, upstream to make sure that those drafts were well done and researched and thoughtful and blah blah blah and you know and and and and cited correctly.

And so those are the more expensive resources being used to do some of some of that work. But really, you know, oftentimes the friction in overall workflows is less to do with those tasks and more to do with, you know, sort of operational friction, administrative work, you know, finding information, not knowing where to look to find the information, you know, and having to dig into all these different silos of information to find what you need. So really addressing those sorts of things, you know, help.

the sort of day-to-day of your fee earners just feel smoother and less fewer things in their way. So that’s really what I you know what I think about when I think about kind of system level AI. It’s really smoothing out the whole the whole the whole workflow so that fee earners get to do what they do best which is really apply judgment to their client work and they get to do it in a way that you know they have room to think etc because that you know administrative friction has been removed I suppose by a

AI, automation, systems, whatever it might be.

Greg Lambert (07:05)
Okay. So w we talked recently, I th I think it was the last episode on

called executable knowledge and and private graphs. And you’ve noted that while AI can draft a contract, you know, AI sometimes is is biased and you know what clients are really paying for during the the high stakes milestones that is is for that lawyer’s kind of unbiased judgment. There and we we talk a lot about judgment here for the for the lawyers. So

On the the mid-market firms that that you work with, how is it that they’re actually capturing the expertise that’s kind of locked in that that partner’s head and put it into so that it’s it’s that systems process that kind of guarantees things like

The context is right, the controls are there, the governance is there to to help support that judgment. How do you how do you pull it all together?

Triona Buckley (08:05)
Yeah, well

Interestingly, what we see at larger law firms is that, you know, they tend to have huge knowledge teams and innovation teams who are sort of their job to a certain extent is to, you know, collate all of that knowledge and bring it together and and extract that sort of thing from people’s heads. I mean, mid-market firms don’t have the luxury usually of of having those sorts of resources. And so if it’s not been captured throughout your day, it’s probably not going to happen. It’s probably not going to be, you know, a a concerted concerted effort, I suppose, for you know, everyone at the firm.

To sort of download what they’ve done that day and the reasons why they might have done things, etc. And so what we what we can see with with and more so with AI, but even with you know automation is if you’re if you’re capturing if you’re capturing you know what people are doing as they move through their day and you know prompting the right questions, asking them why they might have done things just as they as they actually go about their their normal work, it’s much more likely that you’re going to be able to you know to add to that instant.

institutional knowledge and really add that context of that layer of context around what they’re doing and why and so what from an action stamp point of view what we’re really trying to ensure in anything that we bring to market from an AI point of view is that it’s not it’s not interruptive it’s very much you know really just kind of following along as they move through their day and and sort of you know capturing everything. What’s really important there is there’s a governance layer to that obviously you know you have to know who has done what whether that’s an agent or human.

And so, you know, making sure that you’re actually, you know, capturing that that that level of detail on on on what’s been done and when and why and all of that is really, really important from a governance point of view, because you have to be able to look back and see and we all we all we all we all we’ve all seen you know the reasons why if you don’t have that it can be a problem.

But actually there’s also just a you know like it beyond the sort of governance and the safety point of view, it’s really more around you know adding to your institutional knowledge, you know. and I suppose you you want to, you know, if we’re you know, the things that might have been learnt, I suppose, in conversation, you know, need to be also captured within within the system. And so that’s a AI is a great, great use case for that, you know, because it can act like you’re you’re sort of your your sparring partner in understanding where you’re coming from.

Greg Lambert (10:17)
Yeah. I’m I’m curious on how you kind of set that up and monitor it with because I n I know you mentioned that you wanna you wanna do this in a way that doesn’t interrupt the workflow of of the attorneys. So, you know, just kind of peeling back the onion a l a little bit. How how how did you work to set something like that up and and kind of determine what’s w how how it is that you don’t interrupt the workflow?

Triona Buckley (10:47)
Yeah, look, I think a lot of it is again back to sort of, you know, having most things happening in one platform, you know, so that you’re you’re you’re not having to sort of, you know, change how you do things in different places and therefore piece together all of that kind of audit trail. So having that sort of centralized place for all of that information to live is is is thing number one. And so and we see the most mid-sized law firms tend to prefer to have fewer tools rather than more. Some of that is you know budgetary point of view.

of it is just there you know a preference for a firm that size that what that allows actually is that you can have you know with all of that connected data you have you can put like ai on top of that and it it has all of the context you know and so it it doesn’t have to read five different tools you know it actually has the information it needs at its fingertips and so that that’s a big piece of it is just the fact that it’s actually already kind of connected and when in one place it doesn’t have to be one platform

A lot of it is just more to do with how things are interconnected. So that that’s sort of the the underlying, I think, philosophy really for us around how we’ve been thinking about AI and how we see AI being successfully adopted by our customers is again back to that kind of system thinking. You know, how do I how do I make sure that everything I do is is is being captured, but it’s it’s not sort of you know proliferated across too many tools.

Greg Lambert (12:08)
Yeah. Well let me get away from the from the the partner level and let me let me ask you about some of the training, especially on the the associates. and I and I think you know I I was reading articles today that that kind of played on this fear that you know that the AI is is going to just cut all of the junior associate class, you know, compound that with the fact that

a lot of clients don’t wanna pay for that. They see it as as training. I always laugh. I I love the fact that the clients love taking the the experienced lawyer later and bringing them in house, but they don’t wanna train them them

Triona Buckley (12:47)
Yeah.

Greg Lambert (12:52)
As a junior associates, but you you mentioned that we need to move away from relying on training by osmosis, and I think a lot of us understand that. what you mean by that. So, how should firms re-architect their training module or models to so that AI acts more as an accelerator?

that teaches critical thinking rather than just a machine that that just does some some work for them. I was I I like to call this using the AI to help you AI. but how how do you see the AI being used in the training sessions?

Triona Buckley (13:30)
Yeah, I mean, you know, one of the things that I saw at law firms when I worked there was this, you know, model of, you know, summer associates and more junior associates like in office with partners or in office with senior associates. And like there’s a lot of value in that osmo like training by osmosis, as you said. some of the sort of more nuanced learning happens there. But you know, the the reality is that, you know, s some of that has already gone away, you know, through COVID and everything else, and people not necessarily being

Being

co-located. you know, that hasn’t been as possible in the last number of years. And while firms have you know really invested in their training programs and their more formalized sort of structured training systems, I one of the things that they hasn’t quite caught up yet is this yeah, is using AI to be the trainer, you know. So if we think about connecting what we talked about a moment ago in terms of institutional knowledge, there’s decades and decades of you know client work and you

know, you know, advisory work captured, you know, within the systems of a law firm, how do you put that to work? Not just for your client work, but actually for training your, your, your juniors. And so, you know, and it goes back a little bit to what we spoke about as well in relation to creating drafts. you know, if you actually have AI that is acting like the trainer, is asking your associates questions, is using the institutional knowledge to prompt them to ask deeper questions, to understand this.

More to make sure they’re checking what they should be checking. You know, it it really is an opportunity, I suppose, for it to feel like that sort of in-office, you know, more nuanced sort of training without that lift being on, you know, the partners to do that. Because, you know, as we all know, that’s a very inconsistent model. It’s very much reliant on, you know, how good that one, you know, partner or senior person is in terms of training. Whereas AI can really standardize that, you know, it can take all of the good parts of all of that sort of learning biosmosis, but add in a level of structure.

And and sort of rigor that that wasn’t there before us. So I’ve just you know we’ve I’ve started to see a few of the more innovative firms really see that as an opportunity, particularly at the mid market, where again you don’t necessarily have the resources to have a dedicated you know training program and that sort of thing. so so yeah, that that’s really the opportunity that I see is just sort of you know adding adding the nuance and the rigor, but really the institutional knowledge comes to life. This is how we do things with this firm, but not even just at a sort of a

you know, a workflow level more of the sort of like, you know, we tend to advise this way with our clients, you know, and like really, really very much more I suppose nuanced, you know, which is really the secret source I suppose of why clients choose a particular firm over another. Yeah.

Greg Lambert (16:03)
Yeah.

I’ve I’ve been using and and I know I know that Mark Andreessen didn’t come up with this, but he had he had said something earlier earlier this year about the number one thing that moves someone from the fiftieth percentile to the ninety ninth percentile the fastest is individual tutoring. and he said over you know the

long history of of humans that you know the the royal families did this, that the the the wealthy families did it, but it you know, the it took a lot of money to to do to do this individual tutoring and he’s saying now we have these tools that can be set up to be tutors. Is that is that how you’re seeing it is to train the train the AI to be the trainer or the tutor?

Triona Buckley (16:52)
Absolutely.

One hundred percent, yes. And and again, this isn’t to replace, you know, training programs or to replace that really important, you know, senior junior interaction. It’s about augmenting it and just making sure you’re standardizing it to a certain extent so everyone’s getting what they need. But I do really think that it’ll ex that it accelerates you know, the the pathway for for junior lawyers to actually get to a level of knowledge and and and and that more quickly and consistently, like I say. So yes, absolutely. It’s like using AI as your tutor.

not exclusively augmenting your training programs and and all of that. But but yeah, I think it’s a huge opportunity. And I think for mid-market firms in particular, a way that they can really use AI to help give their teams an edge, you know, and compete at whatever other level they they want to compete at with larger firms, whatever it might be.

Greg Lambert (17:39)
Yeah, yeah. I I I know a lot of trainers that that would love to have the help on this because they’re overwhelmed. In fact my my trainer was was telling me ‘cause we were trying trying to think about how we were setting up the training program.

And she was it was kinda tongue in cheek and she was like, Well I’m you know, I’ve already got a full time job doing training and now you’re now we’re bringing in AI, so I’ve gotta do all that what I was doing and now I’ve gotta do AI too. So I think any type of leverage that we can get to to help the AI help us would be appreciated.

Triona Buckley (18:12)
Yes. Well that’s it. And then that’s

another that’s another bottleneck, I suppose, that we’re we’re seeing at firms is that yes, everybody has their day job, you know, and everybody is looking at how do they how do how do they sort of have this AI dividend brought into the mix. You know, how do we make AI actually create some ROI for us and make things more efficient? And in the meantime we’re double jobbing trying to figure out how to do privacy.

Greg Lambert (18:33)
Yeah. I was gonna say I’m still waiting on

that dividend.

Triona Buckley (18:36)
Yeah. Poor

poor IT teams trying to figure out, you know, what tools and when and for who and et cetera. Yeah, it’s it’s tricky.

Greg Lambert (18:44)
I I know when I interview attorneys that one if I ask them what are their top what are what are the top tasks that they do during a day that they would love to be automated.

typically those two are email and time entry. and so I wanna want to dive in on the the time entry, which you know there’s a lot of what was called the data debt that that is involved in the manual time entry tracking. and

Recently, Action Step acquired a startup called Tr Traced and brought in its founder, Aiden Bub, and integrated a new passive time capture module called Trace. so you’ve mentioned that when lawyers wait to enter time, and I think a lot of us know this, that you know, there’s that recency bias which causes them to you know write off time even before they’ve written the time down. so why

Why

is it so critical for you to for lawyers to capture time in the present tense as as as quick I guess as quickly as possible? And and how do you how do you use this AI module to help you kind of decouple the the time from the from the strictly hourly rates?

Triona Buckley (20:02)
Yeah, there’s there’s two aspects to it. Yeah, number one, I think, yeah, the the the panic, you know, at the end of the the week or the month where you get your push from rebelling team to to enter all your time. my goodness, what did I do? There’s that. And obviously, you know, most most fear earners have figured out systems and, you know, ways of of of helping with that in in you know, whether it’s their own individual kind of systems. and then the other piece is that yeah, you just you know, you you you automatically as a human write down the value of what you’re doing sometimes, you know, and so

Greg Lambert (20:12)
Yeah.

Triona Buckley (20:31)
you you just you you you think it’s fifteen minutes and it was actually an hour or whatever it might be. You know, you’re losing minutes all of the time and so

Frankly, sometimes it takes more time to enter the time of what you did, you know, and trying to recall what you did, it’s not almost not worth your time to create the time entry. And so there’s you know, there’s a few different aspects to it there, but what we know. So, but what we’ve seen is that actually again, if you’re as you work through your day, you’re getting regular you know reminders or prompts, or you know, you know, almost you know, bill ready you know, time entries presented to you, then all you have to do is go, Yes, I did that tick. you know, we’re we’re seeing huge.

Huge gains for our fee owners who are using Trace in Action Step. And you know, there if if you’re even capturing an extra, you know, 30 minutes a day, which is work you already did. We’re not telling you to do any more work, we’re just saying capture more of that work so that it’s because it’s billable work. you know, if you’re even just capturing 30 minutes more a day, that’s obviously a huge gain that all adds up every week, etc. So so that’s really what Trace is all about. It’s just in you know, it monitors both the work that you’re

you’re

doing in action step in our application, but also the work that you’re doing in in Word and email and et cetera, et cetera. And so it’s and and really presenting it all in a way that is, you know, bill already very easy for you to say yes. It’s obviously connect, you know, it connects it connects the work through to the matters using AI and using the context of of where you’re working. So

Really exciting one. I think there’s a huge ROI gain for law firms on that. You know, it’s sort of the most obvious one our customers have really gravitated to. so yeah, excited to get it into. We have we we already have our first version of that in market and being used by customers. We have our our next version actually coming out next next month, which crosss across more tools. so yeah, really, really good ROI for our customers on that.

Greg Lambert (22:15)
Well, I imagine even beyond just the, you know, tracking time, having a more accurate assessment of how long it actually takes to do something. you know, having those data points and and being consistent, rather than relying on somebody trying to remember at the end of the month who’s there’s you know is it’s gotta be a huge value add too.

Triona Buckley (22:35)
That’s it.

Yeah, down downstream the huge value adds in terms of yeah, like you know, accuracy of of client quoting, for example, you know, and and and on and that whole area, you know, and so there’s there’s you know, so pricing, you know, how you think about those things, you know, just just you know utilization across teams, you know, there’s so many different things that this plays into. and really lifting that information up to management so that they can see, you know, where people are spending them the the the most time and how can we help. Again, it helps to identify some of those bottlenecks that you

Mm-hmm.

Greg Lambert (23:07)
Yeah, yeah. I c I can I can imagine the the partner, the billing partner looking over there and going, my god, it took that long to do this. So one

Triona Buckley (23:14)
Ha ha ha.

Greg Lambert (23:18)
We talked briefly about the fact that the, you know, there’s a lot of tools that are being used. and it’s interesting because Action Step actually did a mid-market priority report, and it was just kind of showing the amount of tools they get used. and I think it was like 83% of firms still use like three or more tools just to manage a particular matter. and

a a third of of those use six or more tools. So and and being in a in a large firm, I I mean we’ve s we’ve seen tools sprawl where it’s like, it’d be you know it’d be great if we added this one thing or this will make make things easier. And the next thing you know you you’ve just got this huge stack of of technology with each one doing its own little piece. So

Triona Buckley (23:48)
Yeah.

Mm-hmm.

Greg Lambert (24:07)
Why why do you believe that mid-market firms are actually better positioned to to apply the the AI transition than say an an AMLOD, you know, 100, 200 firm? and and so, you know, how how are you seeing them being better positioned?

Triona Buckley (24:27)
I think there are three reasons why mid market firms are are better positioned than most. number one, they they tend to have fewer tools, so they’re probably more in that that middle camp of you know, kind of three ish tools, maybe that they’re they’re working with, which actually isn’t a lot, you know, being having some well integrated tools is is good. and so they have fewer tools to to work with, and so it’s actually it’s easier for them to sort of get their their their house in order and get sort of AI.

ready, you know, because their data’s living in fewer tools, frankly. And part of that is because you know that they don’t necessarily have the budget to buy all of these other two other tools. And part of it is that they’re, you know, they’re they’re able to you know the they they there are products out there that support them, you know, that that give them that sort of you know cleaner, cleaner environment. So there’s that piece of it. I think you know number one, just that their their tooling is set up well for you know for for AI to work well for them. Number two

It is just a function of size, you know. the you know, you have a professional layer in mid-market firms who are experienced, who typically are quite close to the bottlenecks and the needs of that firm. but because they’re slightly, you know, they’re not these huge big enterprise firms, they’re they’re easier ships to turn. And so change management is something that can happen somewhat more easily if you do have that professional layer. And I’m really talking about firms who sort of invested in having, you know, you know, strong management resources that can help we can help that to happen.

And then the third thing is that, you know, generally clients choose mid-market firms because they have, you know, they they the closest they get to senior, senior resources, you know, the closest they can have to the partners, the advisory relationship that they have there, which the you know, you don’t necessarily get, you know, in other segments of the market. And so mid-market firms already have, you know, good relationship managers, you know, so their fear earners are good relationship managers, their partners are very focused on the growth of the firm and making sure that custom you know, clients are are well.

taken care of and all of that. And that that’s one of the things that is going to shine most in a world of AI. You know, it’ll be less about the sort of the the quality of the of the you know the documents and that sort of thing and more about you know those sort of really nuanced personal you know advisory relationships you know which mid market firms have already had to hone those skills you know over many years. And it’s one of the reasons why you know

talent comes to those firms because they want to have more of that client engagement, you know. So so those are the sort of the three reasons why I think mid-market firms are in really, really good position when it comes to AI. They can their data’s well set up, they’ve got the client relationships already, which are gonna be their value point going forward. and they, you know, they’re they’re easy they find things, you know, easier to adopt and easier to change.

Greg Lambert (27:03)
Yeah, I wanna I want to pull on the the the change management thread that you mentioned there in the the middle of that. and I imagine that having fewer tools or tool you know tools that that kind of crosses across multiple steps.

makes it a little easier on the change management front. how how do you work with your with your c customers to kind of understand that change management aspect of it? ‘Cause I I think a lot of us don’t realize how important that is.

Triona Buckley (27:35)
You are absolutely right. So many firms do not realise how important that is. and we try and advise firms who come to us to really think about these projects as, you know, think about them in a matter of sort of, you know, months, not weeks. You know, like these are th you know, you have to plan for this. You have to make sure that you have you know, you’re you have an awareness, I suppose, of how your systems currently work, your actual, you know, how your people work. you know, and that you’re you’re planning around that and like less focus on feature functionality.

And more about you know your own processes today and and and how you want it to work in the future, that sort of thing. So, I mean, a lot of the change management piece is around, as you know, yeah, you know, being clear about what actually needs to change, where those bottlenecks are, but also having the right team in place that are going to actually drive it forward, they’re going to be champions, who are going to make sure it’s successful. So, a lot of the times when we’re engaging with you know firms who are coming to us for the first time, we’re asking them those kinds of questions, you know, like who’s who’s going to be involved in this in this process.

project. you know, what are the things that matter to your firm? You know, where do you see your firm, you know, next year, the year after? What sort of growth are you experiencing? What sort of clients you do with so really more like asking about their business so that they’re already starting to think about those sorts of things in the context of technology and less sort of you know comparison tables of features and functionality, like I say. So that’s really important. We also work with a really really experienced cohort of advisory partners who you

Know work with firms to help them to adopt new technology, like their entire jobs are around change management and technology and adoption and that sort of thing. So you know, while our team is very much because most of them have worked in law firms, they’re very much of that mindset. the actual implementation work is is is is done by our our partner group, you know, in direct collaboration with our customers. so that works really well. It means we don’t have any implementation bottlenecks at Action Step. You know, we have we have kind of an

Endless supply of you know of of of well-vetted partners that customers can choose from. So that works that works well. And it’s a very different, like we’re a software company, you know, and so it’s a very different mindset. You need to have business process people involved in that. but like I say, I think having those conversations from the very start, we know is really important. We have a lot of firms who will come to us, have those initial conversations, realize they need to go and do a bit more homework, and then come back to us, you know, kind of three months later. So yeah.

Greg Lambert (29:52)
Yeah. Yeah.

Triona Buckley (29:56)
That’s that’s probably how it works best.

Greg Lambert (29:56)
Yeah.

Yeah, the you know, kinda kinda keeping your eye on where you know the the s strategic focus of of the firm. because you know especially when you’re bringing in a a a substantial bit of software that that can kind of change the day in and day out for people that the you know it’s not just turn it on and say here you go. There’s there’s a lot that you need to think about as you’re as you’re

So it sounds like you’ve got a good handle on on that implementation. Well done.

Triona Buckley (30:27)
I like to think so. I mean look, these things are always they’re they’re huge projects, you know, they are huge projects, like you say. You know, it’s it is it’s a big undertaking. and I suppose for that expectation to be set is also important, you know, like let’s be let’s be clear, you know, this is going to take, you know, a lot of effort, a lot of thinking, but you’re gonna come out the side in a really, really good position, you know, and sort of you know, working through that th those, you know, th those periods of of of of pain. you know, it’s it’s an isn’t

Inevitable, you know, but but again, as long as as long as you’re having those conversations early and that’s understood, it’s not an issue. It just shouldn’t be a surprise.

Greg Lambert (31:04)
Yeah.

well, you know, the one thing law firms are known for is is really accepting change. So it’s or or or maybe not. Maybe maybe we’re not known for that.

Triona Buckley (31:13)
People.

But

you know, there’s a bit of a talk track about law firms sort of being, you know, like you know, slow to change and and and you know that sort of thing slow to adopt. but I actually I I respect the thoughtfulness that you know generally the legal industry has to, you know, to to to these things. I also think if you look at the history of law firms, like to be fair, there’s there’s few enough industries that have actually survived and thrived as well as as as law firms have, you know, so through all sorts of booms and busts and whatever else. So I don’t

Yeah.

Greg Lambert (31:43)
Yeah, yeah. Well, one thing we do thrive through uncertainty. So that’s so well, Triona, let me let me ask you before we get to our crystal ball question, we’ve been asking our guests for the past year or so, just because there’s so much going on in in in the industry, whether whether it’s legal tech or just AI in general, you know, there’s just

Triona Buckley (31:47)
Ha ha ha.

Greg Lambert (32:09)
So much to try and keep up with. So we’ve been asking, you know, what are what are some, you know, one or two of the you know, must read or must watch or must listen to resources that you l use to kind of help you kind of keep up with with what’s going on in in the market right now.

Triona Buckley (32:30)
I have to admit that while I am a obviously this podcast, but while I am a I I am I while I I listen to most things, I think the voices that I listen to the most are are are not necessarily the you know the podcasts or the articles or that sort of thing. It’s it’s actually, you know, folks are in the industry working with law firms every day. I mean obviously yourself and Marlene are in practice, so that’s you know different. You have that very specific

Greg Lambert (32:35)
Obviously.

Triona Buckley (32:57)
perspective but but yeah you know that I mentioned that partner community and so I would say actually that partner community I probably listen to the most and obviously directly to customers and really try and form our own view based on based on them because one of the things that I I see happening in the industry is a lot of me too me too, you know, just a lot of kind of jumping on bandwagons. And so actually kind of going directly to the horse you know, and and and sort of hearing what they have to say is is

it’s in my mind more powerful. so yeah that tends to be where I where I lean for sort for some real market reads. Like these are partners who are working with you know dozens of dozens of customers, law firms, you know, every single every single month you know who who really do understand things from the inside out. So that’s that’s where I lean.

Greg Lambert (33:43)
Yeah, it’s it’s definitely still important to to build your own community. So it’s I I think I think a lot of us have relied upon that, especially the last two or three years. So well now it’s it’s time for a crystal ball question. So pull out your crystal ball. It’s really kind of cheating because you’re already in tomorrow, so you you you probably already know some of the future.

Triona Buckley (33:48)
Yes, most important.

Ha ha ha

Greg Lambert (34:07)
But look looking ahead you know a a few months to a few years, as you know, a agentic AI and harnesses and you know, all of the stuff that that we’re being hit with lately affects and becomes embedded into the business of law. You know, what what are some things that you think especially in the mid market law firm model?

that they should be prepared for, that that that they may not be ready for yet.

Triona Buckley (34:37)
a few things. I mean, I think I mean some of the things we’ve touched on. So I think you’re gonna see firms that you know have invested in you know client nurturing, client relationships, that sort of thing. I think they are gonna be the ones who who thrive. And so if firms are not doing that already, that should be the thing that they are prioritizing over and above anything else, frankly. you know, the second thing I mentioned, I think the firms that have their their data house in order are gonna do better, you know, in in an in terms of AI enablement. but in general, I think I

have a very positive perspective on where law firms will end up being. I think again, you know, law firms have been through a lot of cycles. They always end up you know surviving, thriving, you know, changing, evolving, you know, coming out stronger forward, sort of thing. And so I so you know I think I I think just for them to think forward to that point, you know, and what that looks like for them and and and work towards that. The the pieces that I don’t see as much conversation around at at mid-sized firms, I think is probably happening

With enterprise firms, but not necessarily mid-size, is around how their commercial model is going to change, you know, how their whole their pricing, their

you know, like the types of clients’ engagement and client engagements that they have is going to change. We’re already hearing from some law firms that clients are saying to them, you know, I don’t wanna I don’t want to pay for your juniors, I only want to pay for your seniors, that sort of thing. So I think, you know, you’re just gonna see a lot more which is has already been happening, but you’re gonna see a lot more, you know, project level pricing, you know, those sorts of fixed fee type of pricing, which is less less about attaching, you know, rates to people and more about the output. you know, so I

And I I I think that’s an area that I think mid-sized law firms really need to need to focus on and plan for and understand how that’s going to impact their their kind of economic model as firms. it’s one of the reasons why, you know, we we talked about time tracking earlier, but you know, you know, and sort of understanding the dynamics of you know, where you spend your time and how that plays into profitability. it’s one of the reasons why I think that’s a that’s like something that we’re investing in and that we can help law firms with to plan for in future.

Greg Lambert (36:35)
Yeah, yeah, great, great answers. So Triona Buckley, think I wanna thank you very much for taking the time to talk with me today and sharing your your vision of where where we’re heading. It’s this has been fun.

Triona Buckley (36:49)
Thanks, Greg. Really appreciate it.

Greg Lambert (36:51)
All right. And thanks to all of you who are listening to the Geek and Review. If you enjoyed the show, please share it with a colleague. we’d love to hear from you on LinkedIn and Substack. And Triona, where’s the best place for our listeners to learn more about you and Trace and Action Step?

Triona Buckley (37:09)
Actionstep dot com. Very simple. Yes.

Greg Lambert (37:11)
Easy. I like it. And

as always, the music you hear is from Jerry David DeSecca. So thank you, everybody.

A standalone chapter of Beyond the Model. The Weather Inside, Part Two of Four.

The second slice of survey results arrived on a Thursday, and Nora refused to email it.

“Come down,” she said on the phone. “This one you need to see standing up.”

The firm was three weeks into taking its own temperature. The instrument Nora and Cooper had adapted from the tech world’s big sentiment survey had already been through the associates, who answered fast and answered plainly, and now it had finished with the business professionals: the accountants, the pricing team, the paralegals, the marketers, the technologists, the KM staff. Four out of every ten people the firm employed, filed by the survey under a single cohort, which is a word nobody on the floor itself would ever use.

They had taken longer to answer than the associates. And they had written. Triple the open-text volume of any other group in the firm, paragraphs where the attorneys had left sentences. People write long, Nora had said when the volume report came in, when they doubt they will get another chance to say it.

She had the identity question up on her wall display when Cooper stepped off the stairs. What has AI done to your sense of who you are in your role? The associates’ answers had come back as a single hill with its weight on unsettled. The chart in front of him had two peaks, one planted on amplified, the other on diminished, and the valley between them ran through the middle of every department on the floor.

“Same tools,” Nora said. “Same rollout. Same training sessions, same lunch-and-learns, same licenses. The attorneys split by seniority. This floor split down the center of individual teams.” She picked up her tablet. “I can’t brief a two-humped chart to the partnership as an average. The average is the one number on this page that describes nobody. Walk it with me.”

The first peak

They started in KM, which had been Cooper’s department once, back when his title said librarian and the firm treated the word as a compliment with a ceiling on it. The department had tripled its floor space since then. The knowledge heat map he remembered from years ago had grown into a bank of displays, and in front of the largest one stood Wes Carver, a KM analyst with his sleeves rolled and a practice group’s precedent library open in four panes.

“Five years ago my job was fixing broken search,” Wes said, when Nora told him what they were walking. “People called me when the document system ate a brief. That was the job. Last month, two practice group leaders asked me to sit in on staffing calls. Staffing calls.” He said it the way other people announced a promotion.

He pulled one of the panes forward to show them what the month had actually looked like. A decay alert, ten days old: the firm’s standard indemnification rider, four hundred uses across six years, quietly stranded by a statutory amendment that took effect in the spring. “The system caught it. Nobody asked it to. It cross-references the precedent library against the legislative feeds and it flagged this one at two in the morning on a Sunday.” He closed the pane. “Then somebody had to figure out which active deals were carrying the old language, and which partners needed the call, and how to say it so the call got returned. The machine reads a million documents. Somebody still has to tell the lawyers what it found, and where its blind spots start. That somebody became me. I marked amplified, and I want it on the record that I meant it.”

“On the record with the anonymous survey,” Nora said.

“You know what I mean. Some of us finally got visible. I’m not giving that back.”

The second peak

Marisol Vega worked two departments over, in litigation support, at a desk with twenty-two years of closing dinners and trial-team photographs pinned to the cubicle wall behind it. She had volunteered for a follow-up conversation the day the sign-ups opened. She talked the way experienced paralegals talk, in complete paragraphs, with the citations ready.

“Let me save you the suspense. I marked diminished, and I’m going to tell you exactly why, because it isn’t the reason the training deck thinks it is.” She turned her chair to face them fully. “I’m faster with these tools than half the associates I support. Privilege logs, exhibit sets, closing binders, the machine assembles and I review, and my error rate beats the old way by a mile. I will outwork any piece of software on the worst day of my life, and I’d bet my kitchen table it knows that.”

“Then why the box you checked?” Cooper asked.

“Because of how this firm says thank you.” She watched him write that down before she went on. “In 2019 this pod was nine people. Today we’re five. Nobody got walked out. The desks just never refilled, one resignation at a time, and the work folded into the machines and into the four of us who stayed. Every time somebody upstairs praises this floor’s efficiency, the praise arrives as a headcount freeze. So when your survey asked what the tools have done to my sense of who I am here, I stared at that question longer than any other on the form, and the honest answer was that my work has grown more valuable every year for five years while everything around the work kept getting smaller. The survey only had one box for that.”

“What’s the part of the job the machine can’t touch?” Cooper asked.

“Teaching. Who do you think shows the fourth-years how a closing actually runs? Where the signature pages really come from, which local counsel answers the phone, what the client’s treasury team needs three days before anybody asks for it.” She nodded toward the trial photographs on the cubicle wall. “Half the associates in those pictures learned it sitting where you’re standing. Nobody automated that part. It just stopped getting counted, right around the time we stopped having the desks.”

She walked them to the elevator when they left, out of an old habit of finishing what she started, and answered one more question on the way. Cooper asked whether she would recommend her career to someone starting out.

“I already did. My niece. Two years ago, and she’s thriving.” Marisol pressed the call button for them. “People come to this floor to stay, Mr. Graham. I picked this firm twice, once the day I took the job and once every year I didn’t leave. You grade a place kinder when the staying was your own idea. Ask your survey if I’m wrong.”

The word

Claire from marketing had booked one of the floor’s interior conference rooms, the windowless kind with the mismatched chairs and a whiteboard that had been erased to gray. She had fifteen years at the firm and a stack of printouts sorted into piles, and she had brought Sarah from IT and Marcus from risk management, all three of them veterans of the governance committee’s long meetings.

“The open-text answers,” Claire said, dealing the piles out like discovery. “Nora asked me to theme them before the report goes up. Most of it you’d predict. Workload. The attendance policy. Then there’s this.”

The top page had a single answer printed on it, and someone, Claire presumably, had circled one word in red.

You named forty percent of the firm after the thing we aren’t.

“Non-lawyer,” Claire said. “It shows up two hundred and fourteen times in the floor’s answers. Unprompted. No question asked about it. I’ve spent fifteen years writing this firm’s bios, and I can tell you we have nine different ways to describe an attorney’s brilliance. For everybody else we’ve got one word, and it starts with non.”

She pulled a second sheet from her own pile. “And here’s your two-humped chart living inside one hallway, since you’re collecting examples. My content team came back amplified nearly across the board. The tools tripled their output and the RFP win data made them famous with the pricing committee. My events coordinator marked diminished. Fourteen years of knowing every client’s name at the door, and the new system auto-builds the invitation lists now. Same manager, same hallway, same tools. Twenty feet apart.”

“It’s on my badge access request form,” Sarah said. “Checkbox. Attorney or non-attorney. I run the systems this place bills through.”

Cooper asked about the friction question, the one that measured whether people felt systemic resistance when they tried to change how attorneys worked. Sarah laughed without much humor in it.

“When the new intake system shipped, this floor had it mastered inside a week, because for us it was mandatory from day one. Three partners still email their matters to an assistant instead. The workaround we built for those three partners has its own maintenance schedule. That’s the honest answer to whose time this firm treats as expensive.”

Marcus slid a thinner pile across. “And the parity data Nora pulled to go with it. Eighteen weeks of paid parental leave for attorneys. Twelve for this floor. Same building, same babies. The four-day attendance policy took effect down here in January, by email, on a Friday afternoon. Upstairs got theirs in March, with a town hall and a question-and-answer session and a transition period.” He shrugged. “Nobody down here is surprised by any of this, Cooper. The surprise is that somebody finally wrote it in a place the partnership has promised to read.”

The window

The administrative floor had exactly one architectural gesture, a tall window at the end of the corridor that looked out over the atrium, and Nora stood at it with her tablet while the afternoon went long. The floor’s recommendation score had just come in from Leo’s tabulation. Zero to ten, would you recommend your career to someone starting out, scored the way the tech survey scored it.

Minus 12.

“Fifteen points kinder than the associates,” Cooper said. “I’ll admit that’s the opposite of what I would have guessed this morning.”

“Marisol answered it for you already. This floor chose the place twice.” Nora kept her eyes on the atrium. “Most of the people down here picked this firm as a destination, and people forgive a home almost anything. It’s the same reason the anger in these answers reads different. The associates wrote like people trapped in a fast car. The answers from this floor are quieter than that. Passengers, mostly. People who quit expecting a vote on the route a long time ago and wrote three paragraphs anyway, on the chance that this time someone was reading.”

She turned around. “So here’s what I’ve decided, and you can tell me if I’m wrong. When this report goes up, every chart in it shows both populations, side by side, on the same page. Attorneys and business professionals. One firm, one axis. No appendix. The two-class system has survived a hundred years of this profession partly because the reporting always kept the classes on separate pages, and I write the reports now.”

“And Rule 5.4 still stands,” Cooper said. “Even if Mike wanted to hand Wes Carver a piece of this firm tomorrow, the profession’s rules say he can’t. That ceiling wasn’t poured by this building.”

“No. But the leave table was. The badge form was. The Friday email was.” She powered off the tablet. “The profession’s ceiling is above my pay grade. The leave table, the badge form, the Friday email, those are ours, and I intend to start there. For what it’s worth, I took the survey too. I marked reshaped. Fifteen years ago I ran what this firm politely called a cost center. These days I run the part of the building where the future keeps showing up first. Put that on a chart.”

Cooper rode the elevator up at the end of the day and, for the first time in all his years in the building, actually counted. Four seconds. One floor. He thought about how far the answers had traveled in that distance, the two peaks, the circled word, Marisol’s twice-chosen career and the nine desks that became five.

At his desk he opened the notebook to the grid and filled in the second row. The business professionals didn’t cluster the way the associates had. Their weight gathered at both ends of the row, amplified and diminished, a valley in the middle, one cohort holding two entirely different forecasts of the same sky.

Two rows filled. Two still empty. Under the grid, beneath the note about Tessa’s six hours, he added the day’s arithmetic before he closed the cover:

We measured the whole firm and found two firms. The elevator ride between them is four seconds long.

Predictions about artificial intelligence often focus on job losses and shrinking demand for lawyers. Filevine CEO and co-founder Ryan Anderson and product manager John Rizner offer a sharply different forecast. Drawing on the Jevons paradox, they argue greater efficiency will make legal services accessible to more people, encourage deeper legal research, and create work once excluded by cost. AI might reduce the effort required for individual tasks while expanding the overall volume and ambition of legal representation.

The shift holds major implications for the access-to-justice gap. Faster drafting, research, and document review would allow lawyers to serve more clients without sacrificing professional judgment. Anderson expects family law, immigration, bankruptcy, criminal defense, and employment litigation to experience some of the earliest growth. Motions, witnesses, and legal theories once abandoned over expense become economically viable, although courts face their own capacity crisis as more disputes and arguments enter the system.

Rizner explains how Filevine’s legal AI platform, Lois, applies machine learning to one of legal research’s oldest problems: traditional citators often return different results. Lois combines citation graphs with semantic analysis to locate opinions discussing related legal doctrines even when no direct citation connects the cases. A panel of models then evaluates potential conflicts and produces a structured memo. The goal is richer legal analysis focused on the precise holding or proposition a lawyer needs, rather than a simple flag attached to an entire opinion.

Accuracy still demands disciplined human review. Filevine organizes citation verification into three levels: confirming the cited case exists, determining whether the case supports the claimed proposition, and checking whether the authority is still good law. The conversation also examines Rizner’s research into how different large language models approach efficient breach of contract. OpenAI, Google, and Anthropic models produced dramatically different recommendations, revealing embedded legal and economic preferences beneath seemingly neutral answers.

The guests also explore how AI changes legal drafting, law firm economics, and the billable hour. Filevine’s acquisition of Pincites, now Lois for Word, reflects Microsoft Word’s continuing role as the shared language of legal documents, redlines, formatting, and negotiations. Efficiency does not automatically eliminate hourly billing. Lawyers might instead use saved time to produce more thoroughly researched arguments, stronger contracts, and work product approaching senior-level depth. Firms still need incentives rewarding efficiency rather than treating faster work as lost revenue.

Looking ahead, Anderson and Rizner predict a proliferation of frontier and open-source models tailored to firms, individual lawyers, and specific client relationships. Legal teams will increasingly pair proprietary knowledge with selected models to produce highly specialized analysis. Yet model choice introduces jurisprudential bias, accuracy risks, and serious training concerns for junior lawyers. AI expands the range of available options, while experienced legal judgment decides which arguments deserve trust, which sources require verification, and which advice should reach the client.

John Rizner Slides Filevine Primary Presentation - 2026

Listen on mobile platforms: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Apple Podcasts⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ | ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Spotify⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ | ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠YouTube⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ | ⁠Substack⁠

[Special Thanks to ⁠⁠Legal Technology Hub⁠⁠ for their sponsoring this episode.]

⁠⁠⁠⁠⁠Email: geekinreviewpodcast@gmail.com

Music: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Jerry David DeCicca⁠⁠⁠⁠⁠⁠⁠⁠⁠

Transcript:

Nikki Shaver (00:00)

Hi Greg and Marlene. Ever since Anthropic launched Claude for Legal, a lot of focus has been on whether firms and legal departments should be using that in place of or as well as legal specific applications like Harvey and Legora But Anthropic is not the only big player to brush up against the legal market. Microsoft has launched its legal agent, embedded in Word and other 365 applications. OpenAI has reportedly hired someone.

To help build in Legal. Perplexity has launched Computer for Council with a host of legal MCP connectors. And now it seems Amazon may also have its sites on our vertical with Amazon Quick for Legal. It was already a tough market to navigate when all you had to worry about were the thousands of legal applications. Now you also need to track what’s happening in the broader tech ecosystem.

We’ll soon be publishing a helpful comparison of the big tech offerings for Legal with an examination of where in the Legal Tech Stack they might be useful. Stay tuned for that on our site at legaltechnologyhub.com or follow along on LinkedIn at Legal Tech Hub. It’s a pleasure to see all of you. Until next week.

Marlene Gebauer (01:13)

Welcome to the Geek and Review, the podcast focused on innovative and creative ideas in the legal industry. I’m Marlene Gaybauer.

Ryan and John (01:15)

because don’t it’ll be never.

Greg Lambert (01:20)

And I’m Greg Lambert, and today we are going to be digging into a fascinating and and probably somewhat ⁓ counterintuitive theory that’s turning the traditional panic over AI on its head. instead of asking how many legal jobs that ⁓ the AI will destroy, we’re actually gonna explore why AI may actually trigger an unprecedented explosion of legal work, opening up massive markets that were historically priced out.

Marlene Gebauer (01:48)

in hearing what our guests have to say about this. And to lead our conversation, we are absolutely thrilled to welcome John Risner, product manager at Filevine. And joining John is Ryan Anderson, the CEO and co-founder of Filevine. John and Ryan, welcome to the Geek and Review.

Greg Lambert (02:03)

Welcome, guys. All right, John. Well, let’s ⁓ let’s start off ⁓ with you because we wanted to talk about the Jevons paradox, which I I know a lot of our listeners have have heard before. but you know, let’s look at the big macroeconomic picture that ⁓ that you know you guys have brought to our attention as well. you know, many pundits out there are predicting that AI will contract the legal industry.

Ryan and John (02:04)

We’ll see here Greg and Marlene.

Greg Lambert (02:30)

But you guys are arguing the exact opposite of that with using the Jevons paradox, and that’s the economic theory that you know the technology makes the resources cheaper and more efficient to produce, and then that way the consumption actually skyrockets. So do you mind ⁓ just talking a little bit about how the paradox applies to legal work and and how you know, how we should be prepared for that?

Ryan and John (02:51)

Yeah.

Yeah, totally. So

I’m gonna talk about it ⁓ in connection with a product that I’m working on, a citator, a la you know, Shep you know, Shepherd style citations within Lois. and when we started off this kind of investigating our tooling, I had a number of conversations with some you know old law school classmates of mine that did far better in law school than I did and went, you know, our you pellet litigators went through the elite clerkship rounds and I was talking with them, I’m like with them, and I’m like, you know, if you have the ability with

with large language models and with ⁓ kind of new AI tooling, you know, would you be excited to r have to kind of chew through fewer kind of research materials as part of your work? And what was interesting is all of them were like actually if I had unlimited time and unlimited resources, I would actually want to find more and read more as opposed to find less and read less. and they’re like, yeah, in a in a perfect in like if they could if they could, they would love to just like exhaust, you know, boil the ocean.

Exhaust every possible resource in kind of reading through opinions and reading through ⁓ you know relevant literature on their case before drafting or writing anything or preparing or preparing anything. And so I think that as you know, as AI tools, especially in the kind of legal research space, the citator space, as they make finding the relevant literature easier and kind of making available more of the relevant literature kind of

⁓ kind of an immediate re an immediate delivery to the user, I think rather than lawyers spending less time doing research, we’re gonna see lawyers do more, kind of spend more time doing substantive research. With the difference really being as instead of instead of legal reasons like instead of lawyers being like, all right, I’ve got you know X amount of time, I found these handful of cases, let’s quickly draft this and get this out the door, I think you’re gonna see far more in depth and nuanced arguments in legal work product.

due to the fact that lawyers will have an easier easier access to greater amounts of of research material. And I think as a consequence of that too, you’ll probably have better opinions come out of courts because the arguments lawyers are making are going to be more nuanced, reflecting the deeper research. So I hope reas lawyers like doing research. I hope that’s kind of why they went to law school because in my view of it, we’re gonna I think you’re gonna see greater and greater research and far higher quality research as opposed

to kind of lesser or more shallow research.

Greg Lambert (05:23)

Yeah,

that was gonna be my my follow-up question on on that we had last week and and one of the things she mentioned was, you know, used to we would get these one, you know, one page surveys or reports or and now we’re getting these forty page surveys. So, you know, and it’s it’s not necessarily better, it’s just more. ⁓ but I think your your argument is that the you know the outcome, the results, the research is actually

Ryan and John (05:35)

Ha ha ha.

Greg Lambert (05:52)

Better rather than just more research. Is that am I interpreting that correctly?

Ryan and John (05:55)

Yeah. ⁓

So I think I mean at at their core, and you know, obviously we all know, you know, as our ethical duties, lawyers should and should be diving through the cases that they’re possibly looking at. And I think as these tools evolve and become more mature and can find more relevant and more kind of on point items for the lawyers to read, they will read the same amount, maybe even read more, but in terms of what they the conclusions they come to will be stronger because they have that

better picture of how the law is at that particular moment. So I think you’ll you’ll volume will maybe said say volume may grow, and in addition to volume growing, I think quality becomes far better in in the long run.

Marlene Gebauer (06:38)

Yeah, I think that

that makes sense because you know, AI kind of allows you to kind of separate the you know, the wheat from the chaff and then you can focus more deeply on the things that you need to focus on and sort of leave the other ones.

Ryan and John (06:52)

Yeah, I’ll I’ll speak to that for just a moment too from a kind of a market perspective.

You know, two, three years ago there were a lot of predictions around what would happen to engineers, coders. Fileine employs a lot of coders. I think we will end the year with something like four hundred to four hundred and fifty engineers. about half of which are ML engineers, the other half would be kind of your traditional coders. And there were quite a few predictions that AI would reduce the need for corporations, for tech companies specifically, to have as many engineers

Because the theory was that there will be so much code written and it’ll be so much easier to be written that you won’t need as many engineers. And so you’ll they’ll actually you’ll see some layoffs and you’ll see fewer engineers hired. Well, it’s been the case at our company, and I think it has been the case broadly across the industry that the exact opposite has happened. The engineering market sort of hit a low maybe two or three years ago and has actually kind of climbed out of that, and more are being hired, not than ever, but certainly more than kind of post-COVID low.

Lows and more companies are bringing on more engineers to do more work, and there’s a very sort of logical reason for this.

We definitely write 10 times more code today than we did before. It might even be more than 10 times more. It’s a lot more code. But for every piece of code that we write, it has to be reviewed, it has to be QA’d, it has to be tested. And as the aperture of the product increases, the surface area for problems increases, but also what else we might may touch. The sort of ambition of the product also grows. And so what that has allowed us to do as a company is to build ⁓ a much richer.

broader, more end-to-end product offering that we think is really unique. But I think it’s the same for a lot of companies. The analogy really holds true for legal. As John brought up, you’re gonna have lawyers be able to do research more quickly and sort of find truth for their clients, I think in a much broader way and be able to zero in and find creative ways to apply their the facts of their case to the law. But even consider the lawyer who says, you know, instead

Of bringing that motion that the client just didn’t want us to do because it was going to be expensive, and so I didn’t bring the motion to compel the deposition of that witness. I just we just let it go. ⁓ we just didn’t do it in this case. That happens all the time. If you talk to litigators, they’ll say their clients didn’t want to pay for the you know witness number six who might have had some interesting information in the case. But boy, now that you can bring that motion to compel, and it’s much easier to draft, and much much quicker.

the opportunity costs change. And so the search for truth becomes sort of super powered. And of course litigation in particular is a counterparty affair, so what one side does, the other side o ⁓ is having to respond to and then maybe bring a counterattack themselves. So you know, for better or for worse, I think truth will get sought out more quickly, but also I think lawyers are going to be very busy. And I honestly I wouldn’t want to be a judge in this atmosphere. I think it is really tricky for the judiciary at this time. And I think

Greg Lambert (09:47)

Yeah.

Ryan and John (09:50)

One other item to bring on in the and in this truth question is that I think the uncovered it’s not like we are looking at a field where all the beared treasure has been dug up. I think I we one of the things we were looking at early on was research by Paul Hillier and Susan Mart, two legal scholars, and they had done kind of quantitative analysis of all right, how often do say, you know, Lexus, Wooden West agree or disagree on pr on a particular citation, or how often

Often are they returning kind of the same relevant opinions for particular citation analysis? And what’s really interesting is that you know the differences between the two could be substantial depending on the opinions looked at. So, you know, if you’re a subscriber only to West or a subscriber only to Lexus, you may be missing items that Lexus is returning but what West isn’t, or vice versa. And so where AI tooling makes it cheaper and easier to allow

to apply different methodologies to find so you can have your your multiple citators used taking different approaches where you’re where you are able to do that at a larger scale. I think then you start to discover more of those opinions that were otherwise hidden is you were stuck with one citator and only uncovering those ⁓ those hidden truths via just one type of methodology.

Greg Lambert (11:10)

R Ryan, I wanna I wanna go back to something that you mentioned about the using the engineers as a parallel path to to some of the things that we’re seeing ⁓ in legal. ⁓ I w I was listening to and it was I believe it was Adam Mazzari from ⁓ from Instagram that was talking about how his engineering teams are made up now versus how they were pre-AI terms, where he was saying he he had like a

You know, every project had like a baker’s dozen of of people that were on a project and that ranged from engineers to project managers to to researchers and data scientists. And he says now it’s it’s essentially like a team of three. and are you are you seeing similar things like that on your engineering team? And do you think there’s a parallel for how how lawyers will will be doing it?

Ryan and John (12:00)

It is so

Yeah, and sorry, sorry to interrupt you, but yes, we are definitely seeing a reduction in what a single person can do. And let me give let me give an example. This just happened today, about two hours ago. I was going through a new feature with our team. we were thinking through about how to design ⁓ an experience on our deep retrieval engine. We think we have really world-class retrieval. and we you know, we think we can kind of search through more documents than almost any product out there.

There. Well, we think we we literally think we’re the best in the world at this. but you might imagine there’s some UI constraints. Telling the user where you are when you’re searching through millions of pages of documents can be tricky. And there was some back and forth about how to do this, and the team’s debating, and there’s about 10 people on the call, and we’re saying, Well, how do how about this method to tell the user what’s going on? Here’s some other methods to tell the user what’s going on. And about 15 minutes into the conversation, one of the engineers goes, Well, they actually just coded it. Does anyone want to?

see what I just did. And we literally shared a screen and he showed something, not in a production environment, but on his local machine using real live code and said, here’s how I think we should do it. And it was working. And we can actually play with it right there during the meeting. I can tell you that kind of interaction never happened before. Never.

⁓ we have lawyers who say I was exploring an argument that I didn’t think would be a good idea, but I used Lois, and all of a sudden it found a couple cases, and it was a line of theor of of case law theory that I didn’t think applied to my case, but it does. And so we’re gonna actually attack the other side in a in a way that we hadn’t realized at all. And you when you lower the cost for exploration, for creative exploration, people become much more creative. I’ve always thought the best lawyers are actually like exceptionally creative.

It’s funny, you know, lawyers don’t think of themselves as creatives. They think of of artists and and movie stars and ⁓ musicians as creative people. But I think great lawyers are exceptionally creative. And by lowering the cost to to kind of go on a a creative thread with a legal theory, you enable much more creativity. It’s very exciting.

Marlene Gebauer (14:09)

So Ryan, when we talk about lowering costs to capture a dormant market of middle class clients and small businesses, what specific practice areas or legal needs do you think will see the quickest influx of new work? how does Filevine view this unfolding from a macro strategy perspective?

Ryan and John (14:29)

It’s it is there’s many great things about AI. There’s some scary things about AI.

But in the legal industry, perhaps the most exciting is we all know that getting a lawyer is actually quite a challenge. My brother, who’s our chief product officer, talks about how he needed a lawyer to help with a relatively small real estate dispute. here’s a here’s a guy who literally leads product for a a pretty significant legal tech company. has a brother who was a lawyer, ⁓ and knows many, many lawyers in his day to day job, and he could barely get somebody to answer the phone. it is hard sometimes to find a lawyer

And

that’s a well connected guy. if you’re somebody who’s ⁓ you know indigent, working class, maybe you you you know you you don’t grow up with the same kind of privileges or connections, it can be extremely challenging to find good legal representation. And it’s it’s pretty awesome that lawyers are going to have much more capacity now. I draw a really kind of quick example. you asked kind of what industries. I think you’ll see a lot more in family law. I think you’ll see a lot more.

immigration law. I think you’ll see a lot more bankruptcy. and I think ⁓ underrepresented ⁓ criminal defendants are going to be in much better shape than they are today. And in fact a lot of those state ⁓ public defenders offices are clients of Filevine. the Innocence Project is a client of Filebein. You you’re gonna be able to have more people who have meritorious claims for wrongful prosecution or for you know not n having a lawyer who kind of didn’t do the job they should be doing to

To get win their case or at least get them a lower sentence, many more of these cases can be taken on. It’s incredibly exciting. So I think you’ll see it across the board in those areas. And the the example I give is if you have a lawyer who has well these are lawyers, so we’ll say they have 10 hours in their workday, and it now takes two hours to draft a will instead of eight hours. Well, that means how many more people can they serve in a given day? ⁓ if they could if they could draft

one will in a in eight hours or maybe it was four and now can do it in two, well you’re now serving you know something like double the number of people during a day. You don’t even have to reduce your rates that much. In fact maybe your rates stay the same. They might even go higher, but you’re able to serve more people and those people wind up paying less because they’re taking less of the lawyer’s time. So it’s it’s kind of a very exciting world we’re gonna live in. I don’t know how that we know exactly how it will all play out. But more legal customers

will be served than ever before. It’s very exciting.

Marlene Gebauer (16:55)

do you

think about like labor and employment? Would that be one area area as well? Mm-hmm.

Ryan and John (16:59)

for sure. No, un undoubtedly.

you know, I would love to say that this is maybe good news for corporate defendants. It’s probably not good news for corporate defendants. so they’ll have more tools at their disposal to investigate these claims, and they’re often not meritorious. And so, you know, Filevine wants to help those those customers.

Greg Lambert (17:06)

Yeah.

Ryan and John (17:17)

But but also you know plaintiffs’ lawyers who have meritorious claims against ⁓ corporations also use our products. And so, you know, we really want to help lawyers find the truth. We want to help the justice system find the truth. we are huge believers that the American system of justice not only is the best in the world, but forms the infrastructure for a fair and just capitalist society. We think it’s really critical. the lawyer’s role in a in a in a system.

capitalism because otherwise it’s hard to keep business in check. So we’re really proud to serve our customers, you know, sort of whatever side of the V that they they may be on. But yes, I think you can expect an increase in litigation across the board.

Greg Lambert (17:52)

I I’ll go back to when you were saying it it’s gonna be a a a difficult time to be a judge right now because I think that that’s one area that ⁓ i if if we don’t figure out how they are gonna handle this massive influx of new cases, ⁓ it doesn’t matter really how much of the efficiency gets on the on the plaintiff defendant side, if there’s no, you know, if the if the court system is inaccessible. ⁓

Marlene Gebauer (18:20)

I I just wonder if they’re

gonna get like AI version of Judge Judy. You know, it’s it’s like people peop people will agree that like okay pe people will agree to to the the AI judge and and let them decide, you know, in s in smaller matters. Yeah.

Greg Lambert (18:24)

Are you guys working on that, ⁓ Ryan?

Ryan and John (18:24)

Yeah.

Yeah.

It might.

Yeah, I I you know, you could see kind of smaller matters, private matters. you know, obviously I think there’s a different need to be there, but Yeah, I think so. I think you you can envision s you know, small arbitrations that are private where we you where at least they agree to maybe some AI tooling being used. I think that’s I think that’s almost certainly going to to happen.

Marlene Gebauer (18:38)

See that for like mediation, sure, like stuff like that. Yeah.

Ryan and John (18:55)

it probably should. It probably should happen to a degree. let me be clear though. Filevine and and and Lois, which is our

Which is our AI product is the core economic engine of this business. We basically only sell Lois and AI products today, which is quite a difference from where we were three to four years ago. So AI is incredibly important to me, it’s incredibly important to this company. We are fully AI pilled on how bullish we are about this industry. Having said all of that, I think the judgment of the lawyer, the judge, the legal professional.

Is isn’t going away for a really long time. it is one thing to see an A an AI output as ⁓ unsophisticated consumer of that information and say, geez, this kind of looks pretty good and pretty persuasive. But I think all of us have seen enough AI results and prompts and frankly slop to go, hold on, hold on. This looks like it’s right, but it is in fact not right, and sometimes in really critical but perhaps nuanced ways.

That kind of legal judgment is going to exist from a long time, for for quite some time. And so I think we’re a very far ways off from lawyers being replaced, from judges being replaced. but as I’m sure everyone in your audience already realizes, we it the the age of AI and legal is here. it has been here now for a couple of years, and is very squarely in our era. We will talk to our grandkids about this transition, it is a very big deal. but but we’re huge believers in the prime.

of human judgment when it were cover when it comes to the law, we we believe that they lawyers play a really critical role. And you know, I think I want to tie back to some research that we did this last year on how LLMs respond to and we may get get into the paper in a little bit in this in this in this podcast, but we were looking at how LLM different LLMs or different families of LLMs react to the same kind of law and economics ⁓ breach of contract issue and how they compare to humans. And

what was maybe surprising, maybe unsurprising, was that depending on what model or what family of models you were using, you could see dramatically different results on whether or not whether you know the the LLM just was pushing you as the ⁓ kind of pushing the judge or pushing the practitioner to push for a breach of contract or to to keep a promise. So I think there could be even a future in which you know you have if we’re talking about our L our LLM based arbitrators where folks are are fighting

over which LLMs or which which tech tools to use because there is that that risk of the tool you use could affect what answers you’re coming to. And so I think that’s an even an another argument why the human attorney needs to remain a core part of the pro the practice because we I think there’s a danger or there’s a there’s there’s a I the I don’t think we’re yet comfortable giving over our judgment to tools that might already have a judgment

That we don’t even agree with built in with that. Yeah. And of course we’re all SQL. Yeah. Go a go ahead.

Marlene Gebauer (21:47)

John John, I no, no, go ahead,

sorry.

Ryan and John (21:50)

No no, I mean look we all see the hallucination news daily. and ⁓ it’s it’s everywhere. I think lawyers are right to be very concerned about it. I think judges are right to take a really strict view of hallucination. and it’s it’s out there all the time. It is an extremely challenging problem. We’re you know, we’ll hopefully we’ll get a chance to discuss sort of what we’re doing on anti hallucination. That’s an entire team at Filevine, the anti hallucination team, certainly with respect to case law. But

I’ll just give you one example. We had a customer ⁓ put some data through Claude versus Lois. we think Lois is much more precise than Claude. and Claude came back and it had taken some some testimony and it had put in quotations something that a witness had said, literal quotation marks around what a witness had said. And you know, it turned out that witness hadn’t said that, but it was sort of an amalgamation of three or four things the witness had said. If you if you looked at each statement the witness had said,

They

You can understand how Claude would have arrived at the conclusion that the witness had said this quote, but the witness had not said that. And any lawyer looking at the quote versus what the witness had actually spoken and verbalized in the deposition, totally different. Totally different. And so like where the the legal judgment to know the difference between what is close and what is precise is gonna be needed for a long time. So we’re not there yet. but a AI tools are very important, but they have to be watched over for sure.

Marlene Gebauer (23:15)

Well, I want an anti-hallucination t-shirt. Like anti-hallucination team t-shirt, that’s that’s what I want. so, John, Greg and I saw your presentation at Texas Trailblazers a few months back, and you noted that AI adoption follows incentives and that trust in AI is ultimately a workflow problem. So, since of course we’re talking about

Greg Lambert (23:18)

Yeah.

Marlene Gebauer (23:38)

Gen AI, it wouldn’t be fair to not have a billable hour question. So if if efficiency gains allow lawyers to draft, you know, a complex contract or review a file in, you know, fraction of the time, how do firms need to restructure their incentives so they aren’t punishing efficiency under this traditional hourly model? And honestly, I mean, if you have this, that’s the secret sauce because we’re all grappling with that.

Greg Lambert (23:43)

Ha ha.

Ryan and John (23:43)

Yeah.

Yeah.

Yeah.

Greg Lambert (24:05)

This is this

is the billion dollar, maybe trillion dollar question.

Marlene Gebauer (24:07)

Yeah, so here you go.

Ryan and John (24:09)

And I think

I I think on the one hand the buildable hour has survived a lot of technological change so far and has not disappeared yet. so I think the billable hour has some some weight to it that’ll be hard to remove. I think coming back to our

to kind of Ryan’s point on the will piece and as well as our opening point about quality, I think say ⁓ say writing a writ writing a doc you know producing a particular piece of work product today takes eight hours and we push it down to two to get that same that same exact work product. Well where

Where there’s still runway to improve on that particular document, a better researched argument, a more ⁓ a more just you know ironclad contract, kind of a will that really thinks through all the the the elements of you know this particular family situation and gets it exactly how the the this is throwing back to my law school days, the testator, is that the right term? Gotta back to the to trusted states.

But I think with that extra time, now the associate or the partner has the ability to provide a better work product, something that really reflects what their particular client needs at a same same price or less price, or maybe maybe a slightly more price, but with a depth that wasn’t even close to what we could we could we could achieve today.

it then the the the deep nuance that an associate may not have been required or expected to have today will be a require I think will be a requirement in the very near future and in the in the long run. So those extra six hours might be will be spent producing that, you know, higher level, almost partner level, you know, deep quality that just wasn’t accessible or wasn’t doable today.

So I think the billable hour survives for a while, unfortunately or fortunately, depending on what side of the debate you’re on. I know I never enjoyed the bilbable hour, but I think the the economics of it still work, especially if you think that volume and quality become a a greater and greater focus.

Greg Lambert (25:57)

What?

Yeah. I I’ve yet to meet someone that loves the billable hour, but yet here we are.

Marlene Gebauer (26:15)

When you find one, let us know. We want them on the podcast.

Ryan and John (26:15)

Yeah.

Marlene Gebauer (26:19)

It’s like explain yourself.

Greg Lambert (26:21)

Yeah. ⁓ John, I wanna I wanna ⁓ also jump in on your presentation that you did ⁓ a couple months ago at ⁓ Texas Trailblazers because you you actually kind of I I wasn’t sure how well that was gonna go over because it was it was a pretty in-depth ⁓ very very deep dive on citation ⁓ systems.

Marlene Gebauer (26:38)

We loved it.

Ryan and John (26:39)

Mm-hmm.

Yeah.

Greg Lambert (26:44)

but I asked a couple of partners ⁓ at my firm who were in the room there, and they were like, this was the best part of the whole conference. I love I love this. and so let’s jump into that and and more on the the lowest legal research product that you guys have as well. you’ve noted that you know traditional cite haters tell you about the case, we kind of talked about that earlier.

Ryan and John (26:52)

yeah.

Greg Lambert (27:08)

⁓ but lawyers really care about the specific holding that that’s going on. do you do you mind kind of giving us a ⁓ almost like a reader’s digest version of of ⁓ what you what you presented there and and talk about kind of these dual pathway retrieval systems under the hood? I’ll I’ll turn it over to you.

Ryan and John (27:26)

Yeah, absolutely.

⁓ so it all came kind of our citator approach all came with the idea of all right, are there path are there workflows in the current legal system that we think we could mimic using ML tooling, LLMs to come to you know to come to the same answer that they that the normal that the current you know legal process works through. And so what we what we ended up falling into is, you know, on the one hand, citation graphs in your your traditional citation approaches does a really good job when

And we’re looking at those cases that directly engage the mother, you know.

Chevron and Loper Bright, for example, where you have you know Loper Bright talking expressly about Chevron. But there’s all those cases that might talk to issues that are around the issue that you really care about, but might not expressly be cited by what is, you know, what the the the the potential treating cases because it may be only a part of the larger case or it might be you know the two the the the treating and the your target citation.

might kind of those for whatever reason, a clerk that didn’t include it, a judge that wanted to make a strategic decision, or those those, those links just might not exist. And so we thought is all right, can we find a bunch of opinions that might be relevant either on a citation graph or as as we came to it, ⁓ via semantic similarity, kind of the the inherent meaning the kind of inherent meaning of the text as turned into into into into kind of a mathematical representation. Could we find all of

The opinions that could exist and could be relevant to the lawyer’s issue. And then from that, let’s pass that into almost a mock and bond process where we have a bunch of LLMs operating as judges trying to decide: hey, do we care about these opinions? Are they is this a relevant opinion to the issue we’re looking at? do we think that it conflicts with the wish issue we’re looking at? And from those kind of mass LLM runs, then produce a structured memo for the user saying that, hey.

Hey,

of you know, of this in this universe, we found you know these opinions that might cut across your specific chosen issue for these reasons. And what’s what we’re really excited about is finding those opinions that are really that do negatively engage one another, but that are really hard to find because they’re not on they’re not cleanly on a citation graph. so like the one ⁓ that I always go back to, I’m gonna go look and you know look here in Utah. And in Utah there’s this case called

Brinkerhoff East Salt Lake City. And Brinkerhoff East Salt Lake City looks at the last kind of few paragraphs talk about this old doctrine of governmental immunity. And there’s this this test where it’s like, you know, this government this question of governmental immunity turns on if the government activity is like proprietary or governmental. And if you pull up Brinkerhoff on some traditional citators, you’re not going to see any negative flags on it. And the reason is that though kind of Brinkerhoff has.

Has been talked about in a negative way in later down the later on the road Utah opinions. But the problem is the this doctrine that Brinkerhoff used in the final few paragraphs was ultimately cut down by Utah, the kind of Utah Supreme Court and Utah legislature. And because we are one, surfacing the fact that Brinkerhoff has talked about this doctrine, regardless of whether or not those later cases mention Brinkerhoff, and two, because we’re giving it to this LLM panel approach, this kind of

voting and memo writing approach where the LLMs can recognize, wow, this doctrine is being used for Brinkerhoff. Wait a minute, here’s Standerford, this later in time case that attacked the doctrine. We think there’s a connection there, that’s a conflict there. We are able to find these items that are, again, otherwise unfindable on a ⁓ traditional citation graph. And so we’re really, that’s the thing I’m really excited about is we’re able to find for lawyers these ideas and conflicts that may be relevant to whether or not you know other sides put an opinion

and the we want to kind of surface to the lawyers, hey, there might be other attacks on that particular proposition of law that we can now can help you find that were just undiscoverable before.

Greg Lambert (31:36)

Yeah. man. I it just it brought it that presentation all back to me. I was I’m I’m remember how how I geeked out on on that train.

Ryan and John (31:44)

It’s very cool. I don’t know, you know, I know this will be posted ⁓ in transcript form. If there’s a way to show some of John’s slides, I mean that you know this the slide that shows that Yeah, yeah. I mean I you know, I think as you all probably were when you saw the presentation, the just the notion that Westlaw likes us find different sets of cases. Good night, that’s terrifying.

Marlene Gebauer (31:53)

Yeah, you if you can give us a link to the slides, we will post that up in the show notes, absolutely.

It’s so cool.

Greg Lambert (32:08)

Yeah.

Marlene Gebauer (32:09)

We’re just like, yeah, yeah, someone’s finally saying it

out loud, yeah.

Ryan and John (32:13)

Right,

right. and then you know, we’ve we’ve just shown that this method actually picks up a lot of cases, that in some cases neither of them found. ⁓ so it’s it’s really interesting.

In doing the when we were kind of starting to first benchmark our tooling, you know, we would have been pulling down those those studies by like you know Susan Mart at the ⁓ in Colorado about the different cases they were finding. And we were like, I wonder if, you know, some of these some of these research was was done, you know, eight years ago or five years ago. I’m like, I wonder if because of technological change, these differences have started to go away. In our current benchmarking, not so much. Still the dramatic divergence of what site tater you choose, you have

using is gonna give you a particular answer and if you’re using a different one you’re gonna get a different answer. So it’s amazing just how we think you know just how much opportunity there is to help fill those gaps and find those and you know give folks an ability to find things that they otherwise aren’t finding right right now with their current tool.

I’ll I’ll just briefly note, you know, we have such a wealth of case law, rich a rich history of case law in this country. It’s so cool.

That you know, most things have been decided. The vast majority of things have been discussed at some point by some court somewhere, which is gives the such stability to businesses, to human beings, to politicians, to people operating in our country. And it gives you know everyone a tremendous amount of confidence when they make decisions, especially business decisions, to sort of understand what the nature of the law is. And yet the citation system that we use was was written and it was the best we had, but in a really ⁓ in a deterministic code kind of.

Of way. Here’s a citation, it links to this other citation, which links to this other citation. And that’s great. I mean, that’s what we had available to us, and it took humans to sort of chain those cases together in sort of logical trees, you know, progenies of different cases. But LLMs are particularly good at saying.

These words, th these phrases are semantically similar. They have similar meaning to these to this other set of cases over here. And even though there’s no hard coded citation, we find similarity here and you should take a look at it. it’s it’s really fascinating. It’s a great use of LLMs. And the other like go ahead, Marlon.

Marlene Gebauer (34:22)

‘Cause I think about like I ⁓ I think about I just think

about the discrepancies that you’re talking about and like how does that impact and and we’re gonna talk about the hallucinations, but like when you’re trying to check, you know, for hallucinations and for legitimate sites, like, you know, how does this play into that and and how you know the courts are are coming down on this, so how do people rely on the tools that they have? Or should they?

Ryan and John (34:48)

Yeah, I mean we think we have a solution. I’ll let John talk about that, but So you know, in my view, it doesn’t matter in this is you know this is j this is doesn’t even matter it’s anything specific to legal research in terms of case law.

But you know, there’s such an importance of having corpuses of your of what you the information you care about available and engineered in a way that is kind of best presented to the LLM in the right way at the right time and in the right process. I think the the the real question I think lawyers need to be asking their tech providers is: all right, what corpuses are you using? How are you presenting them to your AI technology? and you know, what what benchmark

what what kind of evidence do you have that your approach is working well. So you know it it doesn’t matter if it’s a, you know, you’re trying to find you’re using AI in your case file, you’re trying to find the right document that covers that you know outlines a you know a scientific scientific expert expert’s you know findings on something or a point from a deposition that you really care about or if it’s on the legal research side, a particular opinion or particular citation that you that you care about. Either way, it’s really important that

you have that your legal tech has has engineering designed to sort to get to the right point in the corpus for the answer to care about. And for us, you know, on the we’ve got a on our on our kind of document side, and and Ryan’s was talking about that earlier with our ⁓ with our kind of enormous ⁓ data science groups, you know, they you know they’re really focused on our corpus of of case file material, on the on the and kind of stopping hallucinations by using what we call knowledge

engine to get the right answers surface the LLM. On the citational side, what we’re focused on doing right now is taking open source corpuses of kind of grounded ⁓ of ground decisions, in this case court listener being one our our our big partner in that. And for any time text is produced within in thin Lois in the chat, whenever we see a citation pop up to have to have tooling then check okay, you know the you know this

This

has been brought up. Does this case exist? And more importantly, does the user have an opportunity to check if the opinion, if the discussed opinion is discussing the issue we care about? Because I think one of the things that we’re now seeing judges talk about is okay, the case you cited exists, but it doesn’t support all of the propositions you’re trying to support. And in my mind, and this is what we’re building towards, there’s almost like three levels of evaluation you have to do for any particular piece of case law.

Marlene Gebauer (37:10)

It doesn’t say that.

Greg Lambert (37:11)

Yeah.

Ryan and John (37:21)

First level, is this case actually real? Does this citation actually exist somewhere in the corpus? And that’s like your your very base level, right? Is does John is John B. Ryan in the territory of Guam wasn’t entirely made? Was that citation made up? On the second level, you have that second order hallucination. Does that citation actually refer or that case refer and support the pres preposition you care about? sure the case exists, but you know, i are we tr are we talking about, I don’t know,

you know, Roe v. Wade in a contracts dispute. Like there’s no the there’s no the what you’re what’s being cited doesn’t support what you care about. And I think on the third level is that citational analysis. Okay, the opinion’s good, the the opinion exists. The it supports what you care about. Now is it good law in the larger common law analysis? So I think in terms of building, that’s how we’re trying to think how we’re thinking about it is are we building towards hitting all three of those evaluation levels?

Marlene Gebauer (38:16)

All right, I want to switch gears for a little bit and move away from kind of litigation and more into the transactional ⁓ area. So, you know, Filevine started with a strong focus, you know, in plaintiff’s litigation, but this this January you acquired PinSites, which is a legal drafting and redlining tool, and brought in Sonia and Merriam ⁓ Solakian ⁓ into your executive team and rebranding the tool as Lois for Word.

⁓ and just as a little bit of background for listeners who might not know, Sonia was formerly a legal strategy expert at Ropes and Gray, and Miriam is a former GitHub meta product engineer. So congratulations to you. You brought in the dynamic duo. why and this I have a couple questions here. Like, why is it, you know, why is it critical to corporate and transactional expansion?

to have something inside of of Word. And particularly because I think we’re seeing now with with some of the larger LLMs that they’re actually doing, you know, they’re actually doing a lot of the work inside that environment. And I I realize they’re doing stuff in in Word too, but you do see a lot, particularly when you’re working with you know larger sets, that they’re actually doing it within the LLM environment.

Ryan and John (39:31)

Yeah.

Look, it’s a great question. I don’t think Word is the final surface for all legal drafting, but it is definitely the dominant surface still today. and we’re not even talking about AI legal drafting, just legal drafting. The lingua franca of law is still Word. and and probably will be for some time. If I said to you, hey, here’s the red lines to this agreement.

Go ahead and and ⁓ open this link with this other document type you’ve never heard of. Don’t worry about it, you’re gonna have to sign in and make an account to see it. Marlene, you would say, No, thank you. yes, ⁓ hard pass, I need the Word document, please. and you know, I I think pretty much every lawyer feels the same way. They have all learned to use Word, they understand it, they understand how comments work on Word, they understand how red lines work on Word. So getting an entire industry

Greg Lambert (40:10)

Yeah.

Ryan and John (40:26)

to sort of replatform may be challenging. certainly in the short term, probably in the medium term, maybe not in the long term, but replatform on a different drafting kind of

modal is just it’s gonna be really hard. So I think we’re stuck with Word. I mean maybe that’s the right way to say it, maybe not. I mean Word everyone uses Word because it is the most fully featured drafting product the world has ever seen. So there’s some good things about Word.

Marlene Gebauer (40:52)

Greg likes word

perfect. Sorry. He’s a big fan.

Greg Lambert (40:53)

I was gonna say Word Perfect four point two on DOS. I mean I’m I’m up for bringing that back.

Ryan and John (40:56)

There we go. you’d be surprised. I’ll think so recently

we still got a lot of customers asking us to integrate with

Greg Lambert (41:01)

Yeah, ⁓ those reveal codes, man. Still

need those.

Ryan and John (41:06)

so I think that’s really critical. So first of all, s Son and Marion would be the first to tell you, and by the way, Marlene, you are right. they are the dynamic duo. These are two very sharp women, sisters, of course, ⁓ and it first of all, lovely human beings, just a delight to work with. but as as smart of folks as I’ve ever worked with in my career and and really fun to have them on the team. And they’re building out they’ll they’ll anchor our

Francisco office, and you know that was some time ago now, six months ago now, so we’ve actually built quite a large team around them at this point, ⁓ and that team is solely focused on AI legal drafting. and of course, they use a lot of the work that that John has done and others, but they would be the first to tell you that Word is not the only surface that they’re going to work on, and they already work on ⁓ other ways to draft. But yeah, given the gravity that Word has in the industry, I think it’s gonna be around for

while there is a benefit to having everything be in word getting the formatting right is almost impossible without word depending on the chord so

Marlene Gebauer (42:04)

Formatting is

a tough thing.

Ryan and John (42:06)

It is very challenging. And look, I I understand judges, I fully understand judges’ particularity and frustration around hallucinations. you know, the but they get equally as frustrated around some of the tiniest formatting issues. I I’m a little less like, you know, okay, I mean, is it really does it really matter ⁓ i if if if you know you have ⁓ the the margins are one point two five instead of one point one or something, but but I can tell you that it does to them and

It does to them, and if it does to them, it does to the lawyer practicing in their courtroom. so it’s really not optional. And ⁓ Word can reliably produce the the best format of documents for legal in the world. And so we need to be on that surface, and we want to be world class there. And we do think ⁓ Lois Forward is world class. We’ll put it up against Claude for Legal or or Harvey or Lagore or any of our other competitors. We think it’s the best redlining and drafting tool out there. And you know, to that end, we should note that.

⁓ not only is Lois for Word very good at redlining, we think we think it’s the best at redlining, but

The ambition is much stronger than that. It’s much broader. It is it is drafting. and drafting long form, sophisticated, complex legal documents grounded in evidentiary citations and case law citations. That is an ambition that I I don’t think many of our other competitors have have gotten to quite yet. ⁓ maybe some, maybe some at the top of the market with us there. And ⁓ that is that’s the product we’re building and we feel really proud to build it. with with those two at the helm. They’re gonna they’re

They’re doing great and we’ll do great.

Greg Lambert (43:37)

Yeah, sound sounds like they’re a lot ⁓ a great team to work with.

Ryan and John (43:41)

They’re awesome. Yeah, I don’t know if you’ve seen an interview before or met them individually, but ⁓ sharp, lovely, fun, and i incredible, I would say like impeccable product taste. ⁓ we just we think that their sort of thought processes and and intuition around how to build products that lawyers really love and want to work with day in, day out is second to them. Sure, well I can tell you they would they would be better. Yeah.

Greg Lambert (44:00)

Well now I’m regretting bringing you two on. We should have brought those two on.

Marlene Gebauer (44:02)

I was gonna say now we have to bring them on. So I think were you teeing them up, you are teeing them up. Good. Cause

I think that would be a fascinating discussion as to why, you know, lawyers like tools. I d I don’t even know that they I don’t even know that they know why they like tools. So I think that would be cool.

Ryan and John (44:15)

Yeah.

I think you’re right. well, we’d love to have Soda Marium ⁓ come on the podcast sometime, so

Greg Lambert (44:27)

Well well, John, I’m gonna ⁓ jump ahead because I want to do one one more geek out with you while while we got you here. ⁓ and ⁓ I want to bring up a ⁓ a working paper that you co-authored called The AI’s Philosophy of Contract, where ⁓ use ⁓ you know im empirically studied how frontier large language models are handling, you know, classic concept like d efficient breach and remedies. So

Ryan and John (44:32)

Sounds great.

Greg Lambert (44:55)

You know, here’s here’s our chance to geek out. so what what did this ⁓ reveal in in you know, what kind of interested you in in writing this paper?

Ryan and John (45:04)

Yeah, yeah. So we I we came across a paper that

Talk to Babeso is an empirical study of how humans respond to kind of your classic efficient breach scenario of you know it is it is more economical that you breach the contract as opposed to adhere in and follow the contract. And humans, interestingly enough, you know, they had certain breach rates, but then if you ⁓ if you included a a specific remedy in the contract, you could you would ch you could change how humans kind of in the behavioral economic sense respond to those.

efficient breach scenarios. And so we were curious as LLMs become a place where you go to kind of ask for legal advice, how they might respond to the will they follow humans in how they respond to efficient breach? Or will they take a more kind of cold economic law and economics view? Or will they take a softer view? And so that was our kind of big question. So what we did is we took the at the time kind of what were all the main frontier LLMs, anthropic Google, open AI,

And we created just large sets of efficient breach scenarios and would kind of determine, like you know, kind of tracked how the these LLMs would respond to those scenarios. And the diversions was wild. I mean, open or you had Google and OpenAI who were more like, all right, you know, it’s economical, go ahead and breach. Well, Anthropic was like, we are never breaching, we cannot breach. And you know, you might have a swing from like, you know, high 90% breach rates to like under 10% breach rates depending on what.

Model you were choosing. And you know, the big striking, that was kind of one of the our big takeaways, was that you know, this is a for practitioners, this should be a thing to think about because what model you’re using might determine if you are getting advice of all right, let’s tell the client to do X versus let’s tell the client to do Y. And the other kind of big kind of notes that we were finding as well is the LLMs likewise respond to whether or not you had like

specific remedies or specific you know specific breach remedies within the contract as well in a way that a that a human would would also respond to those types of kind of hints within the contract. I think the the the big thing we have to there’s a lot of discussion in the Frontier Labs in San Francisco about you know alignment of of humanity generally with these tools. I think lawyers have to think need to start thinking about alignment as in does this particular LLM align with my

particular you know jurisprudential philosophy or how I would approach this particular question at like a you know a more a theoretical more philosophical element that kind of element of ⁓ that element of of of judgment because if we just if you just hand it over to the LLM what answer you’ll get and what legal advice you might get will determine if you happen to be you know talking to anthropic or talking to open AI on a particular day. I think a really interesting question will also be in you know when we ran

That empirical data and the open the open kind of open source models hadn’t yet had their big their big day in the limelight, it’ll be interesting as well to see to what extent open source models become a way in which to choose a particular piece of jurisprudence baked into the model that either you prefer or that a particular court prefers or a particular judge kind of prefers. It’ll be, I think there’s real questions on that as models proliferate, of folks kind of choose.

Choosing models that might fit their particular jurisprudence jurisprudential, say that five times fast.

Greg Lambert (48:32)

Yeah. Yeah.

Marlene Gebauer (48:32)

This

is this go ahead. Sorry, Greg.

Greg Lambert (48:33)

It it just made me just made me think whether or not it follows the University of Chicago economics theory or Berkeley economic theory. So I guess how it’s trained, right?

Ryan and John (48:38)

Yeah. Yeah. Yeah.

Marlene Gebauer (48:39)

It

Ryan and John (48:43)

Yeah, exactly.

Marlene Gebauer (48:43)

It’s

it’s super interesting because I was I just did a client presentation this morning and they were talking about like okay, even if you know it’s an approved f ⁓ enterprise foundational model, but if you know people put in and it’s it’s like the the clauses are are are not what you know they would they would put in, but that’s what’s recommended and that it’s just a a you know it’s just it just sort of fo i it

It highlights how important it is to have a playbook or have some sort of template or or guidance to use in addition to the model, because you know, you may be doing something, you wrong. So that’s one point. And and the other point is I think about more junior people using these tools. Like, you know, someone who’s got more extensive experience would, you know, look at this and automatically say, absolutely not. But

Ryan and John (49:39)

Yeah.

Marlene Gebauer (49:41)

somebody who’s more junior and who’s actually using this as a learning tool in addition to a drafting tool is is sort of really going down the wrong path.

Ryan and John (49:51)

I couldn’t agree. I think it’s very scary. ⁓ I I think junior attorneys boy, you almost wonder how much they should be using it.

it it it’s very challenging and you know, we see at the company, you know, some of some of our employees who are who are junior at the company or new to the company, they often rely far too heavily on on AI analyses and you know, responses from from frontier models and gosh, you you wonder did they even think about these things? so i it’s a big challenge. and ⁓

For that very reason, we’re we’re we’re huge fans of lawyers and legal judgment. We think it’s gonna be around for a while.

Greg Lambert (50:28)

all right, guys, we’re gonna jump to the ⁓ crystal ball question. So looking out you know into the near to future, what are some challenges or or changes that you think ⁓ we’re we’re gonna have to be prepared for as as we move along in this age of AI?

Ryan and John (50:46)

Will see model proliferation. You know, we’re we’re speaking to you on a day that Meta just came out with, maybe their first really good model. and I don’t I don’t think anybody sort of had that on their bingo card, but here they are with a model that looks at least initially, based on sort of you know what the commentary is online and even some testing done, that it’s a pretty good model and pretty good at legal and so now all of a sudden there’s at least a fourth player, probably a fifth. you know, if you say Gros.

Gemini, Claude, OpenAI, and now Meta, you’re gonna see a lot more of these. We internally are using more open source models than ever before. it is it is now sort of you know part and parcel of the work we do to ⁓ fine-tune and train open source models. and so you know a lot of folks I think believed that the labs were there was gonna be maybe two, maybe three dominant labs, and people are gonna mostly build on top of them. That does not

not look like it’s it’s going to be the case. and in fact, you know, we’re seeing seeing the open source models be as good or better with you know a relatively limited fine-tuning than even some of what the frontier models can do. So that really changes the economics for for everybody. First of all, it means we can use a lot more inference. It’s it’s not as costly as before. It also means that there will probably be a lot more folks choosing to kind of you know end customers

Law firms, I think, choosing to have a stake in helping to build their own models, and we want to be there when when customers do that. That’s part of the beauty of Lois is law firms can really customize it to them and keep sort of what makes their firm great within Lois. So I think you’re gonna see a huge proliferation of models, which is gonna be different. It’s gonna be confusing. It’s not, I think, the environment a lot of people expected to see even just a year ago. So that’s my big crystal ball prediction.

and I think ⁓ the one piece I would add to that too is

I think with the proliferation of proliferation of models, you’ll see just increasingly and I like to I like to call it like synthetic secondary sources. I think with your proliferation models and folks having their preferred, you know, their preferred work product and really what they have spent, you know, for these some of these firms decades upon decades tuning and building their particular approach to the law

I think you’ll have tool y tooling a me kind of tooling and models emerge where your the the ⁓ the analysis provided to a particular lawyer is so customized where it is the specific model and specific data source coming together for like the exact you know, if the the if the firm name is you know is Ryan and John L O P, it is a it is a technol technological marriage of

our exact corpus with our exact preferred models to get our exact answer at a scale that’s just not been achievable because there’s a lock the you know the the knowledge and the the style and the preferences were locked in the brains of the partner or maybe the senior associates. So crazy ability to get just ultra custom legal outputs for users. I think you maybe not even are going far enough. It maybe it it may not even be at the firm level. It might be the lawyer level and it might even be the lawyer paired with the client.

You could envision a world in which you say for this client we prefer this model. and I I can see that very easily being the case. we already orchestrate through multiple models just to give you a certain response to a query. ⁓ and so ⁓ it is it’s going to be ⁓ a a world where you see a lot of models, they’re gonna be used in a lot of different ways and they’re gonna become more and more specialized. It’s very exciting. thank you all both so much for the time today. Yeah.

Greg Lambert (54:08)

Yeah.

Marlene Gebauer (54:10)

Mm.

Greg Lambert (54:22)

Yeah, you got it. John John and Ryan, thank you very much. Appreciate it.

Marlene Gebauer (54:22)

thank you.

Ryan and John (54:22)

Yeah.

This is great.

Marlene Gebauer (54:26)

Yeah, and

thanks to all of you for listening to the Geek in Review. If you’ve enjoyed the show, please share it with a colleague. We’d love to hear from you on LinkedIn and Substack.

Greg Lambert (54:35)

And real quick, do you guys if they wanna learn more, where where’s the best place for to reach out?

Ryan and John (54:40)

probably just hit me up on Twitter, DM me on Twitter. ⁓ R Ryan Filevine is my username on Twitter and and you can find me there. I guess X. I guess we call it X now, right? So it’s X, yeah.

Greg Lambert (54:44)

Got

Marlene Gebauer (54:48)

And as always the music

And as always, the music you hear is from Jerry David DeCicca Thank you, Jerry, and goodbye, everybody.

Ryan and John (54:56)

Thanks all. Thanks.