One of the core challenges every AI company runs into is that the demo works, the contract gets signed - and then someone has to actually go inside the customer’s enterprise and make the thing work against their data, their systems, and their edge cases. That last mile is where a lot of AI companies win or lose the account. Quite often, the person responsible for it carries a title that barely existed three years ago: forward deployed engineer. It is probably one of the fastest-growing roles in AI, one of the hardest to hire for, and - apart from Palantir - almost nobody has figured out how to scale it.

That is the conversation I dug into with JJ Refling, who has built forward deployed teams from scratch and scaled it; he recently wrapped up his time at Lorikeet and is now exploring what he’ll build next in the space. For context Lorikeet builds AI support agents for companies in regulated industries - fintech, health tech, and other regulated sectors - and has raised over $50 million from investors including QED and Blackbird. Before Lorikeet, JJ spent his career incubating new businesses inside Google, Meta, and Stripe, and helped launch the embedded payments company Rainforest.

This interview has been condensed from our conversation and lightly edited for clarity; speaker names and company names have been corrected from the auto-generated transcript. Infographics are embedded at the point in the conversation they illustrate.

Topics covered:

1. The origin - incubating new businesses, from Google to Lorikeet

2. The role, defined - FDEs work backwards from the customer’s problem

3. The superstar problem - hiring for three muscles almost nobody has

4. Playbooks and shared learning - turning one-off implementations into repeatable systems

5. The economics - why great FDE teams scale non-linearly

6. The road ahead - presales, evolving roles, and implementation agents

Incubation as a Career: From Google to Lorikeet’s First US Hire

Take us through your journey - Google, Meta, Stripe, Rainforest, and then Lorikeet. What’s the common thread, and how did you end up building one of the earliest forward deployed teams at an AI company? And tell us what Lorikeet does - that context will be helpful.

JJ Refling: In terms of career background, I’ve worked at large companies like Google, Meta, and Stripe, and I’ve also worked at early-stage companies like Rainforest and Lorikeet. The common thread through all of these is that I’ve been building early-stage products from the very beginning of my career, even inside the larger firms.

At Google, I was on a small team aiming to change the way local services businesses advertise - think locksmiths and plumbers. We built an entirely new advertising platform for these businesses, launched it as a small beta in the Bay Area, and I led an incubation team that grew that product nationally and then internationally. That’s really where I got the itch to build and incubate new businesses. At Meta, I worked on consumer products and learned something really important about myself, which is that I’m a B2B builder. So I knew I had to go back to B2B - I joined Stripe Capital, Stripe’s small-business embedded lending platform, and that’s where I architected the operations behind it: how does a company like Stripe essentially build and scale a lending business? We incubated Stripe Capital and grew it nationally, then internationally.

What I found after years of building early-stage products inside larger firms was that I had to do it for real, at a real startup. So I joined Rainforest, an embedded payments company - I joined them pre-launch, which is pretty unusual for a business hire, and I led partnerships through the launch, along with product partnerships and international expansion.

And then Lorikeet is where I built out the forward deployed team from scratch. Instead of incubating new businesses, I was incubating a new function. I did the FDE job myself for the first few months, and then I hired and scaled the forward deployed function into a global team across the United States, London, and Sydney. If you ask how my career threads itself together: forward deployed is the role where incubation stopped being what I did on the side and became what I do full time.

A quick note on what Lorikeet does, because it explains why the function exists. Lorikeet builds AI concierges for complex businesses - chat, email, and voice agents for verticals like fintech and health tech. Lorikeet has raised over $50 million from investors like QED, Blackbird, and others. And really, without a deep integration into the customer’s systems, an AI agent is just a dumb chatbot. That integration gap is exactly where the forward deployed team comes in, and it’s why the forward deployed team exists both at Lorikeet and at other AI companies.

How did you get to know the team?

JJ: I was employee number 11 at Lorikeet, and I actually knew the founder through my time at Stripe - Steve Hind is the CEO and co-founder. They had started out in Australia with about ten people, but they were really building for US companies, and they needed someone to build out their US operations and their US team. So I took a big chance and joined as their first US hire, and it was a fantastic experience.

The Role, Defined: FDEs Work in the Opposite Direction

Walk us through a week in the life of an FDE at an AI company - and how the role differs from a sales engineer, a solutions architect, or a consultant, roles people often conflate it with.

JJ: The week in the life of a forward deployed engineer can vary a ton - the role is very, very fungible. But the work falls into two main buckets.

The first is customer-facing work. Forward deployed engineers are working deeply with customers to solve really hairy technical challenges. And besides the actual technical challenges, the good ones are also driving strategy: they’re figuring out not only what are the hairy technical problems that need solving, but what outcomes the customer is actually trying to drive.

The second bucket is largely internally facing. As an FDE, you’ve got a killer vantage point into how users are actually using your product, and a vantage point into the gaps of the product. Part of the job is bringing that feedback and those insights to your product and engineering teams, and then turning the things you’ve been incubating in the field into first-class product experiences - so that future integrations and future customer engagements become even more seamless.

On your question about what separates an FDE from a solutions architect, a sales engineer, or just a consultant: those roles are largely shaping the customer’s problem to fit the product they’re selling. The forward deployed motion flips that notion on its head. The FDEs are working in the opposite direction - the good ones are deeply understanding the user’s problems and then building solutions from there. And in a world in which AI makes products super fungible, that matters more than it ever has before. Forward deployed teams are sitting on the front lines of both the commercial side and the building side, which means they’re really well positioned to incubate what comes next.

Figure 1 - Shaping the problem vs. building the solution: sales engineers, solutions architects, and consultants fit the customer’s problem to the product; FDEs work in the opposite direction.

Before Lorikeet you were leading business functions - forward deployed engineering was new to you too. How did you learn this from the ground up, and end up leading the team?

JJ: If I look at the throughline of my career, there are two components that made me well positioned to build out forward deployed teams. First, throughout all the different positions I’ve had and teams I’ve built, I’ve been on the front lines of understanding customers - taking a design thinking approach to figuring out the core challenges our customers were trying to solve, then organizing teams against that challenge and incubating solutions from there. Second, what I’ve practiced throughout my career is rallying cross-functional teams. Within strategy and operations, product marketing, and more traditional product roles, I’ve had to figure out how to build and rally teams of different functions to understand what our goal is and how to get there.

That made me well positioned to incubate the forward deployed team at Lorikeet. I knew that from a business point of view, we had to figure out how to help our customers build AI agents. To do that, we had to understand, number one, the core business outcomes we were trying to drive; two, I had to figure out how to incubate the team and who to hire, what positions to hire into; and finally, organize the team to achieve the business outcome - which was making our customers successful.

A Role Built for a Superstar: The Three-Muscle Problem

What are the three or four key muscles needed to become a successful FDE - and why does almost nobody have all of them?

JJ: One of the biggest challenges I faced when I was building out the team was exactly this: what type of background do we want to bring in? Especially when I was hiring two years ago, there were very, very few people with the title of forward deployed engineer. What I really found was that I was looking for a superstar. I was looking for three different muscles - which almost no one has all of.

Number one, I needed them to be very strong technically. I needed them to have strong technical aptitude and be able to meet customers anywhere on the technical stack. This wasn’t simply about coming in and throwing problems into Claude and figuring things out async - they had to meet customers and tech stacks wherever they were, whether it was a super legacy stack or a very, very new-age stack. That’s really hard to find.

Second, you need someone with a strong product muscle. We need folks that are able to capture patterns across the integrations and across the customers they’re working with. We don’t simply need folks that are parroting the feature requests their customers are asking for - we need folks who think in a more product light, and think through, okay, what are the actual challenges their customers are trying to solve?

And the third muscle is a commercial muscle. The forward deployed team in many companies is a commercial team sitting at the nexus of product and go-to-market. Folks need to know and understand their customers and their business outcomes - and forward deployed engineers also need to understand their own company’s outcomes that they’re trying to drive. That’s where traditional implementation teams can really get blinders on: simply getting the implementation done and then missing the forest for the trees. Folks need to be able to zoom out through that commercial mindset as well.

The persona you describe sounds like a founder to me, honestly.

JJ: Yep, exactly. And that’s where, as I hired for forward deployed engineers, when resumes crossed my desk from ex-founders - those were the folks where I’d be like, alright, I need to talk to them right away.

Figure 2 - A role built for a superstar: the three muscles almost no one has all of; the closest proxy is an ex-founder.

Hiring Without a Template: Generalists First, Spikes Later

You ran a deliberately experimental hiring approach at Lorikeet - the team was distributed across the US, UK, and Australia. What profiles were you looking for, what worked, and what didn’t?

JJ: This is a core way that I build teams - the way I built teams at Lorikeet and at Google and Meta. I take a design thinking approach and really think through: what are the broad skills that I need, and how do I analyze every person that comes across my desk and every hire that I make, so I can build the most effective team possible.

Tactically, during my time building the forward deployed team at Lorikeet, sequencing mattered quite a bit. We were early - and again, there were very, very few people with a forward deployed role on their resume. As I was incubating the team, I also didn’t have a great idea; I didn’t know exactly what we were going to need. So in terms of sequencing, I first hired generalists. I knew I needed folks that would be able to come in with a unique approach to solving ambiguous problems - folks that really get their energy from getting these ambiguous problems in front of them, working through them without needing a ton of direction. But I also knew I needed a generalist who’d be able to observe patterns across their implementations and then help me build out the team and build out processes.

Then, as I continued to hire, I found a few non-negotiables. Number one, the technical base is non-negotiable. One strong suit of FDEs is when they can go into different customers, troubleshoot with them, and ideate with them on the fly - being able to see a really hairy legacy tech stack and untangle: alright, what is your customer really trying to do? The second finding was that client-facing work is really, really challenging to find in someone who is also very technically savvy.

So as I grew the team: I first hired generalists, and then I was able to evaluate - based on the makeup of my team and the demands our customers were putting on us - where the gaps were, and hire folks that would spike in different areas so I could deploy them effectively. Tactically, that meant later on, after I had a couple of forward deployed engineers, I hired someone with a more pure software engineering background, because I needed someone really deep technically. Then later on, I hired someone who is very strong commercially. You have to think about, as you’re growing the team, how you sequence your hires so that you eventually have a deep bench of largely anything you need.

I’m assuming some learning osmosis happens across those three types of people?

JJ: Exactly - and that’s honestly one of my favorite things about leading teams more broadly: really seeing the development of the forward deployed engineers. They come in and they may be spiking in one area. But what you really have to make sure of is that when you’re hiring these folks, they know what they’re getting into - they know they’re coming into a role that is very ambiguous. It’s not well defined, and it’s going to be a grind for them. What I always tell the folks I work with is that it will be a career-defining experience - I will coach them into someone that will be so strong, during their time at Lorikeet and beyond. I’m a little biased, but forward deployed engineering and AI implementations are just the skills folks need right now, and it’s an incredible opportunity for folks as they’re building their careers.

Let me force a spec on you. If you were hiring an FDE today - years of experience, background? Ex-Palantir is the obvious answer, so let’s remove that.

JJ: I appreciate the question - holding my feet to the fire. What I need is someone with ideally, say, three to five years of experience, number one. Number two, I need someone who has worked in a technical role before, or has a classical technical education. That means I don’t need someone who has worked as a software engineer, but I need someone who has that sort of background and proven aptitude. And number three, I need someone who is a rock star with clients. From a background point of view, that means someone who has done client-facing work - that could be at a tech firm, at a startup, or at a management consulting firm.

Last one on the spec: an MBB consultant with a bit of technical knowledge, or a product specialist or product manager at a software or AI company - which one would you prefer?

JJ: I would probably actually prefer the latter. Again, it depends on the bench of the team that I’ve got, because both backgrounds add something unique that a forward deployed organization would need. But early on, if I had to choose one and I didn’t have either, I would choose the person who has worked in tech previously. It’s funny - my team often jokes, as they ask me questions, “we know you’re going to say it depends, but give us a real answer.”

The Two Anti-Patterns: Custom Dev Shops and Rebadged Solutions Engineers

What do people typically get wrong when building up an FDE team? Mistakes you’ve made or seen peers make?

JJ: First, what I think the industry needs to move into is: before folks build their FDE teams, they need to get really, really clear about the value they’re trying to bring from the teams - take a first-principles approach. I’ve seen a number of folks that just know the FDE role is really hot, and they need FDEs, and they’re just like: alright, we’ve got to hire, and they’ll go into our customers, and they’ll build, and they’ll be FDEs. What I’ve found as I talk to these folks is that they’re falling into a trap where they may be building teams based on anti-patterns.

Specifically, if you don’t build out your FDE function and the supporting functions strategically, what you can end up building is a custom dev shop - where you’re just building custom solutions for every single customer with no common thread. That’s one failure mode. The other failure mode is where folks simply build a solutions engineering team and call it a forward deployed team - where they’re just throwing solutions at their customers and implementing them based on cookie-cutter specs. That isn’t what the industry needs - and when I say the industry, what I really mean is the customers these firms are building for.

It’s almost like it’s fashionable to have FDEs - you substitute them for your traditional solutions engineer, and it’s not actually driving the value it’s supposed to.

JJ: Right. It’s not driving the value, and it’s confusing for all the folks involved - the candidates, the organizations you’re working with that are your customers, and your own organization. By thinking about it very strategically and getting folks that have really done this before, you’re able to supercharge your product development lifecycle through forward deployed engineering.

“Every Customer Is Special” - and You Still Need a Playbook

A lot of companies that started as agent companies now require heavy custom solutioning, and there’s a power law across implementations - some are class acts, some middling. As a CEO you’d want to maximize value for the customer and the company. So: how are you creating and documenting playbooks around implementations? And in an age where AI products evolve this fast, do those playbooks even stay relevant?

JJ: You’re asking the real questions here - these are the challenges for forward deployed teams. If I put it in the words of forward deployed engineers: they say, hey, every customer is special, every implementation is unique - how do you build a playbook around this? And sure, yes, every customer is special. However, what you mentioned is absolutely key: in order to scale, you need playbooks, and you need to figure out, okay, what actual system do you build to make every implementation easier over time?

What I really found is that you’ve first got to map out your customer lifecycle - map out how a customer will typically implement with you. That customer lifecycle map can be very broad at first, but the idea is that you map the lifecycle and then start to identify patterns across implementations; you can work with your team to do that. Once you see the common points across implementations, what I found works well is that folks don’t even need to build a playbook - they should simply have a repository of assets that they use for each phase, that the entire team has access to. So that over time, as folks are hitting the same stages of the customer lifecycle, they have prior art they can look at, review, and build off of. And even better - once you start to have these examples, you can start to observe the trends and actually build out real templates or playbooks.

The challenge, and the trap folks get into, is one of two things. Either they take the approach of “we can’t playbook anything” - and then they can’t make a playbook out of anything, and they’re not going to be able to scale. Or they take the alternative approach where they build something that’s too rigid - and that simply doesn’t work, and people don’t use it. You’ve really got to take that design thinking approach in the middle and simply give folks reference points to use. Then once you identify the trends, you can build it out into a core playbook. And of course - easier said than done. We could talk about this for hours.

Can you give examples of this customer lifecycle?

JJ: With any product launch, there are going to be different phases. There may be a technical scoping phase, where what the forward deployed team is trying to do is simply understand the tech stack their customer is working with - and there are a certain number of discovery questions you’ve got to ask them. So the first step is: write down those questions, and keep that question log in a repository for the rest of your team to use. Then there’s also going to be a phase around commercial discovery - what are the overall business goals your customer is trying to drive? You should have a standard set of questions that you’re asking; put those questions in the repository. Taking this a step further, another example of the cycle would be the actual product launch - the go-live date. You should have a process for what that go-live actually looks like. Of course, the minute details are going to be different for every customer, but there should be a set of principles that are followed across implementations.

And one thing I found very early on - and this has been true across my time working with forward deployed teams, product teams, and commercial teams - is that folks just want to be led. They want someone working with them that has done it before, that has a hypothesis on what needs to be done in order to be successful. So what I am constantly telling my teams is: even if you’re working off of hypotheses, you’ve got to commit and go for it. Either it’ll work great - congrats, great success - or you’ll learn something and you can change something moving on to the next one. But either way, you’ve got to be working from a level of conviction.

Figure 3 - The customer lifecycle as a shared repository: playbooks are what patterns become, between the too-loose and too-rigid traps.

Making Knowledge Sharing Compulsory: Critiques and Cadence

Certain people will be really good at implementations, and there may be a secret sauce very specific to them. I was a management consultant in a previous life, and I loved being the go-to guy for certain projects - that hoarding incentive will exist in these teams too. How did you manage this at Lorikeet, and how do you democratize learnings across the team so the whole organization gets better?

JJ: There are really two parts to that question. Number one is: how do you incentivize folks to share knowledge? As a leader, I like to say that folks need to hear something seven times before they really internalize it, and one of the things I say to my team over and over and over again is: rising tides lift all ships. In the end, we’re a team. And with the forward deployed role, oftentimes there are so many little tasks that folks don’t really like to do - that can be kind of a grind. So the way I really approached this was that I encouraged the team to share these learnings because they were going to help their teammates avoid these traps, these points of grinding that they didn’t enjoy.

And then you start to think through: okay, are folks going to be concerned their jobs are going to be automated away? The reality there is that the real value of the forward deployed team isn’t in turning work. The real value is on the strategic side - working with customers and figuring out what are the overall business outcomes they’re trying to drive, and then how do you connect your company’s product to those outcomes. That is the real value of the forward deployed team, and you’re not going to be able to automate that away right away. So as I was framing this to my team, my framing and incentives really aligned on: hey, let’s work together to get this grinding work out of the way and make it easier for the team.

The second part of your question is really around how you tactically do that. There are going to be teams that are not co-located and may not be in the same time zone. You’ve got to create structures for them to share learnings - and you’ve also got to make it compulsory. You can’t wait for folks to volunteer; rather, you’ve got to create a structure around how folks are sharing learnings about the implementations they’re building. Give them a set of questions that they need to answer and present to the team, and then create some sort of forum. The cadence of the forum can vary - it could be weekly, biweekly, depending on the needs and the scale of your team. But you’ve got to make sure that you’re really getting folks to share their knowledge, and you’ve got to give them the structure to do that effectively.

A comparable in the investment world would be the Monday meeting - interesting deals get discussed, partners across the board ask really hard-hitting questions, and that’s how the team learns. I’m assuming it’s a similar version of that: the team presents an implementation roadmap and others absorb and cross-question.

JJ: Yep - and it’s almost like taking a design thinking approach again; it’s almost like an implementation critique, if you will. There’s a concept of critiques in the design world, and this is not dissimilar. Where I find the real magic moments happen is when folks are presenting on a challenge that they have with a customer, and then someone’s able to chime in and be like - oh wait, I did that last week, I solved that last week. Or folks can say: have you looked at it this way, or have you taken this approach? The fact is that folks oftentimes feel like, as they’re working with customers, they’re in this vacuum where they aren’t able to zoom out and see the forest for the trees. But what I found is that as folks are sharing together, it creates a virtuous cycle: once they see other folks being able to help each other, they’re more willing to share these learnings.

Playbook or Product? Running Everything Through the Incubation Filter

You speak to customers constantly, so you must be receiving a ton of feedback. Some of it will become playbooks for implementation, and some will become updates to the core product itself. How do you differentiate between a product update and a playbook when you’re receiving so much feedback?

JJ: It’s a good question, and the way I think about it is that every customer request takes on the same pattern and the same lifecycle. Number one, you get the customer request and you figure out: okay, what are they really trying to do - what actual problem are they trying to solve here? Oftentimes, it’s not what they’re asking for. Then what the forward deployed team will do is build a solution, either through some sort of process or some sort of playbook. But what they also may do - and again, this is the theme of incubation - is incubate some product experience. The level of technicality of that solution can really vary. Sometimes - I’ve had FDEs that write PRs, and they’re actually able to incubate a small part of the product and make a product shift. Other times, it might be more operational.

But what they’ll do is build that solution for a specific customer - and then what you’ve got to do on your forward deployed team is have a way to, again, track the patterns. Is that solution something that would be applicable to other customers? If so, then what is the level of conviction you have on that feature, and what metrics will it drive? What is the real impact? And based on that question around the impact, that’s when you can really figure out: okay, does it become a first-class product feature, or does it need more time to incubate within the forward deployed team?

So really, it’s not entirely dissimilar from the way traditional product teams prioritize new features. But with the forward deployed team, what we’re able to do is build a more structured and, I think, fungible feedback loop - so that eventually, innovation and customer value can happen much, much faster than the traditional product lifecycle would enable.

Figure 4 - The incubation flywheel: every one-off field solution is a product hypothesis, graduated by conviction and impact.

The Economics: If Your FDE Team Scales Linearly, You’ve Built a Dev Shop

Does the FD team scale linearly with the scale of the company? Imagine Lorikeet were to do $500 million of annualized revenue at some point - will the FD team scale like that, or is there a non-linear correlation here?

JJ: No - I think it will be a non-linear scale across companies. And this is a hot take, and here’s why I think this way: if you’re scaling linearly, then what you’re building is a custom dev shop, or you’re building an implementation shop. That is, I think, a failure mode of some current FDE teams - they’re not actually building for scale. Really, I think the real value of a forward deployed team is that they shorten the product development cycle, and when they do that, they’re able to really scale themselves out of a job - so they’re going to be working on other things.

If you’re finding that your forward deployed teams are scaling linearly - again, it’s going to depend on what you’re investing in as a company in terms of the different functions; there might be a good reason for scaling when you’re early at first. But if you find that you’re continuing to do that, you’ve got to take a hard look at both your forward deployed team and the systems and structures you’re building around it. Because I think that’s something a lot of businesses need to internalize: if they’re building a forward deployed team, they can’t simply build the team and have that team work in a vacuum. They’ve got to make sure they’re willing to modify the operations of their product teams, their engineering teams, their design teams, to really get the most out of this new function that we’re all developing together.

Let me replay it back. Some of the tasks FDEs are doing right now - a bunch of the more cookie-cutter ones get automated by agents, or FDEs leverage agents for them. Understanding the tech spec of the customer can get more automated in future. But marrying your core product value proposition with what the customer wants to achieve in terms of ROI - that remains more human, very high value, and also commercial; decently high value in negotiations too, where historical precedent can be used. That’s the framework in my mind.

JJ: Yep, exactly - and it gets back to this point that forward deployed teams are not implementation teams. The real high-value work they’re doing is not simply being the specialists who know how to turn the knobs and dials. As companies are building out their forward deployment functions, sometimes forward deployed teams are going to need to fill that function - to be able to turn the knobs and dials when the product is still developing. But like you mentioned, what companies really need to realize is that the true value of the forward deployed function is on the strategic side - and in being the connector between customers and the product and engineering teams. Being able to essentially understand: what are the core outcomes your customers are trying to drive, and how do you solve those as effectively as you can?

Figure 5 - Two curves: linear FDE headcount growth past the early stage means a dev shop; the real value scales itself out of the job.

Moving Upstream: FDEs in Presales and the Fate of the Sales Engineer

Do you see FDEs these days also being part of presales conversations? It seems like an important part of the job.

JJ: Oh yeah, absolutely. What I found in my career is that the earlier you can get deployment teams into these conversations, into deals, the better. It builds so much credit to have someone who honestly has the battle scars of implementations, and who can really come in and say: hey, I’ve solved this problem before, in a different way. Ultimately, it builds credibility across the board. As folks are building out their forward deployed teams, you should absolutely have them in presales conversations as well.

A hot-take question then: do you see the role of sales engineers getting diminished in the future? They’re more implementation engineers optimized for sales, whereas FDEs can maneuver many more things - hence very high value.

JJ: I wouldn’t say that - I wouldn’t predict the demise of any one specific role. I think, though, that as the industry changes, every single role is going to have to evolve. That’s going to be the case across sales engineering, solutions engineering, and forward deployment. And like you and I were talking about earlier, the profile of someone who is a forward deployed engineer - that is a very specific, hard profile to find as well. So what we’ll have to figure out as a forward deployment industry is: where is the highest-value place to put these folks in relation to the other roles that already exist? There may be some cases in which you don’t hire a sales engineer - but there are other situations in which you still would want to.

The Forward Deployed Platform: Automating the Grind, Keeping the Strategy

Where are these learnings getting recorded - are you using some kind of software stack, or new AI tools?

JJ: In my past, I’ve used pretty traditional tools - Google Docs, Notion. Though as I build more forward deployed teams in the future, I have a very specific way that I’m thinking teams need to do this. With these new AI tools and products coming along, there are so many ways you can use agents to essentially automate away how this playbook can work for you. As teams are building from scratch, what you’ve got to do is really take advantage of the full suite of tools you could use. I don’t have a specific forward deployed tool to recommend, but right now there are ways you can do this through a combination of recording meetings, Claude Code, Slack.

One thing that I have been building out is what I call the forward deployed platform - which is essentially an intelligence layer. That, I think, is really going to be key to automating away a lot of the work that teams find really challenging. And maybe, as you’re hiring for your forward deployed team, either folks won’t have to hire for more operationally minded folks, or they simply won’t have to - they can build a much leaner team, because a lot of this work in terms of building processes and identifying patterns - agents will be fantastic at this, if you build them correctly.

Any lasting thoughts before we end the conversation?

JJ: These questions were incredibly insightful. I think the core thing folks need to think about - again, I’ll be a broken record, because folks need to hear something seven times before they internalize it - is that as folks think about forward deployed engineering, they really need to keep in mind: okay, what is incubation to their companies, and how do you incubate solutions to really get to solving the challenges that your customers have? But overall, I’ve had a fantastic time building out forward deployed engineers and building out engineering teams, and I’m excited for where the industry is heading.

JJ Refling built and led the global forward deployed engineering team at Lorikeet, and previously incubated new businesses at Google, Stripe (Stripe Capital), and Rainforest. He’s working with builders, founders, and investors on the challenges they’re facing with building forward deployed teams - reach out to him on LinkedIn to collaborate.

Thanks

One of the unique things AI-native product is that the entire product development workflow is inverted. Traditional software is a backward chain: customer says they want X, you scope X, you ship X. Roadmaps are essentially long lists of customer requests with priorities attached. The Founder’s / PM’s job is to listen well and translate accurately.

AI-native product development runs in fairly opposite direction. You start with what the models can actually do today - what’s reliable, what’s flaky, what gets cheaper next quarter - and then you figure out which moments in the customer’s day map onto those capabilities. The customer often can’t tell you what they want because they don’t know what’s now possible.

This only works if you understand the customer’s day at depth. Models do many things; only a few are of consequence for any given workflow. Ranking which capabilities are central vs not imp. is the real job. One of the common feedbacks from customers is that a lot of “AI features” feel bolted on - most of the teams are still in spec-collection mode, optimising the wrong capabilities.

Take Granola - It works because the team understood that for a knowledge professional - VC, founder, CS lead - the meeting itself is just the input. The job is what you do with all your meetings stitched together; decisions get made by cross-referencing 7-8 customer calls, not summarising one.

So they built around this loop, not the meeting: cross-meeting query (”what objections came up in pricing convos last qtr?”); pre-meeting prep that surfaces what someone committed to last time; structured push into Notion / CRM / Linear so context lands where decisions actually happen. The summary is the least interesting part. For a growing cohort of professionals, it has quietly become the central nervous system of knowledge work. Had they gone ahead w/ spec collection approach by just asking customers (instead of observing them) - they would have highly likely remained - an “AI meeting notes - summaries + action items.” company.

Legora’s founding team physically sat inside a Stockholm law firm to learn the data model first-hand. They didn’t force diligence into a chat UI - they built a 10K-doc parallel matrix because that’s the actual shape of the job.

This feels less like “what feature do you want?” and more like “if I sit inside your week, which model capabilities actually compound, and which are decoration?”. I imagine founders / PMs will still need to continue to listen to customers but use their feedback to guide their own instincts.

Disruption (according to Christensen's seminal paper) occurs when incumbent companies ignore new technologies that don't serve the needs of their customers or fit within their existing business models. However, as the new technology, which excels on completely different attributes than the incumbent's product, continues to mature, it eventually takes over the market. Another core feature of disruption is that the initial set of adoption is driven by cost-led advantages with a "good enough" value proposition that incumbents don’t about. However, in (enterprise) software, the calculus of cost and experience changes. How do you provide something (i.e., software) at cheaper prices that is already getting cheaper (and better) - in fact, more so with AI, which has led to an exponential drop-off in complexity to launch new (and high-end) products - solutions such as Cursor, Codeium, GitHub Copilot, etc. have greatly democratized engineering. Most of the needle-moving disruptions in the last 5-10 years have been in hardware (arguably in sectors ridden by bureaucracy).

1. SpaceX vs NASA: SpaceX made orbital delivery 10x cheaper through vertical integration, reusability, and software-defined control systems. It vertically integrated everything - from rocket engines to software - and prioritized reusability from day one. Meanwhile, NASA was operating with bloated supply chains, fixed-cost contracts, and slow bureaucratic processes.

2. BYD vs Legacy Car Companies: BYD bet early and aggressively on battery innovation, vertically integrating both cell production and EV assembly while legacy automakers outsourced both. This gave them control over cost, quality, and supply chain resilience - especially during lithium price surges. While GM and BMW hesitated or leaned on hybrids, BYD went all-in on electric and built products tailored for domestic demand in China.

3. Anduril vs Defence Primes: Current theaters of war are living proof that a higher number of cheaper autonomous systems are better suited than large military platforms (such as battle tanks, howitzers, and others). Anduril had this foresight and built a software-first autonomy stack for surveillance, targeting, and decision making - with rapid iteration and off-the-shelf hardware. Traditional primes like Lockheed and Raytheon were optimized for multi-decade programs and cost-plus contracts, not speed or autonomy. Anduril's edge came from productizing AI at the tactical edge - in drones, towers, and battlefield systems - while incumbents were stuck integrating subcontractors. It created a full-stack autonomy company in a market built around fragmentation.

The far and few that happened in software had one common theme - most of these companies 1) fought the battles on their own terms 2) Took advantages of platform shifts and 3) When urgency hit (data scale, AI wave, COVID), the disruptors were already ready with the right architecture and user experience.

1. Snowflake took advantage of Cloud adoption and built a multi-cluster shared data architecture optimized for AWS, Azure, and GCP. While Oracle and others assumed deep tuning and control were must-haves, they didn't see simplicity and elasticity (separation of compute and storage) as serious value props for real enterprises. Snowflake proved that usability, elasticity, and consumption-based pricing were not just "nice-to-haves" - they became the new standard, and the rest is history. In this case, the platform shift was → 1) Cloud adoption and new vector was 2) Elasticity offered by Snowflake.

2. Databricks unified AI + Analytics + Data Engineering in a single "lakehouse" architecture - combining the reliability of data warehouses with the flexibility of data lakes. Hadoop vendors focused on storage and batch ETL, underestimating the rise of machine learning pipelines and real-time analytics. Traditional warehouses (Teradata, Oracle) were focused on BI/reporting use cases, not predictive workloads. Databricks positioned itself at the intersection of ML workflows and cloud-scale data processing, years before incumbents saw ML as a separate workload, and became the default platform for AI-native data teams, riding the ML and generative AI wave with real architectural edge. In this case, the platform shift was → 1) ML/AI Adoption and new vector was 2) Lakehouse architecture offered by Databricks.

3. Zoom was built from scratch with a video-first architecture, optimized for low latency and high reliability even in low-bandwidth environments. It focused obsessively on ease of use: one-click join, no installs, clean UI - when incumbents had bloated desktop apps and clunky integrations. Cisco and Microsoft assumed video was a secondary feature behind email or voice. Video was bundled, not prioritized. They focused on selling to IT departments, not to end users, and ignored usability complaints. COVID hit, and Zoom's 10x better exp. made it the de facto choice - not just for companies, but for schools, courts, and families. What was dismissed as "nice UX" became mission-critical - and a $100B+ market cap company was built. In this case, the platform shift was → 1) COVID and new vector was 2) End User focused application, that was easy to scale from Day 0.

One core thing to note about the AI wave is that applications need to think deeply about "fighting on their own terms." Founders should assume that incumbents will take advantage of platform shifts and are likely to move faster than we've given them credit for. There are some interesting approaches founders have taken to insert themselves in the enterprise stack.

AI Native Companies are going after IT budgets (and not software budgets)

AI is redrawing the boundaries of IT budgets. If you break it down, ~50% of IT budgets today go to headcount, ~25% to software, ~15% to hardware, and ~10% to consultants. More importantly, only ~15% of that is truly "up for grabs" on an annual basis.

Best-in-class AI companies aren't just targeting the ~25% software slice. They're going after the entire 60%+ tied to human labor and services, embedding themselves into workflows that previously required headcount, consultants, or internal dev teams - the "shadow services" budget. This is a shift from selling software to selling outcomes. The ratio of software to services spend is ~1:11

AI enables companies to replace full-service teams with scalable automation, driving down the unit cost of outcomes. This unlocks a massive budget shift and valuation shift. Services companies typically trade at 3–6x revenue, while software companies trade at 10–20x. Reclassifying services into software doesn’t just expand the pie - it re-rates the revenue itself, compounding equity value creation.

Conversely, if you're building a pure software tool without touching labor-intensive workflows, you risk selling Point AI tools - like summarization or UI assistants - are often perceived as “nice-to-have,” and rapidly commoditize as LLMs evolve. On top of this there is added complexity of CIOs preferring to wait rather than commit early in a market moving this fast - specially when enterprise platforms are great at bundling stuff.

A good case study on this is Norm AI - which is building compliance AI agents for highly regulated industries such as Finance and Healthcare. Typically in regulatory-sensitive organizations such as banks, there are internal compliance teams that regularly sift through the impact of new regulations and their impact on internal processes. There are internal tooling/software solutions that monitor if all processes are followed per mandate. Additionally, there's a huge budget allocated to audit firms such as the Big 4 to audit these processes partially. Even smaller banks, which lack large compliance teams, particularly feel the pressure - compliance costs can reach ~2% of their total assets - majority of these costs (~75%) stem from labor. Norm essentially automates the overall workflows targeting the overall compliance spend - i.e., going after 1) Internal Compliance Team, 2) Software Tooling and 3) Audit firms. Net-net: The real action is where AI collapses headcount + software into a single system. This isn't about replacing software. It's about replacing workflows. And often, teams.

Unlocking Net New Use Cases

Too many companies constrict AI agentic application use cases to those currently served by existing software applications - just faster and cheaper. But in practice, the biggest agentic opportunities are likely to be in areas that are currently unexplored - where agents get deployed to solve net new problems. We treat modern enterprise setups as an optimized engine, yet in reality there are significant inefficiencies: projects stall, headcount freezes, skill gaps are very much the norm, budgets hit the ceilings, and customer pipelines can be erratic. In a lot of these choke points - talent, resources, and capital run thin. Aaron Levie at Box articulates this really well

One of the key features of the agentic approach is that it provides an elastic cost model for knowledge work that wasn't earlier possible. AI agents give organizations the ability to augment and automate work in areas they weren't able to spend on previously or in areas where they were supremely resource-constrained. Most of these tasks are likely characterized by a "cold start" problem, which requires some form of focused investment that gets deprioritized. Some of the very basic manifestations of this I have heard across enterprises are:

1. Ability to A/B test at scale that wasn't possible before: e.g., a performance marketing team at a portco is churning out 1000s of short-form AI-generated video content to A/B test quickly to identify needle-moving creatives, improving its targeting significantly.

2. Automate loss-leading services at scale to get larger contracts: Harvey is enabling legal firms to take on low-margin projects in private equity more efficiently (such as through side letter compliance work). Traditionally, law firms would do these lower-end tasks at a loss to build relationships with private equity firms, hoping to eventually land larger M&A deals. Harvey helps by building software that allows firms to: a) Automate side letter compliance work b) Charge a flat fee for these previously unprofitable tasks c) Reduce the time and cost of completing these projects and d) Create a more attractive entry point for securing future high-value work from private equity clients. By making these low-margin projects more economically viable, Harvey helps law firms expand their market share and build relationships with potential high-value clients more effectively and profitably.

Circumventing to End User if possible and then tackle the Enterprise

One of the most interesting shifts in the AI-native era is how companies are bypassing traditional enterprise sales cycles and going straight to end users - developers, nurses, operations staff—through self-serve, freemium, or viral entry points. This GTM motion isn't new, but its speed and scalability in the AI context is unprecedented. The most obvious example is Vibe Coding. Companies like Cursor, which now does ~$200M in ARR, have quickly become the go-to IDEs for developers across enterprises.

A more unusual vertical where this is playing out is healthcare. A very interesting case study is that of Open Evidence - an AI-powered medical platform designed to help doctors make better clinical decisions by providing quick access to the latest peer-reviewed medical research (medical research 2Xs every 73 days). It allows doctors to quickly find specialized medical information for complex or rare patient cases, helping them make more informed treatment decisions by accessing the latest medical research that might not be easily found through traditional search methods. Open Evidence essentially scaled from 0 users about a year ago to now serving ~25% of active physicians in the United States. They rejected traditional top-down healthcare enterprise sales approach and adopted a consumer-centric, direct-to-user approach. Instead of pursuing complex institutional approvals, which are mired in bureaucratic complexity, taking up to 2-3 years to obtain final approvals, they released a free, high-value app on the App Store. The strategy focused on creating an exceptionally useful product that would naturally spread through word-of-mouth recommendations.

The direct-to-doctor approach also created a powerful technology flywheel for Open Evidence. Open Evidence had trained its models on publicly available medical literature, but there are many top peer-reviewed medical journals that aren't available publicly (such as The New England Journal of Medicine - NEJM). As doctors began using the app, senior editorial board members of these journals (such as NEJM) became users themselves. These high-profile medical professionals experienced the value of the platform firsthand, which was crucial to the subsequent partnership. Unlike traditional approaches where companies would pitch partnerships through lengthy negotiations, NEJM's senior leadership reached out to Open Evidence because they were impressed by the product. These journals wanted their content to be shown to users. This organic, bottom-up approach led to a strategic content partnership that would have been typically very difficult through traditional approaches. The partnership benefits were mutual: Open Evidence gained access to (exclusive) full-text NEJM content, NEJM received millions of visits to their journal pages through Open Evidence Platform, doctors got access to cutting-edge medical research, and the platform drove traffic to specific, deep methodological sections of medical journals that might otherwise go unread

Additionally, many AI-native healthcare companies such as Scribes (Freed, Nabla) are inserting themselves through PLG-led motion. In many cases, companies are executing unique PLG-led Enterprise Sales motions (going after entire health systems), which has not been the norm in this industry. Clinicians start using these scribes to tackle burnout where adoption of these tools is slow at the healthcare system level (Note: Clinicians spend 1–2 hours/day writing notes during and after patient consults instead of spending time with patients). Once a critical mass of clinicians start using these AI products, the companies engage in enterprise sales motions, leveraging their power users to push the product into the system.

While incumbents focus on compliance, governance, and integration, AI natives are building utility-first products that earn adoption in weeks - not quarters. It’s a fundamentally different motion - and it’s working.

If there are other interesting case studies - would love to hear!

Klarna is a fintech company that started with, providing buy now, pay later (BNPL) services and has now become one of the largest commerce networks, providing consumer shopping experiences and facilitating payments. It has ~93M active consumers and >675K merchants across 26 countries, and drove >$105B of GMV in 2024.

There are 2 key revenue drivers of Klarna: Transaction and Services Revenue (76% of total revenue) and revenue earned through interest (24% of total revenue)

1. Key sources of Transaction and Services Revenue

   1. Merchant Revenue (75% of transaction revenue) – Fees from merchants for processing payments, interchange revenue, and dispute resolution - primarily for BNPL use case

   2. Advertising Revenue (8%) – Paid placements, search ads, and affiliate marketing within Klarna’s ecosystem.

   3. Consumer Service Revenue (16%) – Fees charged to consumers (e.g., reminder fees for late payments).

2. Interest Income: Earned from Fair Financing products (loans >3 months) and payment flexibility features like “Snooze”.

Merchants pay Klarna because of: 1) Increased Sales - Higher checkout conversion rates 2) More Customers – Klarna brings shoppers to their platform 3) Lower Risk – Klarna covers fraud and credit risk 4) Marketing Power – Merchants gain access to Klarna's advertising ecosystem. In 2024, Pay Later represented ~76% of the total transactions (79% of GMV), Pay in Full (after excluding transactions processed through KCO unbranded channels) ~23% (~16% in GMV). The ~675K merchants include some of the largest global brands - on average, ~44% of the top 100 merchants in each of the major markets that Klarna serves, which include the United States, the U.K., the Nordics, Germany, Austria, Belgium, Spain, France, Italy, the Netherlands and Switzerland (based on data from eCommDB and Digital Commerce 360), used Klarna in December 2024 to facilitate payments.

In 2024, average balance per active Klarna consumer was $87 (compared to an average balance per credit card of ~US$6,730 in the United States in 2024, according to Experian), and average loan duration was ~40 days (compared to a typical loan duration in 2023 of more than five years at a typical Nordic bank, according to publicly available information, and an average of ~2.9 years of a typical U.S. personal bank loan, according to the U.S. Federal Reserve).

Klarna has - 1) Compounding Network Effects (More consumers → More Merchants —> which in turn leads to better consumer experience) and 2) Trusted Brand and Global distribution - some of the most interesting brands include Airbnb, H&M, Nike, Uber and eBay.

Klarna’s AI Story

Klarna has been an early and leading adopter of AI - It has crafted a good story around AI capabilities built on a unique proprietary dataset with SKU-level insights - including ~2.5B data points collected in 2024 and ~5.2B transactions processed through their network. This data provides a comprehensive view of 1) Customer Intent (such as time spent browsing products, engagement with deals or campaigns and brand affinity) 2) Purchase History (SKU-level purchase history, brand loyalty, payment method, price sensitivity) and 3) Consumer Profile (such as purchase power, credit score). It makes you think how companies with large distribution can easily build data-led defensible moats that new AI-native companies don't have the luxury of. Klarna claims ~96% of their employee base uses generative AI capabilities. It’s famously considered engineering 1st organisation with ~48% of the entire org - either in engineering or data science functions.

Consumer-Facing AI Products

1. From a user experience perspective - Klarna uses AI to provide - personalized shopping feeds, dynamic brand recommendations, and strategic product suggestions (features include personalized recommendations, price drop alerts, cashback offers, product search, and location-based recommendations), boosting consumer engagement and ultimately transactions. Additionally, users also have access to AI assistant designed to help with shopping and payment experiences. From product recommendations to managing expenses, the assistant guides the consumer throughout the entire commerce journey. There is strong data trust relationship with users - consumers and merchants share data with Klarna because it directly improves their experience on the platform

2. One of the core drivers of Klarna’s revenue is ARPAC (average revenue per active customer) - purchase frequency is the largest driver of ARPAC; typically consumers transact more frequently the longer they are on network; there is also a positive correlation between the number of merchants in the geography and purchase frequency. For newer cohorts to increase the pace of purchases, AI-led recommendations is likely going to matter quite a lot, given AI-powered shopping feeds drive higher conversion rates and accelerate revenue growth through increased transaction volume. Klarna’s location based recommendation systems had an 8x higher click-through rate than a top competitor and generated ~3.6B drive-to-store offer impressions. Consumers using Klarna's location services made ~53% more purchases per month than those who had them disabled

3. Additionally, Merchants start with Klarna’s payment services and expand into advertising and other solutions. AI led recommendations also have key role to play here as well.

AI driving Underwriting Capabilities: Machine learning enhances Klarna's proprietary underwriting model, enabling faster and more accurate high-frequency, large-scale, and real-time credit decisions. Klarna operates ML-enabled high-frequency, large-scale, real-time underwriting process across a standardized set of products

1. It has built underwriting capabilities based on access to a proprietary data set, including first- and third-party data, and a unique credit underwriting process that becomes more accurate as it scales.

2. The ML models analyze growing amounts of data (including leveraging purchase behavior from an average of ~2.9M transactions/day made by ~93M active Klarna consumers). As Klarna processes more transactions, credit models continuously improve. The predictive accuracy of the models is demonstrated by a notable improvement in their Gini score over time (higher is better). In 2024, their consumer credit losses represented 0.47% of total GMV (which is low and partially also driven by nature of short term loans).

AI Led Efficiencies within the business

Total operating expenses as a percentage of revenue declined from ~151% in 2022 to ~104% in 2024 (implying ~47%pp improvement), even as GMV increased 27% (28% on a like-for-like basis) in the same period (operating expenses include processing and servicing costs, consumer credit losses, funding costs, technology and product development expenses, sales and marketing expenses, customer service and operations expenses, and general and administrative expenses.). Klarna saw ~48% annual revenue growth in 2024 compared with 2022, while operating expenses increased by only 2% for the same period. The average annual revenue per employee at period end has increased from ~US$344K in 2022 to ~$821K in 2024. Not all of it is driven by AI, in fact the majority of the operating leverage is driven by economies of scale

1. Technology and product development expenses as a percentage of revenue decreased by 7 percentage points,

2. Sales and marketing expenses decreased by 16 percentage points,

3. Customer service and operations expenses decreased by 8 percentage points,

4. G&A expenses decreased by 7 percentage points.

However, Klarna's S1 does paint an interesting picture of AI adoption in enterprises - although AI is one of the core pillars of their story, the direct tangible AI-led savings that are explicitly discussed in S1 is ~US$85-90M on a cost base of US$2.9B, representing ~3% of the overall spending. However, the AI-led efficiencies are likely more - it's difficult to estimate the productivity enhancements driven by the internal tools and consumer-facing AI products. One of my takeaways from this is that AI adoption in enterprises is still in its infancy and will likely grow significantly. If we dig in deep, it does tell us more about which areas AI is impacting more than others.

Customer Support is by far impacted the most. The AI assistant (powered by OpenAI) is handling ~62% of customer service chats i.e. ~20M conversations), doing the work equivalent of >800 FTEs (estimated based on the average monthly reduction in chat and telephone conversations handled by full-time agents following the launch of AI assistant), and in 2024 delivered ~$39 million in cost savings. This translates into ~US$50K cost savings per FTE / ~US$2 per Conversation. I know per-seat pricing is likely downhill in the AI era, but I am amazed how many founders price their product dirt cheap compared to the cost savings you see here.

1. AI-handled consumer chats rank on par with human agents in consumer satisfaction and demonstrate higher accuracy in errand resolution.

2. AI-handled consumer chat resolutions averaged 2 minutes, compared to the 12-minute average for human agents in 2024.

3. Repeat inquiries dropped by 25% between December 2023 and January 2024.

4. Customer support agents also use AI-chat interface that explains credit denial decisions. This tool has been used >987K times internally in the last 18 months

Klarna's engineers use an AI-assisted case log classification tool that organizes documents and categorizes >1M monthly chat conversations for doing RCA analysis. In August 2024, it had a 96% root cause clarification accuracy. By automating a critical and time-consuming task, this tool in 2024 delivered ~US$4.8 million in cost savings. ~85% of the engineers use AI copilot that can create and review code

Klarna also operates a Glean like product - an internal knowledge chatbot powered by AI called Kiki that helps employees find information across internal systems, which boosts productivity, compliance, discovery, and collaboration.

Sales and marketing costs declined to $328M in 2024 from $531M in 2022, driven by AI implementation, cost efficiencies, and process centralization. They particularly call out reduction in spending on marketing agencies by 75%, from $33M to $8M. Multiple other areas of cost savings that leverage AI include reducing image production costs (despite running more campaigns), copywriting expenses, and spending on external marketing suppliers, such as translation, production, CRM and social agencies, which led to ~$US7M in cost savings in the 1st quarter of 2024, of which ~37% can be attributable to AI; Klarna also canceled contracts with >1.4K suppliers following AI adoption/standardization and claims to have saved >$2.5 million in the first quarter of 2024 from AI-powered marketing.

I will also be doing this for a few more companies!

Hey Philipp, thank you so much for taking the time to speak with us. We really appreciate it. Just for the listeners, I want to mention that Philip leads product initiatives at Honeycomb. He is responsible for ensuring a high-quality developer experience. He also leads their open telemetry focus and their open source initiatives. He recently gained fame when he published his three-part essay on the challenges of building LLM applications that are often overlooked in production environments. I thought it would be a good idea to have him as a guest on this podcast. One of the motivations for having him on the podcast is that personally, I have encountered over 120 companies this year in the AI application infrastructure space. Everyone talks about their success stories, but no one talks about the challenges when things get tough. So, welcome Phillip. Thank you for joining us on the podcast. Let's start by discussing how you ended up at Honeycomb and what your background is?

Yeah, sure. Thanks for having me. So, back in 2015, I finished college and joined Microsoft in Seattle. I worked on the dotnet team, and we embarked on a journey to make dotnet fully cross-platform and open source. We wanted to modernize it for where software development was happening. I spent about six years at Microsoft, working on the compiler and compiler tooling stack, as well as the runtime itself. It was a mix of different tasks, but we successfully made dotnet a more modern platform, not limited to Windows. After about six years, I decided to look for something different. I had been working on compilers for a long time, so I wanted to try something new. That's when I joined Honeycomb, working on observability. It was a different set of problems, but still unsolved. My focus was on the open telemetry project, which aimed to provide a standard protocol for generating and owning data across applications. This initiative aimed to give development teams more freedom and avoid vendor lock-in. I spent about two years working on stabilizing and adding features to open telemetry, collaborating with the community and within Honeycomb. Currently, I'm still involved in that work, but now I'm focusing more on AI. In the fall of 2022, just before ChatGPT was released, I wrote an internal document based on my observations during my time at Honeycomb. I noticed the different challenges users faced with our products, and many of these challenges didn't have a clear, correct answer. The right behavior often fell on a spectrum, and it was up to us to guide users in the right direction. I emphasized that machine learning, being probabilistic in nature, could provide solutions in areas where it made sense. By effectively incorporating machine learning into our product, we could offer more assistance to users compared to the previous state of Honeycomb, where users either knew how to use the product or didn't. As everyone knows, ChatGPT took the world by storm when it was released. While large language models existed before, it was in November of last year that people started to realize their general-purpose utility. Then, earlier this year in 2023, OpenAI significantly reduced their prices, making it more economical for us to build on top of this technology. We didn't have machine learning engineers to build our own custom models, so using an off-the-shelf solution was the way to go. The cost barrier was no longer an issue, allowing us to model worst-case costs and stay within our budget.

Just one question on that one, right? Like, I mean, I'm assuming that you get tons of queries within the honeycomb system. So, how did you even build your comfort around, you know, like the estimation of costs? Firstly, maybe explain to listeners in general, like what is the quantum of queries that you generally get? And the second part is, what were your cost considerations considering? A lot of people talk about inference costs in general, right?

Yeah, I would say, largely, especially if you're using something like OpenAI, Anthropic, or Cohere, the prices are so ridiculously low that you would be silly not to try to build products with them. Oh, you can certainly try to fine-tune your own LLM later on and try to reduce inference costs, but the cost of that will be much higher than just using one of these APIs for almost any task. The way we approached this was by deciding to build one single thing: a natural language querying assistant. We knew from user research that many people came in knowing what they wanted to query from their data but didn't know how to express it in our UI. They asked if they could just type something in a box and hit Enter, and we would give them the results. It didn't have to be perfect; it just had to help them figure out how to eventually get what they wanted. That's exactly what our algorithms are meant for. So we took that and said, "Alright, cool. We are building a way to create queries." We looked at our total query volume for people who manually executed queries, and at the time, there were about a million queries per month, and the number was growing. We used that as our baseline, knowing that there probably wouldn't be a million executions of this LLM, but it was the upper bound. We then calculated the input and output tokens for our current prompt and examined the math. At the time, it worked out to be about $100,000 per year using GPT 3.5. We concluded that $100,000 a year is equivalent to two midsize conference sponsorships.

Like, that's quite low because you're at the end of the day able to handle, I would say, 12 million queries with $100,000. Yeah, this is a surprise to me. Okay, maybe let's take one step back. Honeycomb is an APM platform, broadly speaking. If I were to classify it, right. So, how does the LLM workflow typically work? Can you explain that? Because I'm assuming you're not directly querying the schemas or your LLMs are not directly interacting with the customer databases. So, can you just explain to us how the overall workflow works here?

Yeah. So what Honeycomb has today is the main page that people are on, which we call the query page or query builder. It's like a visual programming language where you click around in different boxes. There's a calculations box, a filters box, a grouping box, and the values that you plug in there are columns from the schema of the specific data set you're querying against. To add an LLM to the mix, we decided to translate someone's input. We started by taking the full schema, which is just a list of columns, and parameterizing it directly inside the prompt. In the prompt, there are a couple of things that explain the structure of a Honeycomb query. The goal is for the LLM to produce a JSON object with a specific structure that follows our query structure rules. When someone hits "run query," that JSON object is submitted to a Kafka cluster, which then runs it against our query engine. The query engine retrieves the results from the streams and sends them back to the UI, where you can see your query results. So our goal was to get a large language model to produce the JSON objects we're interested in. We have everything else we need to plug things in.

The way I understand this is that the output of LLM is a JSON object that is essentially part of the query. Then, you receive the results back and they flow back to the UI. One interesting point you mentioned in the essay is the difficulty of dealing with context windows. It seems that you're including a smaller set of schema along with the query, as you described in the process. Help us understand how you define the maximum limit when it comes to context, particularly the context window. How do you determine the important boundary points? Because this seems like a very complex process to me, honestly, considering the large amount of data involved.

Yeah, it's definitely difficult. And this is something that is going to depend on the large language model that you're using, okay? And the kind of data that you're putting in there because we found two things. First of all, even though we built with OpenAI, which is what we're currently using in production, we were also experimenting with the Claude at the same time. And Claude was performing just as well as OpenAI, but it was a little different. It was better in some ways and worse in others. We found that the models behaved differently when we tried to input the entire schema that we're creating into our prompts. For both models, they would recall information differently. Sometimes it felt like GPT-3.5 was better, sometimes it felt like the Claude was better. It was kind of a mixed result, but they were definitely different in terms of what they seemed to remember. We also found that once the Claude released their 100k context window, it helped with few of our customers who have extremely large schemas with a huge list of columns. For most of these models, we wouldn't be able to fit everything in there. So we used embeddings to extract a relevant subset, but that created its own set of problems that I can explain further if needed. But basically, we found that if we put a subset into GPT 3.5, it performs very well. Once, Claude 100k came out and we're like, oh, wow, the context when a problem is solved like this is more than big enough to fit where we need to fit, let's just shove the whole thing in there. Nope, did not solve the problem. Just because you have a large context window does not mean that the large language model is able to recall everything from the context window. And in fact, around that time, a paper got published called "Lost in the Middle". Okay. And the idea is that in the beginning of a context window, there tends to be strong recall. As you get further along, it just loses memory of everything up until the last quarter or last 10th or something, depending on size, basically the last bit of your prompt, that's when it starts to remember stuff. And so if you have anything important in the middle, it's going to be kind of a crapshoot, and that's exactly what we ran into where this 100k context window was able to accept full schemas that we were parameterizing inside of it, but it was producing a poor experience compared to if we use an embedding to pull out only 50 columns, and then use those 50 columns instead, the end result was far better. And so that's where we found that even if you have a large context window, there's actually a sweet spot in the amount of context that you can stuff into a prompt for a large language model. And if you have too little data, then you might be missing important stuff that could be relevant to what somebody's asking for. Because, you know, you're embedding the model and running cosine similarity on it and stuff. Yeah, that may miss certain things that are important and that somebody may need. Yeah. There are lots of examples. We've encountered situations where selecting the top 10 is actually not enough, but we found that selecting the top 50 columns from a schema tends to be the best approach. And 50 also seems to be a good number for GPT 3.5. Okay, this allows us to differentiate between these values and intelligently select the right ones, without getting lost and forgetting certain things that shouldn't be forgotten. We arrived at this conclusion through a lot of experimentation and constant iteration when we were live.

Okay, the maximum limit is 50 columns, as you have seen. I wanted to get a sense of what happens if a user intends to query a large amounts of data, which may happen only about 1% of the time. How have you solved for those edge cases?

So honeycomb itself has a limit on the number of columns that you can include in a given query, okay. And we also looked at the distribution of how many columns exist in the average squares that most people typically have, I would say around five, once you go up to 10 columns that exist within the entire query specification? Yeah, that's like a very small number of people who actually have that much stuff. Typically, what people do is they will ask, like, hey, what's slow in my services, or, Oh, I think there's an error. Can you show me what the error is? And then they'll iterate and say, boom, what about only for this endpoint, or, you know, only show me this stuff for endpoint equals blah, blah, blah, or whatever? So typically, the large language model tends to select about three to five columns for those kinds of questions, which tends to be very good. The other thing is, even when people ask much more nuanced questions, what we did is we were very intentional with how we prompted the model to try to get it to produce what we call a broad query. Yeah. So to get into why this matters. When you're investigating something that you don't know exactly why it's wrong, you just know that something is wrong and it might be wrong somewhere. You may not necessarily know where or why. When you're querying your data, you have to start with a broad selection of data and iteratively narrow it down. So you narrow it down to a more specific endpoint or a more specific subsystem or the way that we're calling the database or something like that. But you may not know that upfront, that you have to go to that specific spot. And so the model is really good at producing a broad query that makes it very easy to say, oops, I saw the results. For eg. that checkout service is the one that has the higher latency. We filter the checkout service, hit enter, and then it filters only to that guy and says, Okay, now group it by all the standard names, and it'll do that, but you can also click in the UI to accomplish the same thing. But the model is extremely good at giving you a broad query and then letting you refine it step by step. That was by design, that we wanted to do that, and that's why we spent a ton of time iterating in production with users because we had a hypothesis that this is what people would want to do. That hypothesis bore out in real user behavior. Like this is literally what they were doing. We interviewed some of them and they're like yes, I want to just give me broad strokes, show me everything that could possibly be wrong, and I'm gonna iterate my way towards narrowing it down somewhere. And the model is very good at that. So we ended up not running into a situation where people could potentially be querying for 100 different columns in their data because usually, it's just the thing that they're asking for. It tends to only hold relevancy in a small handful of columns that exist in their schemas. Yeah.

You know, there's a growing belief where people are constantly saying that the performance of your open-source models is going to match with, say, commercial models like the ones you spoke about (GPT 3.5 or GPT 4). I think you allude to the fact that OpenAI is still the leader. So I just wanted to understand, what is the reason behind this contrarian view? That is one. And typically, if someone wants to build an LLM-based application, how would you rank these LLM companies in general?

Um, so my position is that I've been working in open source my entire career. Philosophically, I want open models to succeed. They're not actually open source, but that's a whole separate discussion. Let's call them source open models. I want to live in a world where those are the primary things that people interact with. What we found when we used them is that they were horrible compared to GPT 3.5 or Claude. Some of that has changed now with the llama two variants, and I know that a couple of other providers are able to start fine-tuning some of these for very specific use cases. Basically, the open models are not going to be competitive in terms of being general purpose. They are competitive when heavily fine-tuned for a specific task. The problem is that effectively fine-tuning something to be really good at a specific task is really difficult. It becomes a data quality problem, and organizations are generally horrible at data quality. They don't know how to run this stuff, and there aren't enough ML engineers to actually service the bulk of our industry right now.

Most software engineering teams are likely to spend a ton of time trying to fine-tune a model, only to have it be maybe as good as 50% or something for their set of tasks. That's not true for every organization, but I think most organizations that are experiencing success with this approach right now are extreme outliers. Most organizations either work with partners like Open AI / Cohere, who do the hard work for them. In the outlier cases where they are leveraging open source LLMs, they already have data science and ML engineering teams to guide the process.

One amazing thing about closed LLMs is that you don't need an ML engineer or data scientist to build a useful product with them. However, if the proposition is that you need to effectively fine-tune something over time, not just as a one-off operation. That's a lot to ask for software engineers who just want to interact with an API, especially when you compare it with incredibly cheap models from providers like OpenAI. It only takes an afternoon to get something working with GPT 3.5 without having to train or figure out good representative data for fine-tuning.

OpenAI itself has built up an incredible moat. The moat has nothing to do with technology; it has to do with developer credibility and understanding across an ecosystem. The majority of engineers building with LLM today are using OpenAI. They know the quirks and how to work around them, build prompts, and create tools. It's a high bar for the open world to cross.

I'm not saying they can't do it. Open source and fully open source models might take over in the end, and that would be cool. But the bar is too high to cross compared to what closed providers are offering right now. I haven't seen anything that changes that. Capabilities have gotten better, so very motivated teams can certainly do it if they have a business or ideological reason not to use something like OpenAI. However, closed providers also meet enterprise readiness requirements, like SOC2 compliance, which is important for handling sensitive data.

They check all the boxes as far as enterprise readiness is concerned.

In fact, I believe OpenAI is highly enterprise compliant, if I'm not mistaken, right?

Yeah. OpenAI is quite compliant, Anthropic as well, too, especially now that they're in bedrock. And so if you're deployed on AWS, and your customers require certain data locality, that kind of stuff is available for you. So the gap has been closed on the business requirements side of things. As for your second question about stack ranking, I would say OpenAI is number one. The reason why is that it comes down to community again, right? If you run into a problem, you can go to the lazy web on Twitter and say, "Hey, we're trying to build XYZ stuff with GPT 3.5 or GPT 4. Here's what we're trying to do. Does anybody have any tips?" Chances are somebody has already done something like that and they would be able to help you out. If you try to do that with Anthropic, you'll probably find fewer people who have done that and, in fact, the model might operate very, very similarly to GPT, but there will be less of a community and a developer ecosystem built up around their APIs and other cool stuff. From there, it drops off precipitously with options like cohere or things like A21, which have decent tech. However, the developer community and set of tools around them heavily favor OpenAI. I think unless you have a business reason not to start with OpenAI, you should certainly do that. And you should also spend some time figuring out a way to use other models as well, right? Don't be stuck with a single provider. Experiment with Claude, experiment with others. But if you want to get something going, I think OpenAI is by far the best route to go.

Yeah, interesting, because I think recently I heard that one of the approaches people are taking is to try building the product first with a third-party API service, such as OpenAI, to try and achieve Product-Market Fit (PMF), and then figure out whether you want to continue with OpenAI or not. I think that's actually the go-to preference for some of the companies that are building AI-led applications. Interesting. Let's switch gears a bit. You wrote in the essay, and let me actually read it out verbatim. I think you mentioned "something to show versus nothing to show". And you follow this interesting principle that you should always try to show something as an output rather than nothing. The context for listeners here is that the query inputs can be quite poor from the user's perspective. You can get a lot of poor queries, and I'm assuming you do get them quite often. To answer all queries all the time, you may also be doing it at the cost of correctness or accuracy. So, how do you basically solve this dilemma?

So, the trade-off is kind of what you mentioned, right? There's a trade-off between usefulness and accuracy. Yeah, you would think that accuracy is inherently useful, but it's not necessarily okay. So the goal of our courier system feature is that if you're unfamiliar with your data or our products, it should be as easy as possible to get something there that you can gain some insight from. If that teaches you how to interact with your data such that you need query assistance or find that manually plugging away or using the UI is actually easier than typing out what you want, yeah, awesome, right? Like it's done its job right. The goal is more product activation. That's literally what it does if you end up in that realm. So because of that, we find when we talk with a lot of new users or people who are onboarding onto a team and are unfamiliar with the data, we literally heard this verbatim from someone. Someone told us, "It's okay to be wrong." And we're like, interesting, can you explain why it's okay for us to be wrong? And then he told us, he said, "Well, if you're showing me something I don't want, yeah. But if it's producing a useful query structure, like you know, you're saying something is not equal to x, but it's actually producing equal to x, yeah, I'll just change it to say it's equal to x. And then you basically taught me how to express that from here on out. That's actually pretty great”. So, we decided in the very first iteration of what we're building, let's optimize for that and we'll see how it goes. We expected that people were going to be very imprecise with what they wanted or they would give very ambiguous inputs and that kind of stuff. And we said, "Okay, let's try to produce something from that. If it's not the most accurate, like, whatever, that's okay." Actually, prior to that, we tried to see if we could be both highly accurate and also highly available by trying some chain of thought prompting techniques and others, and those indeed increased accuracy. But once we started pasting in broader things, it would always reply with like, "Yeah, sorry, I don't know how to do this. What you're saying doesn't make sense or there's no way to produce any output for this." And like, it's not really helpful. So indeed, once we are live in production, people are pasting in all kinds of random stuff.

One funny example is when you look at a distributed trace. Basically, a trace is a set of structured logs that are correlated by an ID, and that ID gets passed around as context from service to service. That's how you know the sequence of events. The ID is critical, especially with observability tools. Usually, people use a separate tool to generate a trace ID and then input that ID into their observability tool to retrieve and analyze the trace. However, pasting a 16-byte hexadecimal value without any context doesn't provide any meaningful information. The purpose is to find traces with a specific ID and examine them. Fortunately, GPT 3.5 training data contains enough information about traces to recognize a 16-byte hex value as a trace ID. The embedding models also identify the trace ID as a relevant column. When passed to GPT 3.5, it filters the input based on the trace ID and provides the desired result consistently. It's not certain that optimizing for correctness and specificity would yield valuable queries based on this approach.

Another example of this is the honeycomb UI has its own embedded domain-specific language. Okay. We have the ability to create what are called computed fields, or we call them derived columns. But the idea is that you have a bunch of fields in your data, and you want to create a computed one that's based on some mathematical operation on other fields that you have in your data, and there's a language for expressing that. One of our existing customers started pasting in that language into the textbox, which, to be clear, is a nonsensical operation. That language exists for a different part of the product that has nothing to do with querying your data, but they were pasting it into the textbox and saying, "Get query." And we're like, "What?" They would say, "If n equals service dot name, comma my service and not contains a HTTP URL, you know, somebody's URL”. And somehow they're getting what they want to query out of this. It doesn't make sense to me. But GPT 3.5 would produce something. And we have a user feedback mechanism. And they would say, "Yes, this is helpful to me." And I reached out to one of these engineers, one of our customers, and they said, "Yeah, this actually gives me the query that I want," and I'm like, "Okay, I could never possibly predict that this was going to be the case, but I'll take a win when I got one." And I think that also kind of highlights, you know, the fact that large language models can hallucinate is by design because you want something to be very creative, potentially too helpful. You want to try to get it to give you a bunch of interesting stuff because you can't possibly predict what's going to be useful to your users. And so if you try to restrict that and try to lock it down as much as you can, like, there are some cases where you might want to do that. But like, we're basically trying to play to the strengths of this tech in general, right? It's just a question of how useful that hallucination is to you. Well, that opens up the space of useful outputs quite a bit more. And we find that for the purposes of a product that uses an LLM to increase product activation, so that people are more engaged with what we have, that tends to be a much better trade-off than trying to go for maximum accuracy and specificity, if that makes sense.

Makes a lot of sense. And I'm assuming that one of the metrics that you must be optimizing for is Product Activation as compared to accuracy. I'm assuming it has led to an increase in engagement-based metrics correctly. How do you measure the ROI?

So, we have a lot of engagement metrics that we track across our entire product. We had a whole initiative this year around product activation and tried various experiments to encourage users to engage more with the product. We found that certain signals indicate that a user is more likely to eventually use the product extensively and even upgrade their plan. The most significant signal is querying the data. The more active a user is in terms of querying the data, revisiting queries, and creating new ones, the more likely they are to become an activated customer later on. Another important signal is when a user takes a query and saves it to a dashboard, as it shows a strong interest in revisiting it later. We also explored creating alerts based on queries, but it didn't have a significant impact. When we split new teams into two groups - those who use the (natural) query assistant and those who have never used the (natural) query assistant - we found that by week six, about 4.5% of the new teams who did not use the query assistant were still running manual queries. This indicates a 4.5% activation rate for actually using our products. On the other hand, for new teams who used the (natural) query assistant, by week six, 26.5% of them were still running manual queries. This represents more than a ~5x improvement in terms of activation. However, this is not the only measure we consider. When we look at dashboard creation, the difference is 3.6% versus 11%. So, there is not as significant of a jump compared to querying. Similarly, if we consider alert creation, the difference is 3.6% versus 5.8%.

And so, what that tells us is that there's a certain degree of activation that the query system is really helping with. However, once we start getting into the "I am using you to alert me when something is wrong" aspect, the differences become quite marginal. We haven't actually delved into that to try to understand if there's a connection or if there's no connection at all. Maybe the population of people who want to get alerted by something is just small enough that it doesn't really matter if you're using the query assistant or not. There's some uncertainty in that, so it's a little hit or miss. But I would say that for the core goal of getting people actively querying our data, it's wildly successful compared to not having it at all.

And in terms of alerts, it also makes a lot of sense, given that it's a key item for many teams who have already put a lot of thought into it. So I'm assuming that these guys have done a lot of homework, and that's why you don't see much uptick there. I had another question about safety, right? Like, how do you ensure the overall application safety, especially regarding RAG-based access control and prompt injection attacks? It's still an unsolved problem, and people are still working on solutions. Just to give some context to the listeners - an example of a prompt injection attack can potentially look like this - suppose you have an AI application that automatically summarizes and responds to emails, and someone sends an email asking for your password, the application might actually end up sharing it. This is just a broad example of a prompt injection attack. So, I'm curious, how do you ensure the safety of customer data in general? What safeguards and guardrails have you implemented?

Yeah. So, first of all, as you mentioned, prompt injection is unsolved. To date, there is no reliable solution to prevent prompt injection. Nobody should ever be under the impression that they are 100% safe. Okay. I mean, any security professional is going to tell you that's true for all software. Yeah, but it's especially true for AI based use cases. So go in with clear-eyed understanding that this is the world you're entering into. Now, there certainly are some mitigations you can have. I can tell that we did.

So first, and probably the single most important mitigation, is that the large language model itself is cordoned off as just an API call, basically external to anything else that matters. It's external to our main database. It has no ability to read any information or write information into it. One of the challenges with people who want to build things like autonomous agents is giving an agent autonomy. You have an end-user reprogrammable system, okay? Yeah. So, if you have an agent that's capable of running on your own infrastructure, initiated by some external user doing something, there's no end to the creativity they can use to get this thing to perform tasks that you don't want it to do. And if you give it autonomy to perform any kind of task, while that's tough, that's why we didn't go that route to begin with. Okay, and so, we certainly verified this by having some penetration testers come in and try to exfiltrate customer data and stuff like that. It just doesn't have access to the things that have sensitive information in the first place. It has no knowledge of who our customers are, so you can't ask an assistant who our highest paying customers are or something like that. There's just no information connected to it that has access to that. That's certainly a trade-off, right? With prompt injections, the more useful you want something to be with that kind of data, the more risk you expose yourself to by doing that. So in a way, that kind of limits where you can put alarms, but there are plenty of other guardrails you can put in there, right?

Secondly, a full chat interface is much easier for someone who wants to engage in malicious actions than an input and output system, right? So, for example, if the output is visual or if the output is a JSON object that fills out the user interface in your product, it becomes more difficult to exploit. That's a lot more challenging for someone attempting to perform malicious actions. Therefore, you are now playing a statistical game of determining how much annoyance someone is willing to tolerate in order to achieve their goal, when they could target another entity that has made themselves more easily accessible.

We also have rate limiting in place for what people can do. So there's only a certain number of natural language queries that you can enter before you get rate limited for today. Again, that kind of gets to, like, not solving everything. But, you know, if somebody is limited to, like, 50 interactions a day, you know, there's only so much they can do each day. And the question is like, okay, is this really worth their time to try to extract data out of? We also limit the sheer number of output that we put in there. So if somebody is able to do some kind of injection, you know, we're supposed to emit, like, a very small chunk of data. If you have a very open-ended chat interface, then you might not be able to do that. But like, these are sort of like the stacked mitigations that we have.

And so I would say that, in aggregate, what we've done is we've created a system that's really annoying for a bad actor to try to get useful, you know, juicy information out of. Yeah. And so, like, who knows, we'll probably get someone trying to do something at some point. But, you know, at this point, we're just about as safe as we can be, I think, given the current state of technology. Additionally, customers can just turn the damn thing off. So, you know, you can say, "I don't want my team to have any access to this." And, like, you know, that doesn't mean, you know, at some point in the future, maybe, you know, your information could have somehow been accessed by somebody else at some point in time. But if you had a bad actor working at your organization, like, if they don't have access to the feature, then they can't do the bad things in the first place. And also, I think, like critically, we're very, very clear with customers on what data we send in the first place to OpenAI and what this thing has access to. And, you know, the majority of our enterprise customers that we talked to are saying, "Sounds good to me." A couple have kind of had some questions, and, you know, we were able to alleviate some of their concerns. There are, I think, about half a dozen or so of our customers (out of overall 600 customers) who have the feature turned off because, you know, they just don't like it. They don't want it. And, you know, in that case, nothing gets sent to OpenAI. There's no like, you know, ambient machine learning going on or anything like that. And so that seems to satisfy most people.

So, less than, like around just around 1% of the companies have issues with using the Natural Query Assistant. Actually, people would assume this number to be higher for some reason, but interesting. You also make a very interesting statement, especially with respect to security. The output of the LLM call is non-destructive and undoable. What do you exactly mean by that? Can you elaborate?

So, because the goal is for our list to appear as JSON objects that run against our query engine, If it produces a bad query, you just get bad query results and change it. It's inherently changeable, right? Like it doesn't write any data to your liking. It doesn't change anything about the data that you're querying itself. It's effectively only a read-only operation at that, and so you, at worst, you're going to get something that is just like a poor query. But you can always just change it. You can undo it in the sense that you can just clear the query and start fresh again from that standpoint. Critically, nobody is given a pager duty alert as a result of one of these things, right? Both as a customer, if you're using Honeycomb, your use of Query Assistant will never produce an alert that wakes somebody up at three in the morning. Okay, similarly, at Honeycomb itself, nobody is getting paged with a 3 am alert if Query Assistant is performing poorly right now. Right? It's something that we're going to look into, but we're very intentional about it. We're dealing with a non-deterministic system that might go totally bonkers. And it might go bonkers, not because it's working badly, but because one user might be doing something really terrible or something. And you know, it's not worth it to wake up Honeycomb engineers because of that. That may change at some point in the future, maybe when LLMs are more incorporated and much more like highly reliable systems or something. But I think because they are inherently unreliable, they end up being a poor fit for that kind of stuff in the first place. And so if you take this approach where the output is non-destructive, it's reversible, and nobody's notified when, or I should say, nobody has woken up in the middle of the night when something is not right or something is not ideal.

Got it! It seems like the query assistant has been quite successful. In the medium to long term, what does the product roadmap look like? What are the broad focus areas?

Several different things. So our query assistant is fairly scoped right now with what it can do. We want to make it broadly better in certain things. Okay, so usually, when you're querying your data, you're asking something to the effect of like, show me stuff that is slow. But if you want to ask, like, why is this slow? What correlates with this behavior that I'm seeing right here? That's a second-level question and the query assistant has no ability to answer that. So what I mean by that is we have a part of the product, we'll call it bubble up. The idea with bubble up is, let's say you have a heatmap that's showing latency, you know, filtered by some service, and there's like a spike. Well, you can visually select that spike. And then what it'll do is it'll compare all the elements in your data in that selection with the rest, like the baseline, basically. It will run a comparison in parallel, and it will show you a bunch of histograms to be like, hey, this value for this column correlates very, very highly with this selection, right? And you get a bunch of characteristics in your data and you say, oh, okay, well, you know, when these three or four fields have these values in them, that's typically when we see that spike, let's narrow by that criteria, and then really sort of validate - when these things are happening, that's when we're seeing latency spike. The query system has no ability to get that bubble-up feature to sort of activate when you ask a question along those lines. If you sort of say like, Hey, I see a spike right here. Can you show me what's interesting about that data, it won't know how to do anything. So we want to add that kind of capability. That's going to be much more advanced. But I think that's really important because right now the Query assistant system is good for like layer one-level questions. But then there are layer two questions that some of our customers have. It needs to be able to answer as well. There are also other parts of our products that can really benefit from models. So one of the cornerstones of our enterprise business is called SLOs, or service level objectives. The idea is that you establish success criteria for a given service. For example, let's say you're an e-commerce site and there's the checkout service, right? Well, the checkout service should usually work because you want to be able to charge people money. Like I said, a non-empty cart transaction should succeed, and they should usually succeed pretty quickly. That's a pretty good success criteria. Now, it's not the end of the world if it fails once over the course of a week, right? It would be nice if it didn't do that. But you shouldn't necessarily wake somebody up tonight. However, if it starts failing a lot, maybe you should progressively notify people and let them know, "Hey, you have this budget of like 99.9% of the time it should work over this time period. So you have an allowable budget of times it's allowed to fail per week.” That's your objective. It looks like your budget is going to burn down before the end of the week. You should probably take a look at this. You probably have a bug in there somewhere interesting. So, that's the reason why it's the cornerstone of our enterprise business. It helps our enterprise customers distill what really matters about their services into something that's quantifiable, and then it starts getting them to query things pretty quickly. It allows our customers to get into a virtuous cycle where they're no longer being reactive about problems. They are proactively identifying issues before their customers really start to complain about them. That's the end goal that we want people to reach, and SLOs are the best part of our product that helps you do that. The problem is creating an SLO is really hard. You need to use a particular DSL that I described earlier, the drive column syntax. You need to create an expression that represents the success criteria, right? For example, service name equals checkout, latency is less than or equal to a certain value, and error does not exist. These are the three pieces of an expression. If all of these things are true, then it evaluates to true; otherwise, it evaluates to false. Most of our customers do not know how to create those things effectively. They need to be handheld by our sales and customer success team to learn how to create an effective expression for their service level objective. So we conducted an experiment with GPT-4 and it turns out, it's extremely proficient at creating service level indicators, as they're called. Those expressions. It's really, really good at creating them based on high-level descriptions of what you care about. And while it doesn't always get it right, it consistently produces things in the correct structure. For example, let's say I described it as the checkout service, but the actual name of the service is some garbled upstream that only I am aware of. The language model has no knowledge of this weird name, even though it's referring to the checkout service. It simply suggests that you might want to change this before committing it as your expression for the SLO. We shared this with a couple of customers and gathered their feedback. They said, "Hell yeah, please give me this right away." Like, we have been spending way too much time trying to figure this out, and this completely helps in leapfrogging the issues. Our sales team really, really wants this feature because they want to spend a lot less time creating SLOs for customers in the first place. So, this is an area that we're going to be investing in as well. And it's not just the creation of that expression, but there's sort of this iterative process, you know, let's start from scratch. What's the name of your service? What are you trying to accomplish? Okay, this is a representation of that, and this expression, if you think this, you know, modify it to your heart's content, let's move on. Okay. Now, let's move on. What are your uptime requirements? Like, how much money is this service associated with? What's a good uptime percentage with respect to that kind of criteria? Okay, now, we're going to note that eventually you end up with a full SLO. You can create that with the AI assistant along the way. Another thing that we looked at, which is also very good, is reviewing an existing SLO and trying to improve it. So, if you have an SLO with a given title and description, how accurately does your expression reflect that title and description, right? Is it being graded like, you know, thumbs up, some, you know, middle, or thumbs down? And, you know, can you rewrite this expression to more accurately describe this success criteria that you've actually laid out here? Because a lot of times we find that our existing customers have trouble creating a good SLO in the first place, and their SLO is actually kind of terrible. And what I would say is that the GPT 4 is not very good at creating a perfect SLO, but it's extremely good at creating a not terrible SLO. And that's actually the bar that we want to get to because that's actually the problem that most of our customers have. Once they get to the point where they have a non-harmful, actually okay, SLO, it's much easier for them to just manually optimize it to their specific needs. But getting them over that hump in the first place is really, really hard. So, that's a whole area that we're looking at right now. I can't say it's committed to the roadmap at this point, but it's probably going to happen one way or another.

Thank you so much. I have one more question. We are observing a growing number of applications being developed using LLMs. When building an LLM-based product, what are the broader guiding principles to consider? For example, people are building internal workflows and consumer-facing workflows. What should be the overarching guiding principles if you decide to include an LLM in your technology stack?

I would say number one, trying to get it to be as accurate and reliable as possible is going to be very hard. But if you say, hey, well, you know, 95 to 99% of the time, it's accurate and useful. That's gonna be a lot easier. I would say another one. It kind of gets to the security and privacy thing, right? Like don't change data with the LLM. Read information and let the user decide what they want to do with that. If you go and you make changes on behalf of the user, yes, you can, in theory, be more helpful to them. But if you start destroying things in a way that's undoable, it can be catastrophic, right? If you mess up data integrity for someone, oh my god, that can be the destruction of your business. Right? So like, try not to build that with LLMs unless you really, really know what you're doing. I would say most people, most of us don't know what we're doing because we don't even really know the limitations of this tech yet. So I don't want to, you know, build cool experiments like go and try stuff that can suffer when it comes to actually production on production, rising things. Take data integrity and data sovereignty and, you know, non-destructive outputs extremely seriously, and make that decision something that your end users should be doing. Unless you begin with something more basic or focused, you may find yourself endlessly spinning your wheels, wondering if your security team will be okay and not feel overwhelmed. And then I think the last one is, it's going to be really hard for you to go to production with LLMs and stay in production unless you can deploy changes to your own systems reliably and quickly. Accept the fact that your users will sometimes do strange things. As a result, it is important to promptly address any issues that arise. Can you quickly implement a fix? Can you continuously iterate? Can you release a fix and then evaluate its performance the next day? If the fix did not introduce any new problems, proceed with further fixes and subsequent releases. If the idea of constantly releasing your software over time and being able to release it at any point is challenging for you, it may also be difficult to adopt the HoloLens beyond just an experimental marketing moment.

As part of this teardown - we will cover the following sections

1. What is ARM?

2. Value Proposition: Why is it needed?

3. More details about the Business model

4. Assessment of the Business: Key Metrics, Future Outlook, Valuation, and Risks

What is Arm?

Arm is a technology provider of silicon IP for system-on-chips (SoCs), with a focus on CPUs. These chips are used in a wide range of applications, including mobile devices, computers, AI/ML, IoT devices, embedded systems, and automotive applications.

To put it simply, any computing device requires a CPU (Central Processing Unit) to execute instructions for computer programs, such as arithmetic or logical operations. CPUs are capable of performing a variety of tasks and allocating processing power to these tasks (such as allocating processing power to different applications - e.g. Slack versus Excel on a laptop). GPUs (where NVIDIA is a leader) are different from CPUs in that they are designed to perform a limited number of instructions in a massively parallel way, whereas CPUs are much more versatile and are the heart of any computing device.

Arm architects, develops, and licenses high-performance, low-cost, and energy-efficient CPU products and related technology. Many of the world's leading semiconductor companies and OEMs (such as smartphone makers) rely on Arm's technology to develop their CPU products. Please note that Arm does not manufacture Chips but rather develops IP (i.e. designs/software ecosystem) and licenses its IP for a fee to chip partners who then develop chips and ship them to OEMs. Arm-based CPUs run the vast majority of the world’s software, including the operating systems and applications for smartphones, tablets and personal computers, data centres and networking equipment, and vehicles. According to Arm, ~70% of the world’s population uses Arm-based products. Arm also claims that >99% of the world’s smartphones run on Arm CPUs.

There are two main CPU architectures that are most common: X86 (powered by Intel) and Arm. Each has followed a different approach, carving out their own use cases. X86 follows a complex instruction set computing (CISC) architecture. It has a large and diverse set of instructions, which can perform multiple operations in a single instruction. ARM follows a reduced instruction set computing (RISC) architecture. It has a simpler and more streamlined instruction set, which generally executes one instruction per clock cycle. Given these differences, it is understandable that Arm is likely to have slightly less computing power than X86, but higher power efficiency, making Arm - the most widely used CPU processor in mobile devices (~99% of mobile devices are powered by ARM CPUs), IoT devices, embedded systems, and some server applications. Given X86 processors are known for their strong performance, they are more dominant in the desktop and server markets. The illustration below is a helpful way to understand the differences between these two CPU architectures - ARM does sequential processing (1 instruction per clock cycle), whereas X86 does parallel processing.

The key Product Offerings

Below are the key product categories that ARM describes in its F1

Arm CPUs. The foundation of our product offerings is our market-leading CPU products. Our CPU products leverage our common scalable ISA and address the widest range of performance, power, and cost requirements.

Other Design Offerings. We have a portfolio of products that are deployed alongside our CPUs, including:

1. Graphics Processing Units. We offer a family of GPU products providing an optimal visual experience across a wide range of devices.

2. System IP. Complementary design components that enable designers to create high-performance, power-efficient, reliable, and secure chips.

3. Compute Platform Products. Arm’s CPU, GPU, and System IP products integrate into a foundational compute platform optimized for a specific end market.

Development Tools and Software. Our tools and software support the development and deployment of our offerings.

In addition to the Design IP, ARM has also developed one of the largest CPU developer ecosystems to further solidify its position. Every CPU has an ISA (Instruction Set Architecture), which defines the software instructions that can be executed by the CPU. Essentially, it is a common language for software developers to use. The ISA sets the foundation for a large library of compatible software that can run on those CPUs. Given Arm’s CPU is the most popular in the industry - So is Arm’s ISA. This also implies that Arm has a very strong community of software developers to program its CPU chips. Chip designers using Arm chips can add the desired functionality (such as Wi-Fi connectivity, image processing, video processing, etc.) to create a SoC to meet their respective needs

1. Software developers (are incentivised) to write software for Arm chips because it is one of the most popular (CPU) chip ecosystems in the world

2. Chip Designers choose Arm because they have the broadest support for software applications

3. The combination of pervasiveness and ease of portability has resulted in Arm's CPU designs having the world's richest software ecosystem. Arm has partnered with leading operating system providers, including Google, Microsoft, and all major Linux distributions. They also work with software tools and game engines, such as Unity Software, Electronic Arts, and Epic Games, as well as application developers.

Value Proposition: Why is it needed?

This is probably one of the most interesting sections, as it discusses some of the fascinating dynamics of the chip industry. There are several key reasons why one would conclude that a company like Arm is needed for chip IP development.

1. Increasing Complexity and Cost of Designing Leading Edge Solutions

   1. Building chips with advanced computing capabilities is increasingly becoming complex and expensive. The resources required to develop leading-edge products are significant and continue to increase exponentially as manufacturing process nodes shrink. According to IBS, IC design costs were approximately ~US$249 million for a 7nm chip and ~US$725 million for a 2nm chip, representing ~3X increase in design cost as compared to a 7nm chip.

   2. For OEMs and technology companies (such as smartphone makers), design partners (such as Arm) can play an important role in reducing development risks by providing specialized capabilities and expertise that enable these OEMs to focus on their core product differentiation. Many OEMs that utilise off-the-shelf semiconductors can face significant issues

      1. An OEM may require a chip with specific features that are not available from a merchant supplier.

      2. OEMs may end up using suboptimal chips that have a significant number of unnecessary features. This can result in increased costs or the available chip being different from the desired expectation.

      3. Therefore, leading OEMs are increasingly bringing chip design in-house to deliver greater performance and efficiency. (which can be a double-edged sword in the case of Arm - more on this later!). Arm provides the base IP and architecture that can be customized by their customers.

   3. Design partners (such as ARM) can facilitate innovation and enhance customers' competitive positioning by reducing the complexity, risk, and cost of a significant part of the development cycle. For example, to design a 2nm chip, IBS estimates that software development, verification, and IP qualification account for 71% of the overall cost

      

2. Increased need for high-performance power efficient compute

   1. To handle the demand for increasingly complex workloads, one approach has been to improve CPU processing speed. One way to achieve this is by packing more processor cores onto a chip (the more no. of cores you have, the better equipped the system is in terms of processing).

   2. For instance, the core count of many smartphone application processors increased by ~10X between 2010 and 2022, while certain data centre server chips saw an increase of >30X over the same period. Furthermore, the number of cores per "high-end" Arm-based chip has increased from 8 in 2016 to 192 in 2023.

   3. An increase in processing power also results in increased power requirements, which may be limited by the energy capacity. For example, even if it were theoretically possible to pack more than 1000 cores into a CPU chip of a mobile device, the device would quickly run out of power (due to battery constraints as it would have to remain similar in size to other phones).

   4. To address these complexities, providers such as Arm are needed to innovate on-chip designs that manage all key vectors - i.e. the optimal balance of performance, efficiency, size, and cost - across end markets.

   5. On top of this, there is a market trend pointing towards making all devices "smart". According to Deloitte’s 2022 Connectivity and Mobile Trends Survey, the average U.S. household had 22 connected devices in 2022, double the number of devices in 2019 (11).

Business Model

The typical business model for Arm is that of licensing and royalty. Arm provides its customers with access to products (for a fixed licensing fee) that enable them to develop Arm-based processors. Once a processor has been designed and manufactured using Arm's products, Arm receives a per-unit royalty on substantially all Arm-based chips. There are multiple types of agreements between Arm and its customers

1. Arm Total Access Agreements (widest set of chip designs available) - Under an ARM total license agreement, a portfolio of CPU designs and related technologies are licensed for an annual fee, which is determined at the execution of the agreement. Arm Total Access agreements are most suitable for larger, established, semiconductor companies. As of June 30, 2023, ARM had 20 Arm Total Access agreements, reflecting an increase of 16 from 4 as of March 31, 2021.

2. ARM Flexible Access Agreements (set of chip designs available) - Under the ARM Flexible Access License Agreement, the company offers a portfolio of chip designs for an annual fee. However, unlike the Total Access agreements, customers may not have access to the latest chip designs/products. To get access to the latest product designs, the customer will have to pay via single use license fee. Arm Flexible Access agreements are best suited for smaller companies, including startups and business units of larger companies, that want to experiment with different configurations. As of June 30, 2023, ARM had 214 existing Arm Flexible Access agreements with its customers, reflecting an increase of 85 from 129 as of March 31, 2021.

3. Technology Licensing Agreements (Single Chip Design) - Under the TLA license, Arm licenses a single CPU or other technology design to customers in return for a fixed fee and then charges royalty on the basis of chip sales.

4. Architecture Licensing Agreement: Under ALA, the licensee develops its own highly customised CPU design which is compliant with ARM’s ISA. One should think of ISA as CUDA equivalent for ARM with only one difference - i.e. CUDA is only meant for NVIDIA chips, however ARM’s ISA can be compliant with chips outside the ARM’s ecosystem

   1. As creating an optimized CPU is very costly and time-consuming, customers who enter Architecture Licensing Agreements are most likely to also license CPU designs for use as a complementary processor alongside their ARM-compliant CPU designs.

5. Royalty Fees: This accounts for the majority of the revenue. It applies to customers who enter into licensing agreements with Arm. These customers pay royalty fees based on either a percentage of the average selling price or a fixed fee per chip sold (on top of annual licensing fees). In FY ‘23 - revenues from royalties accounted for ~63% of the overall revenue. The royalty rates per chip typically reduce over time as the total volume of chips incorporating Arm’s products shipped increases.

6. Historically, most customers entered into agreements through TLA (single chip design). However, in 2019, ARM introduced the Flexible License Agreement, and in 2021, the Total License Agreement. This makes a lot of sense - providing access to more designs is likely to encourage customers to use ARM’s designs in various product lines, resulting in more royalty revenues for ARM from a single customer in the future. While one may argue that ARM is subsidizing on a per-design/chip level, it is likely to more than make up for it in the long run in terms of volumes.

7. One interesting fun fact is that ARM is able to extract value from older generation chips. In fact, ~46% of their overall revenue comes from products released between 1990 and 2012. With ARM’s full suite of access - ARM’s Total Access and ARM’s Flexible Agreement Licenses - more revenue can be juiced out

Assessment of the Business: Part 1 (Market)

With respect to market assessment, I do think it can potentially get very tricky for Arm. In its F1 report, Arm was considerate enough to discuss the various market segments and its (estimated) market share and future growth potential. What I have done is added a small commentary on how easy or difficult it will be for Arm to navigate these markets. One interesting observation about the market is the limited amount of value that Arm derives from the sale of chips, despite being one of the leaders in the CPU chip market. They service ~US$98B (ARM Share x Market Size) in chip sales annually (representing ~50% of chip sales), but only generated US$2.7B in revenue, implying ~2.7% royalty rates. In my opinion, the majority of the segments in the current US$98B chip market that they service are likely to become more challenging in the medium to long term (see the number of yellows and reds below). For a company to grow significantly - it needs to both defend its market share (in market segments where it is strong) and grow into new ones. ARM is one of those cases, where it will have to fight the battle on both fronts.

One of the most important things to consider here is that Arm depends heavily on slow-growing markets and needs to increase its market share in emerging or fast-growing markets (which are expected to become really competitive).

1. Automotive semis, which is a fast-growing market, is itself going through significant changes. Much of Arm's market share is likely driven by the use of Qualcomm chips. However, there has been a significant rift between the two companies after ARM decided to sue Qualcomm. One of the key drivers of growth for this market has been "Cars of Today has $X of semis" and "Cars of Tomorrow will have $XXX of semis". Today, the majority of chips available on the market are fairly low-priced. The low-end market is dominated by analog providers such as Texas Instruments, ADI, and others. Looking further into the future, as heavy-duty computing will be required for autonomous vehicles, NVIDIA and others are already laying significant groundwork. There is also a market opportunity in the middle of low-compute and high-compute categories, specifically focusing on use cases such as handling infotainment, automated driver assistance (ADAS), and the "digital cockpit". Qualcomm, Mobileye, and others are focusing a lot on this category.

2. The data centre category is also rapidly growing but is expected to become more competitive. Currently, Arm’s market share is heavily reliant on just two customers - AWS Graviton and Ampere. Additionally, Arm is an investor in Ampere. To become a dominant force in this market, significant R&D spends are likely to be required.

3. The company heavily relies on the smartphone market, which is currently not performing well. During its earnings call, Qualcomm stated that the expected increase in demand for smartphones has not materialised yet. This market is expected to experience slow growth for the foreseeable future.

4. The Industrials/Embedded segment is likely to see a significant amount of competition, as much of the expected growth is anticipated to come from Chinese chip manufacturers. These chip manufacturers and designers are heavily adopting the RISC-V open architecture, which is emerging as a strong substitute for ARM designs.

Assessment of the Business: Part 2 (Key Financial Metrics)

1. Overall Revenue

   1. Total revenue for the fiscal years ended March 31, 2023 and 2022 was ~US$2.679B and ~US$2.703B, respectively.

   2. During the same period, license and other revenue decreased from ~US$1.141B to ~US$1.004B, respectively. However, royalty revenue increased from ~US$1.562B in 2022 to ~US$1.675B in 2023.

   3. At the risk of over-analysis, it's worth noting that royalty revenue is typically a lagging indicator, while licensing revenue is a slightly more leading indicator. This is because licensing revenue is more likely to be affected by new revenue opportunities. On the other hand, royalty revenue is expected to increase as the adoption of Arm technology in existing chips increases. According to disclosed documents, Arm attributes its decrease in licensing revenue to the size and timing of new licensing deals.

2. Cost of Sales (Gross Margins)

   1. The cost of sales were ~US$106M and ~US$$131M for the fiscal years ended March 31, 2023 and 2022, respectively.

   2. The cost of sales usually consists of providing technical support and customer training. It also includes some amortization of previously developed technologies. My understanding is that it is strongly connected to new revenue expansion opportunities, which have decreased this year compared to last year.

3. Research and Development

   1. Research and development expenses for the fiscal years ended March 31, 2023 and 2022 were ~US$$1,133M and ~US$995M, respectively. The expenses increased by ~US$138M during the fiscal year ended March 31, 2023 compared to the fiscal year ended March 31, 2022, due to increased investments in the next generation of products.

   2. R&D expenses are expected to remain at similar levels (going forward as well) as a percentage of revenue, primarily due to the increasing complexity and performance requirements of subsequent generations of chips. Unlike NVIDIA, which leads in accelerated computing (GPU-driven), Arm does not have a similar achievement to boast. Apple produces the best CPU chips, and many large smartphone manufacturers are considering (or are in the process of) bringing the design process completely in-house to gain even stronger advantages. To stay competitive in the segments we discussed above, Arm will likely need to continue investing significantly in R&D.

4. Selling, General and Administrative Expenses

   1. One of the most interesting evolutions of financial metrics has been around SG&A. The SG&A (as % of revenue) has heavily reduced from ~41% to ~28%. This decrease is primarily due to lower salaries, related expenses, and employee retention awards. The decrease is partially a result of restructuring activities that were announced in March 2022

   2. Selling, general, and administrative expenses primarily consist of salaries, share-based compensation, and benefits associated with employees in sales and marketing. These expenses also include corporate and administrative functions such as accounting and legal professional services fees, depreciation and amortization, advertising expenses, allocated overhead, information technology, and other corporate-related expenses.

   3. It is important to draw an interesting comparison with NVIDIA, as it operates in the adjacent computing industry. NVIDIA's SG&A spend in the last fiscal year was ~9%, and in the recently announced last quarter, it was just about 5% of revenue. This highlights the real strength of technology moat-led market leadership and strong market tailwinds.

The net income generated as a result from continuing operations was ~US$524M (with non-GAAP income of ~US$657M), compared to ~US$676M in fiscal year FY22 (with non-GAAP income of ~US$663M)

Future Outlook and Risks: Valuation and Risks

One of the most important metrics in contention has been the company's valuation. In its last private financing round, the company was valued at slightly over US$64B. At a US$64B valuation, ARM would have a revenue multiple of 24X and a profit multiple of 122X (based on US$2.68B revenue and US$524M profit). In contrast, Nvidia is currently trading at 27X (revenue multiple) and is likely in its most dominant position ever, making it an inappropriate benchmark (for Arm).

Arm launched its IPO roadshow in the first week of September. The company has updated its prospectus. It has been revealed that they are seeking a valuation of ~US$52 billion for their shares, which is still an extremely challenging goal. This translates into a revenue multiple of 19X and a profit multiple of 99X.

The most critical question for any analyst is to find the right comparables for Arm. Simply, comparing it to the likes of NVIDIA is gross trivialisation (and a sin!) in my opinion. The business models and what these companies do (at their core) are significantly different. NVIDIA is the best at what it does, which is GPU-led accelerated computing and also maintains tight control over its own hardware. On the other hand, ARM only controls the broader design IP that can be heavily customized and caters to multiple vendors, providing equal importance to each of them. This means that it does not allow for significant value accrual. For example, Apple is known for designing the best CPU chips (although they're based on Arm architecture designs) because they heavily customize these designs for their own use cases.

Putting everything above into context, ARM seems to be closest to EDA (Electronic Design and Automation) semiconductor companies. EDA is a market segment consisting of software, hardware, and services with the collective goal of assisting in the definition, planning, design, implementation, verification, and subsequent manufacturing of semiconductor devices or chips. ARM licenses intellectual property (IP) which its licensees use to create products, which is very similar to the business model of EDA companies mentioned above. Although I have listed a potential basket of comparable companies below in the table, Cadence and Synopsys would be my most important comparables in terms of their business models, size, and what they do. They have been highlighted as well.

The two main comparable companies currently trade at ~12-17X TEV/LTM Revenue and ~49-52X TEV/EBITDA multiple. Their annual revenue scale is in the range of ~US$3.8-5.5B and are growing between ~12% to 16% YoY in terms of revenue. To put things in context, ARM's annual revenue of US$2.7B has remained relatively flat since last year. However, this may be partially attributed to the cyclical nature of the semiconductor industry. Depending on this comp set - the valuation is likely to be in the range of ~US$20B to ~US$45B (much lower than the ask of ~US$52B). One of the potential ways to grow into their valuation of US$45B will depend on Arm’s ability to extract more royalty from their serviceable market (remember ~2.7% from the US$98B market they serve). This is likely going to be difficult because - 1) in the markets where they have the lion’s share, they serve customers such as Apple and Qualcomm, who are known to be demanding on their suppliers 2) and In the markets they want to grow, they are likely to see lower growth (and hence lower market share growth in the medium to long term)

Key Risks

1. One of the most important risks is that Apple (and Qualcomm) are bringing back more elements of the chip design process in-house.

   1. The key takeaway is that ARM does not favour any of its customers over others. As a result, market-leading companies such as Apple or Qualcomm may find it difficult to create chips that are differentiated from others in terms of parameters such as computing power and efficiency. Below is a snippet from Qualcomm’s CEO in an interview with Ben Thompson

   2. “We invested in other areas, started licensing CPUs from Arm, but then we had exactly the same situation happening again. We started on this journey with Microsoft, to build the next generation PC, we are the Arm ISA partner of Microsoft, we’re using the Windows on Snapdragon, and we realized that we needed a CPU that wasn’t available in the ecosystem, that could be the leader in performance in the segment. That’s when we acquired Nuvia, and I think that Nuvia has been integrated into Qualcomm and we have been developing our own custom CPU called Oryon, and it’s going to be applied across our entire portfolio of products. The first instance is on PCs and it’s history repeating itself. We needed to go develop something because we believe the level of performance is going to be required for leadership is not available in the ecosystem, and we’re very bullish about what we can do with our new SOC, including our Oryon CPU.”

2. China is one of the biggest elephants in the room that can potentially impact ARM’s revenue potential

   1. Arm China OEM contributes towards ~24% of the revenue (at least in the last year). Revenue contribution from China is likely going to decrease.

   2. ARM has no control over the Chinese entity. While a Softbank-owned subsidiary holds a 48% stake, control of the company rests with the Chinese entity. ARM China's payments to ARM are dependent on the data shared by them. In the past, there have been issues with obtaining relevant data from ARM China in a timely manner. ARM China has a history of making late payments and is currently behind as of this filing. ARM has limited options for recourse other than to pressure ARM China to make payments on time. ARM itself noted in its prospectus about the above issues

      1. “For the fiscal year ended March 31, 2023, although our total revenues derived from the PRC increased as compared to the prior year, the growth in our royalty revenues slowed for the same period primarily as a result of economic issues in the PRC and factors related to export control and national security matters.”

      2. “Furthermore, in light of these issues, we expect to continue to see declining royalty revenues derived from the PRC and we could see a decline in licensing revenues.”

   3. Developments in Export Control Regulation

      1. The US Department of Commerce has imposed sanctions on the People's Republic of China that limit the country's access to high-end computing chips, software, hardware, equipment, and technology used to develop and maintain supercomputers and advanced semiconductors.

      2. There are two key rules (that can potentially impact ARM’s revenue potential)

         1. Implementing new process requirements that require licensees to disclose semiconductor manufacturing facilities used for manufacturing Arm-based integrated circuit designs

         2. controlling previously non-controlled Arm interconnect IP as well as releasing a modified version of such IP that does not exceed the BIS performance thresholds for the PRC

3. The proliferation of RISC-V Architecture (There is a caveat to this)

   1. Several analysts have raised concerns about the significant challenges that the RISC-V architecture can pose to Arm. I agree with these concerns because RISC-V is an alternative architecture to both Arm and X86, and it is open-source, meaning that potential customers do not need to pay royalties. However, I also think that we may be slightly overestimating the implications

      1. Most of the RISC-V adoption is currently occurring in China, particularly within the automotive sector. This is an area where Arm has a limited presence (i.e. the intersection of China and automotive)

      2. Secondly, and more importantly, let's not forget that ARM has spent considerable resources over decades to build a developer ecosystem to promote its own ISA.

      3. Most importantly, the advantage of open source is the power of community and shared learning. The biggest problem with RISC-V is that people are taking the base architecture and adding components to it (including their secret customisations) without making it public, implying that the power of collective learning cannot be leveraged

Finally, a friend said that there is no science to valuation; if you believe Arm's story, then a value of even US$52 billion would make sense. However, I would lean towards the lower end of the valuation range of US$20B-US$45B.

Have a good week folks! 🙂

The Unbundling of Gaming and Sony’s effort to bundle it

The gaming industry began as an integrated market, much like mainframes. However, it was later unbundled with different providers powering different stacks (just like PCs). We have since seen many attempts by large platforms to re-integrate these stacks in order to reinforce their defensibility. Obviously, Sony was one of the main protagonists of this reintegration era.

In the earliest form of gaming, games were completely tied to the hardware. Atari, the company that released the first gaming console, developed its own games. The founding of Activision separated the hardware from the software stack. This resulted in two types of companies - first that built the hardware (such as consoles) and second that built games (for consoles). However, the creation of 3rd party software game developers also led to the creation of poor-quality games, which caused the famous gaming market crash of 1983. During this time, revenues plummeted from ~US$3.2B (in 1983) to ~US$100M (in 1985).

Nintendo, one of the leading gaming companies, slightly modified this model with its famous NES (Nintendo Entertainment System) hardware and created the first version of today's AppStore. Nintendo had tighter controls for third-party developers to publish games on its platform. Nintendo required third-party providers to pass a number of tests and asked them to pay 30% royalties (told ya AppStore!) as part of its licensing fee. On top of this, the developers could publish a maximum of five games in any given year.

Sony also adopted a similar model to Nintendo but was much more generous to 3rd party developers. This made it a more favored platform for developers. Unlike Nintendo, Sony's platforms utilized CD-ROMs instead of cartridges. Although cartridges were faster, CD-ROMs offered more space, becoming first choice of game developers. Despite the Sony Playstation's inferior hardware compared to Nintendo's, it went on to dominate the market thanks to its larger game library.

There was another significant market shift with the creation of gaming engines (such as Unity). Up until this point, games were designed for specific consoles or hardware. These Gaming Engines were designed to facilitate game development by providing developers with a set of tools, libraries, and functionalities to assist them in building, designing, and running games. Rather than designing games for a specific console, it made more sense to develop the game for a specific engine that abstracted away the challenges that came with building games for different hardware. One simple way to visualize this shift is to imagine an operating system (on which applications are created) that can run on any hardware (Sorry I trivialised this a bit too much but hope this makes it clear!). This was the first time that large platforms, such as Sony, started to think about reinforcing their defensibility, as game developers were no longer directly tied to a particular hardware. This was also the time when the gaming market had matured enough to see some of the gaming franchises (such as Super Mario, Legend of Zelda) command supremely large fan bases.

Sensing these shifts, Sony - after the launch of its PlayStation platform - started pursuing an aggressive strategy of PlayStation exclusives - restricting certain games to its PlayStation devices. Sony achieved this by developing its own internal gaming studio and acquiring other gaming studios. The games produced by these studios are exclusively available on Sony's platform. The advent of gaming engines made it possible for game developers to work across different hardware platforms. This made it incredibly difficult for gaming platforms to differentiate themselves. However, with this strategy, Sony was clearly able to distinguish itself in the eyes of users. As a result, some high-quality games were only available on PlayStation. This strategy helped Sony in increasing the share of PlayStation Devices, which in turn resulted in a significant increase in Sony’s royalty fees for 3rd party developers.

Microsoft’s Push to Change the Business Model

Microsoft views gaming through a different lens than traditional gaming platform companies such as Sony and Nintendo. The conventional video gaming industry focused on a product-based business model: you developed the "game" and then sold it via Cartridges, CD-ROMs, or other means. If the franchise was successful, you invested in the franchise every year for the next few decades and sold it via different platforms. Essentially each game was purchased individually by users. All the key stakeholders operated separately in order to offset the risk of developing and marketing the game - These would be Game Developers (Responsible for game development), Game Publisher (Responsible for marketing and distribution), and Platforms (Responsible for providing hardware and software environments in which games are played)

However, one of the key strategic changes in gaming over the past two decades has been the shift from consoles and PCs as the primary platform to mobile devices as the primary platform. By the early 2000’s, >95% of the consumer spending on games was accounted for by the PCs and consoles category. However, the rise of smartphones has brought about a change in this trend. As a platform, mobile devices now represent >50% of consumer spending, which amounts to an annual spending of US$106B, and Consoles account for ~30% of the spend.

Game developers have responded to this shift by making their games available on all types of platforms, including consoles, PCs, and mobile devices, to maximize distribution. This is also one of the fundamental reasons why charging low platform fees to game developers and locking them out of other platforms doesn't work as a strategy. It's not in a developer's interest to restrict their audience to a particular platform, given the exorbitant costs of game development - which can go in the range of a few US$100Ms.

Recently, Sony shared data on game development costs as part of the FTC trial on Microsoft and Activision Deal and did a phenomenally poor job of redacting the document

1. Sony’s top first-party titles have development costs exceeding US$200M: According to the document: “Development on Horizon Forbidden West, a 2022 SIE first-party release, lasted a total of 5 years, starting in 2017 and ending in 2022. The game cost US$212M to develop, and peak headcount was over 300 people.” According to many estimates, the game has ~9M players on PlayStation.

2. The Last Of Us Part II also cost over US$200M and took even longer to create: “Development on The Last Of Us Part II, a 2020 SIE first-party release, took longer at 70 months, starting in 2014 and ending in 2020. Development costs were around $220 million, and headcount peaked around 200 full-time employees.”

The document notes that in addition to development costs, marketing costs for AAA games are also extremely high.

The proliferation of gaming platforms, particularly mobile, and the willingness of game developers to create cross-platform games, has enabled Microsoft to pursue a new business model (which imo is a great one). They are making unlimited gaming titles available behind a subscription paywall. This allows gamers to play any number of the games covered by the Xbox subscription Gamepass, regardless of the devices they own. This includes Mobile, PC, and Xbox. This is very similar to what media streaming companies pursue as part of their monetisation efforts. This strategy is completely different from Sony's, which focuses on locking customers into the PlayStation universe through its exclusives. In most cases, gamers have to purchase games individually to play (Note: Sony has launched a PlayStation subscription, but they are significantly behind Microsoft).

From a gamer's perspective, the Xbox Game Pass subscription makes a lot of sense. For $180 per year, users can play as many games as they want. Typically, a single gaming title would cost around US$60-70 on a particular platform. With the Game Pass, US$180 per year provides access to all types of devices, including Xbox, TV, PC, and more. The multi-platform approach also makes sense given many multiplayer games are seeing cross-platform usage (say a multiplayer game such as “Call of Duty” being played by one gamer on Xbox and another on a Mobile Device).

However, for the Gamepass subscription business to truly scale, Microsoft would need to ensure that most of the popular game franchises are covered under the subscription. This would require Game Developers and Publishers of the most popular game franchises to negotiate new revenue-sharing agreements, as earlier forms of income-sharing agreement were based on per-unit sales of a game. Another option for ensuring that Microsoft has access to popular franchises is to acquire high-quality Gaming Studios. Microsoft’s acquisition of Zenimax Media for US$7.5B (creator of gaming franchises such as The Elder Scrolls, Fallout, and others) and Activision Blizzard for US$69B (creator of gaming franchises such as Call of Duty, Diablo, Candy Crush, and others) is a part of this broader strategy.

Microsoft is taking big bets on the gaming industry's shift from a device-centric era to a device-agnostic era. It plans on empowering people to play the games they want, and with whom they want. However, the journey ahead isn't that easy. Few industry commentators believe that structural shifts like this can be detrimental to competitive dynamics.

The issue with the Activision Acquisition

There are several reasons why the FTC and CMA are attempting to block the Microsoft deal, but one of the core reasons cited by them is that Microsoft has both the "ability" and the "incentive" to foreclose its rivals from Activision’s key gaming franchises. The Federal Trade Commission (FTC) has expressed concern that Microsoft's acquisition of Activision is aimed towards a broader strategy of creating their own version of exclusive ecosystem, potentially blocking access to titles like "Call of Duty" on Sony's PlayStation platforms. By doing so, it can force users out of the Sony ecosystem and place them into its own ecosystem. Although I am a strong proponent of this acquisition, there are some issues that would make one believe that this can happen.

1. Call of Duty is an extremely large franchise (and Sony is dependent on it) - According to some estimates, the Call of Duty franchise generated >US$800M in U.S. spending on Sony hardware (including in-app purchases) in 2021, and ~US$1.5 billion in worldwide spending. As of 2020, the franchise had earned a total of ~US$27 billion in lifetime earnings across multiple platforms. Call of Duty has been played by >43M unique PlayStation users, representing 2/3rds of PlayStation’s ~64M user U.S. account base, since 2019. The game was played on ~32M PlayStation devices. To put things in context, Sony has sold ~38M PS5 and ~70M PS4 devices to date globally (and each new platform comes after 7-8 years).

2. There is a slight caveat in Microsoft’s non-exclusivity promise - Although Microsoft is willing to sign a 10-year deal with Sony to ensure that the games (such as “Call of Duty”) are available on Sony’s PlayStation devices, with the purchase of ZeniMax - Microsoft made its titles such as Redfall, Indiana Jones, and the role-playing game Starfield exclusive to its Xbox and Windows platforms. Microsoft had also made it clear to regulators during the (Zenimax Media’s) acquisition that it may seek to protect future intellectual property for exclusive purposes. Within the FTC hearings though Satya Nadella has vehemently spoken against any idea of creating exclusives. Here is his quote -

   “If it was up to me I would love to get rid of the entire exclusives on consoles, but that's not for me to define especially as a low-share player in the console market. The dominant player there [Sony] has defined the market competition using exclusives, so that's the world we live in. I have no love for that world."

3. Creating breakout franchises is incredibly difficult (especially if Sony were to be denied access to any gaming titles) - ****New games are finding it extremely difficult to dislodge top incumbents. According to some estimates, there are only 8 games with >1M daily active users (DAU) on PlayStation, and less than 60 with >100K DAU, from the entire catalog of PS4/PS5 games. Many of those top titles, such as Fortnite, GTA V, and Rocket League, have built up 5-10 years of content. If we were to look at the top 100 games for PC, console, and mobile - less than 10 of them would be new, while the rest would already be in a live environment.

The reason Activision Acquisition is still great for the industry

It’s in Microsoft’s interest to not pursue “Exclusives” Strategy. The likelihood of Microsoft achieving monopolistic characteristics or power in certain gaming genres if it were to resort to such a tactic - is close to nil. Denying access to high-quality premium content has been a long-standing practice, as seen in Sony Exclusives, in order to benefit from competition. However, as the game companies are moving more towards services based business model - recurring payments with subscription models, and gradual monetization in case of free to play mechanics - poor game experience can lead to reduced monetization. If Activision doesn't consistently improve its games, it may suffer from monetization issues. Moreover, some of the most popular titles, such as "Call of Duty," heavily rely on multiplayer gameplay mechanics. Therefore, Microsoft has no incentive to reduce its target audience, as doing so may result in longer wait times for players to find matches. Reducing the user base of these games is likely to create challenges, which would make it difficult for Microsoft to recoup its investment. This is especially true in the context of the acquisition cost of ~US$69B for Activision Blizzard, which is at a significant premium to market value. The largest publishers are now incentivised to allow their games to be played on all devices, especially mobile devices. Fortnite is a great example of this. It's available everywhere and is very successful. If Microsoft blocks Activision's games on other devices, it will probably lose money in the medium term.

The acquisition will encourage companies like Sony and Nintendo to adopt a Platform Agnostic strategy and incentivize them to pursue diverse business models. Traditionally, gaming companies have focused on providing games for specific hardware, such as consoles. However, Microsoft has significantly challenged this notion and aims to make gaming a cross-play phenomenon (across devices). This would allow gamers to play a certain game regardless of the hardware they own, much like people using different telecom services to communicate with each other. This is especially important in the context of Sony’s and Nintendo’s strategy of focusing on their own ecosystems up to this point, which by no means is the right way to expand the surface area of the gaming market.

Nintendo and Sony, current console leaders, have only recently begun exploring cross-play and digital distribution for their intellectual property. Nintendo uses mobile as a marketing tool to drive business to its hardware ecosystem. Sony only established its PlayStation Studios Mobile division this year, after acquiring Savage Game Studios last August. Arguably, it was not until Microsoft's Xbox game subscription pass began to perform well that Sony launched its own subscription service. As Microsoft invests in cross-platform gaming and emphasizes the Game Pass subscription service, which offers access to 400+ titles for an annual fee of US$180, Sony is expected to diversify its revenue streams. In fact, one can see some changes in their long-term strategy in their recently released annual reports (which is quite transparent honestly 🙂). Sony is aiming to diversify its IP beyond its consoles, with a focus on other forms of gaming such as PC, mobile, and non-gaming initiatives. Below is the commentary by a Sony exec on this piece

“We have a very deliberate and clear strategy of diversifying the exploitation of our IP away from the console space into other forms of gaming - PC and mobile, as well as a number of non game-related initiatives… already, our PC business is a significant profit contributor.”

Additionally, there’s a lot of investment in Sony’s first party live service titles. The proportion of budget skewed towards new titles is ~50%. Historically that number used to be ~20%. Sony is also likely to shift its focus from traditional games, which typically follow more product led business models, to live service games, which typically follow more service-led business models and are likely to embrace cross-play mechanics. The Live Service games are expected to represent ~60% of the budget in FY25 as opposed to ~12% in FY19.

Subscription led revenue models are likely to promote healthier industry dynamics for small / medium Game Developers. With a heavier focus on subscription-led revenue models, small and medium-sized game developers may benefit greatly from game discovery. Many gamers are more likely to try out new games with this business model than they would be without it. In adjacent industries, such as streaming, it is a common practice to bundle popular content with less well-known offerings. As low-risk tryouts are preferred by consumers, smaller developers have a better chance of reaching audiences and monetizing their products.

Reducing the Clout of Apple and Google in Mobile Games Market. The mobile gaming segment is the largest segment within the gaming industry, accounting for ~US$106 billion in annual consumer spending. This segment is dominated by two platforms: Apple and Google. Both companies primarily generate revenue from platform fees. However, Apple has introduced its own first-party gaming subscription service called Apple Arcade, which is expected to generate US$1.2B in revenue by 2025. While this represents only around 6% of overall revenue (from gaming), first-party content is generally more profitable. Large platforms often change their publishing policies to favor first-party content for this reason. Apple, for example, has been known to alter its publishing policy to promote its own games at the expense of third-party publishers. This significantly reduces the negotiating power of small and medium-sized publishers. Even large game publishers like Epic Games have limited leverage against platforms such as Google and Apple. Both the App Store and Play Store removed Fortnite, the most popular free-to-play title, when it began promoting in-game purchases and offered an alternative way to purchase in-game assets.

Microsoft's acquisition of Activision Blizzard's mobile titles, including Call of Duty, Diablo Immortal, Candy Crush, and others, will give the company a significant stake in the mobile gaming market and make it a significant voice against Apple and Google. This move is expected to add ~US$3.2B in mobile gaming revenues, increasing Microsoft's share of the overall market. As a result, Microsoft will become the third-largest game maker, behind only Tencent and NetEase, giving it much greater negotiating power than Activision Blizzard, which relies on both Apple and Google for roughly ~35% of its annual revenues.

Vertical Integrations can come with their own headaches

While there are several reasons why this acquisition could be beneficial for the industry, historically, mergers and acquisitions have not always delivered the expected results.

Certain adjacent industries within the media segment have undergone vertical integration (i.e. multiple stacks being controlled by a single entity), resulting in poor industry dynamics for key stakeholders in the past. One of the key features of the shift towards video streaming services (such as Netflix, Hulu, and others) was that the creator of content and distributor of content became one entity. Consumers had the option of selecting one of the vertically integrated behemoths such as Netflix, Disney+, or Hulu. However, it was a group of large companies fighting with each other instead of a vibrant set of content creators (movie producers) competing and operating in an open market. The power of these content creators actually drastically reduced, which became one of the most challenging problems. Fast forward a few years, it has become incredibly difficult for movie producers to sell to streaming giants and there are significant barriers to entry for a new movie production house. Writers, directors, executives, and other stakeholders all realize that streaming is not the best business for them due to their limited scope and powers. This poor dynamic coupled with the inability (of streaming giants) to produce high-quality (experimental) content has led to poor user experiences. Many stakeholders within gaming believe that Microsoft Acquiring large Gaming Studios such as Activision could lead to a similar scenario, where Microsoft initially accrues a large user base through its attractive pricing strategy only to weaken other key stakeholders within Gaming Industry. Needless to say, if this were to happen, it can be detrimental to the whole industry.

I don’t have a dog in this fight but I am certainly excited about what’s yet to come. One disclosure I want to make is that I am not a gamer, nor do I claim to be one - so happy to engage on DMs 🙂

While there is nothing wrong with the below tweet by Matt Turck describing the new Startup teams, some people share the opinion that the era of turning any enterprise application into a SaaS company has passed. This trend was previously driven by two factors: a) the easy flow of funding due to zero interest rates, and b) the significant decrease in the technical effort required to build a company, thanks to the availability of AWS, React framework, and other tools. Founders who aim to build high-quality AI companies will need to think differently to succeed. The fact that it only requires “few rate determining resources” to build companies indicates that the barrier to creating a product or company has significantly reduced. In the on-prem era (not long ago, by all means), access to server compute was considered a moat. An increase in the number of companies created means an increase in value generated, but it also means that a significant number of them will fail. Thankfully, the significant pace seen in the open-source community has been a great source of strength for new companies.

Let's Talk About How the Incumbents are Killing It

NVIDIA

There is a growing interest amongst large companies, particularly within specific categories, in offering their own custom AI infrastructure (including Foundational Models). This trend is reflected in recent announcements, such as NVIDIA's launch of DGX Cloud, which was highlighted during their GTC keynote last month. DGX enables enterprises to create and deploy their own models, and run AI applications with lower inference costs using NVIDIA's acceleration library suite. Customers can either build or leverage (pre-trained) language, visual, and biology-based models through these services. For instance, NVIDIA Nemo is a tool for building custom text-to-text models. Enterprises can either bring or build their own models, or start with NVIDIA's pre-trained models and build on top of them.

NVIDIA and AWS have collaborated to build next-generation infrastructure for training machine learning models. The collaboration aims to create one of the most scalable environments for providing on-demand artificial intelligence infrastructure. This infrastructure will be used to train complex LLMs and develop AI applications.

1. More specifically, they are launching Amazon Elastic Compute Cloud (Amazon EC2) P5 instances, which are powered by NVIDIA H100 Tensor Core GPUs.

2. P5 instances are NVIDIA's DGX H100, which is basically eight H100s packaged together so that they can be addressed as a single GPU.

Sorry, I threw a ton of technical terms but essentially, what this means is enabling companies to have access to a powerful computing system that offers speed, scalability (addition or removal of compute power), efficiency (minimizing operating costs), and simplicity (Setting up and managing AI workloads). Below is the architecture that NVIDIA envisions for companies

Amazon (Amazon Web Services)

Amazon is offering layer cake services for AI / ML applications. There are broadly three layers we can look at

1. Layer 1: Broad Support - Compute through GPUs and fast network for training and support using PyTorch and AlexNet

2. Layer 2: Training and Inference Pipelines: Sagemaker is a platform for companies that want to train and execute their own machine-learning models. It provides capabilities for training and inference, pipelines, and data management.

3. Layer 3: Proprietary Services: Text generation, chatbots, search, image generation, and other

Amazon has launched its Amazon Bedrock service, which is a fully managed service that provides Foundation Models (FMs) from leading AI startups (such as g AI21 Labs, Anthropic and Stability AI) and Amazon through APIs. With Bedrock's serverless experience, one can quickly get started, privately customize FMs with their own data, and easily integrate and deploy them into their applications using AWS tools and capabilities (including integrations with Amazon SageMaker ML features like Experiments to test different models and Pipelines to manage FMs at scale) without having to manage any infrastructure. The most common use cases they list are following

Microsoft and Open AI

I won't even bother to talk about what Microsoft and OpenAI have accomplished in the past few weeks. Every new announcement seems to cause significant disruption to many companies. Their broader goal is to become the backbone of AI-optimized computing infrastructure. A good indicator of how busy things are at Microsoft is the number of announcements the company makes per month, which is increasing steadily month on month.

Although it seems that we are moving away from a world where model training was a barrier, as engineers can now remotely train models in a single evening (more about that later!), the real proliferation of LLMs and LLM-driven applications will come from inference - both through the cost of inference and increase the pace of inference.

“If you use GPT-3.5-turbo with 4k tokens for both input and output, it will cost $0.004 per prediction or $4 per 1k predictions. As a thought exercise, In 2021, DoorDash's machine learning models made ~10 billion predictions per day. If each prediction cost $0.004, that would amount to a daily cost of $40 million!”

Microsoft is investing heavily in stabilizing the inference costs and increasing inference speeds to facilitate the mass adoption of AI with real use cases. As AI model sizes continue to grow, inferencing will require GPUs networked together. This is why Microsoft has been expanding its ability to cluster GPUs with InfiniBand networking across the Azure datacenter footprint. Microsoft has also invested in systems optimization with the ONNX Runtime. Azure aims to provide AI-optimized infrastructure that is ideal for AI workloads ranging from model training to inference.

Although it's not clear whether LLM's computing ecosystem has matured yet (I believe there is still significant room for improvement), inference costs have already decreased significantly and continue to do so. In March 2023, OpenAI introduced APIs for ChatGPT and Whisper, which are priced at $0.002 per 1k tokens. This is 10X cheaper than their existing GPT-3.5 models.

Typically the core drivers of improving the inference efficiency are as follows

1. Hardware: Specialised processors and systems built by providers such as NVIDIA and others

2. Architecture and primitive re-implementations: For example, there is a rumour that ChatGPT Turbo, which is ~4X faster than the original ChatGPT, isn't a smaller or different model, but rather a different implementation of computing primitives. To put this in very basic terms: if a foundation model is a Lego building, then primitive reimplementation is the process of recreating a new Lego building by rearranging the same Lego blocks.

3. Better Trained smaller models on larger relevant datasets: The LlaMa paper has shown that smaller models can be significantly more accurate and much more efficient if they are fed relevant data over longer periods of time.

Combining all three factors can result in a significant improvement of up to 40 times in inference efficiency metrics (per year). The chart below shows the impact of each individual driver on this improvement.

The Plot Twist - The ongoing journey from The General LLM → Task Specific LLM

There is a general consensus that there will be an oligopoly among large, general LLMs in the foreseeable future. While I do not refute this argument, it is also the first (usual) answer that comes to the mind and in my opinion “first answers are usually wrong” (or in this case needs to be very nuanced).

As of now most hyperscalers (including Open AI in this argument) are putting significant efforts into creating large General Purpose LLMs to solve the majority (~80%) of problems, while still providing a great user experience at the core. The proposition sounds really good, but there is one significant challenge: while it may be suitable for a consumer use case, the needs of enterprises differ. As a consumer, you would typically want to use a single (comprehensive) assistant to complete most of your tasks. This is why ChatGPT plugins make a lot of sense. Considering the distribution that ChatGPT has amassed (i.e., ~200M user base), it makes sense to use its interface to book plane tickets or order groceries instead of requiring users to use separate apps and perform these tasks separately. It makes a lot of sense to have a versatile LLM as long as it scores reasonable marks on latency and inference metrics. However, the game changes when you look at enterprises and their requirements. There are a couple of issues

1. General (Versatile) LLMs, such as GPT4, have not been trained on enterprise data (say slack messages, internal communication, system logs and others)

2. Task-specific LLMs may be better suited for enterprises because they significantly optimize inference costs and to a certain extent provide better posturing on security and privacy

3. General LLMs, may not be performant for enterprises where query volume is large

This nuance can be validated further by the fact that low-end disruption often occurs in business use cases, rather than consumer use cases. Low-end disruption happens when a modular, inferior (yet cheaper) product overtakes a fully integrated, expensive product in terms of innovation. This concept typically applies to enterprises, where a CIO or Enterprise Customer must justify each software purchase with a high degree of rationality.

Product benchmarking is done with respect to different parameters and pricing when it comes to software purchases (by enterprises). However, the definition of rationality for a consumer is much less stringent. Therefore, we have not seen an integrated product like the iPhone being unbundled. Not every attribute that matters for a consumer product can be documented, measured, and unbundled (or felt in the case of the iPhone:P). I have tried to create the broader framework (of choosing LLMs) for enterprises in the below table

One factor in which Task-Specific/Smaller Open Source LLMs are rapidly catching up (in the matrix above) is the quality of output. While some may argue that the quality of output is still measured on general use cases, such as common sense reasoning or reading comprehension, the pace of improvement is nevertheless significantly impressive. The pace of OpenAI's releases itself has been astonishing, and it speaks to an interesting phenomenon in generative AI. Once a certain scale is reached, foundation models can become reusable building blocks for creating new models or platform capabilities, similar to traditional software development. OpenAI built GPT-3.5 on top of GPT-3, ChatGPT is based on GPT-3.5, and GPT-4 is based on ChatGPT. The release cycles in machine learning (within the Foundation Model space) have become shorter to levels we have never seen before. This phenomenon is not only true for OpenAI but for the open-source community as well, which has picked up a rapid pace since Meta's LlaMa was leaked to the public.

At the infrastructure layer, we are seeing highly performant foundational models being built with a fraction of the parameters compared to those of OpenAI, and with a training cost of up to $100. It seems that one of the best approaches the world is moving towards is to build smaller models that optimize for inference and latency, with larger datasets and longer training periods. There has been significant commentary on this fundamental shift in training language models. Andrej's tweet also alludes to this.

A great example of this phenomenon is Vicuna, an open-source chatbot trained by fine-tuning LLaMA on user-shared conversations collected from ShareGPT. Preliminary evaluation using GPT-4 as a judge shows that Vicuna-13B achieves more than 90% quality of OpenAI ChatGPT and Google Bard, while outperforming other models like LLaMA and Stanford Alpaca in more than 90% of the cases. The cost of training Vicuna-13B is around ~$300.

The overall process of building the model was as follows:

1. Collected ~70,000 conversations from ShareGPT (a website that records conversations between ChatGPT and its users).

2. Used enhanced training scripts provided by Alpaca to better handle multi-round conversations. The idea is for the model to keep the context of the previous conversation alive and not treat new prompts as new conversations. This process is also called instruction training

3. Conducted training using PyTorch FSDP on 8 A100s in a single day. The main advantage of PyTorch FSDP is that it distributes the model and data among the GPUs, allowing them to perform independent computations before combining the results. In contrast, the Data Parallelism approach provides each GPU with a copy of the same model and has them process different data points. The PyTorch approach is much more efficient.

The Table below depicts the performance of Vicuna compared with other foundational models on various parameters

With the advent of parameter-efficient fine-tuning methods (also called PEFTs), such as LoRA, that can potentially reduce both the cost and system requirements of fine-tuning, it is very much possible to rebuild (on top of existing Open Foundation Models) and fine-tune them for task-specific processes - all this while maintaining a reasonable cost level. For AI Nerds here’s a brief explanation about one of the more popular methods (aka LoRA).

“In simple terms, when we talk about large language models like GPT-3, training and fine-tuning them requires a lot of computational resources and storage. As these models get larger, the demand for computing power and storage increases as well. This is where parameter efficient fine-tuning (PEFT) comes in. PEFT is a technique that allows us to fine-tune a pre-trained model with only a small number of extra weights, while most of the pre-trained model's parameters are frozen. This way, one doesn’t have to train the entire model from scratch, but rather they can fine-tune it with just a small amount of additional training data.

PEFT also helps in preventing catastrophic forgetting, which is a problem that occurs when a model forgets what it has learned previously when fine-tuning with new data. This happens when the fine-tuning is done too much, causing the model to forget some of the original data it was trained on. With PEFT, one can freeze the original parameters and only fine-tune a small number of additional weights, which helps in preventing catastrophic forgetting.

Additionally, PEFT allows to fine-tune large language models with small amounts of data and helps in generalizing the model to specific scenarios. It also results in smaller checkpoints, which makes it easier to store and use the fine-tuned model.”

Let’s talk about Implications and Opportunities!

As I see it, innovation is flourishing outside of incumbent companies as well, and if anything the flurry of updates in the open-source world points to the fact that new companies can create value across different layers - one of the core reasons being that the potential customers are themselves tackling some core problem statements. We only need to ask whether there are structural shifts that startups can bring. There are a couple that I have been asking myself.

1. Can the new application companies create Native AI workflows instead of turbocharging existing ones?

2. How much core infrastructural changes we are likely to see if LLMs do sit in the enterprise production environment?

Applications: Re-Imagine Workflows (and not Turbocharge Existing Ones)

I meet a lot of (Generative AI Application) companies that are leveraging LLMs/FMs to simply turbocharge the existing workflows (i.e. enabling productivity within the same workflow) - I believe that many of these companies are likely going to be toast, unless they are the first of their kind in that specific vertical or have access to proprietary datasets. For instance, if you are developing a product that enables knowledge workers to analyze and synthesize PDFs, you may encounter tough competition from established companies that possess considerable distribution and are incorporating Generative AI into their product roadmap. For example, Box, which is used by over 100K companies globally, announced a similar offering over the last weekend and may have dashed a few dreams.

Case Study: A company could use an AI Native Approach to rival incumbents (in a hyper-competitive space)

From a design principle perspective, the choice between turbocharging existing workflows and reimagining new ones is not a matter of either/or. Instead, it exists on a spectrum. One good example I want to discuss is an agent-led product that helps knowledge workers achieve their goals. In the past 6-12 months, a significant number of companies have emerged in the space of boosting the productivity of marketers by enabling them to create high-quality marketing copies and other content pieces. Typically most of these tools help in the copy creation part but remain fairly divorced from other core parts of the marketing journey. To create significant value in this market, a new company may need to take a different approach. One such approach could be the use of autonomous agents to achieve marketing goals. These agents can create tasks for themselves, reprioritize their task list, and update their loop until the objective has been met. Below is the framework for a Social Media Autonomous Agent that one can think of

1. Initialize Goal: Set up the initial parameters, such as the target audience, social media platforms, content categories, and posting frequency.

2. Data Collection: Collect data on past social media posts, user interactions, and platform-specific trends.

3. Content Analysis: Analyze the collected data to identify patterns, popular topics, hashtags, and influencers relevant to your target audience

4. Content Creation: Based on the analysis, generate content ideas and create social media posts tailored to the platform and audience preferences

5. Scheduling: Determine the optimal time to post each piece of content based on platform-specific trends, audience activity, and desired frequency. Schedule the posts accordingly.

6. Performance Monitoring: Track the performance of each post in terms of engagement metrics, such as likes, shares, comments, and click-through rates. Gather user feedback.

7. Iteration and Improvement: Analyze the performance data and user feedback to identify areas for improvement. Update the content strategy, creation, and scheduling processes to incorporate these insights. Iterate through steps 2–7 to continuously refine the social media management system and improve its effectiveness over time.

One could argue that this has yet to play out, but the broader point is that we need to consider completely new workflows if we are to take advantage of the tailwinds. Having said that the quality of products in the market is not of high quality yet. I believe that creating such an experience is not easy - understandably so due to the stringent feedback loop required to execute any tasks. Below is the visualisation of the framework.

The additional nuance is that, for various consumer use cases, ChatGPT may want to create agent-led products through its plugin architecture (and ecosystem). One way to look at this perspective is by understanding how your new company can create value - An agent-based product that completes tasks like booking a table at a restaurant is likely to face significant challenges from the ChatGPT plugin ecosystem. On the other hand, an agent that executes customer service requests for an enterprise, such as prioritising key accounts or automating workflows completely, is likely to fare better.

Case Study: An application creating 1st of its kind use case driven by generative capabilities

Let’s talk about gaming, which many believe will be significantly impacted by Generative AI. In most games, there are Non-Playing Characters (NPCs) that are not controlled by the players. NPCs, or non-player characters, are particularly crucial in creating immersion in video games

NPCs are typically created using decision trees, which are limited by their reliance on predetermined rules and conditions. This often results in repetitive actions and behaviours, and may not account for all scenarios, leading to poor user experiences. The increasing demand for intelligent NPCs among gamers has led to the emergence of more advanced AI NPC solutions that are poised to overtake decision trees. Most gamers desire more personality from NPCs and are willing to invest more time and money in games with advanced AI NPCs.

To create a more autonomous non-player character (NPC), Generative AI can be leveraged to allow NPCs to generate their own responses based on context and previous interactions with the player. This can create a more immersive and engaging gaming experience. Machine learning algorithms can enable NPCs to learn from their mistakes and adapt their strategies accordingly. This can make battles with NPCs more challenging and unpredictable. Below is a basic architecture of one of the platforms enabling this (called GAEA), developed by Parametrix AI

The Infra Layer: LLMs in Enterprise Sandbox

Historically, the traditional data stack has (predominantly) supported structured / semi-structured data (such as data stored in data warehouses, third-party applications, and others). However, with the proliferation of LLMs, there is now a possibility to process and query unstructured data, such as system logs and messages, to uncover key insights and take action. If LLMs were to exist within an enterprise sandbox, the emerging stack would broadly consist of a data pre-processing pipeline, vector embeddings, LLM (itself), and a programming framework to stitch everything together (Langchain). I will attempt to illustrate the new emerging stack, where a private LLM exists within an organization and has access to all data sources..

1. Unstructured data, such as Slack messages, is broken down into smaller chunks

2. These chunks are stored in a vector database as embeddings, which are vector representations of the text.

3. Then, when a user asks a question, the system does a similarity search in the vector database to fetch relevant chunks of information and feed it to the language model along with the question

4. This enables the model to provide an answer or take action based on both the question and the relevant information.

These applications can be used for a wide range of practical use cases beyond just answering questions.

This emerging stack is still very ambiguous, and we are still unsure whether this new stack will crystallise or not, or when it will do so. Suppose there was a world where LLMs could be leveraged, and unstructured data could become useful in making business decisions. In such a world, there are some key questions that need to be answered.

1. Is there a need for data pipelines?

2. What kind of vector databases would be better suited for broader use cases (say a Pinecone or PGVector)? There are a significant number of companies receiving funding in this area.

   1. Pinecone is a cloud-based vector database and search service that allows you to store and search large-scale vector data. It is designed to handle high-dimensional vector data and can scale to millions or billions of vectors.

   2. PGVector, on the other hand, is an open-source library for vector similarity search that is built on top of PostgreSQL. It allows you to store and search vector data using PostgreSQL's familiar SQL interface. PGVector is designed to work with lower-dimensional vector data, typically ranging from 10 to 100 dimensions.

3. Should you rely on third-party, general-purpose LLMs (fine-tuned for specific tasks), or build your own task-specific language models?

Many things are developing rapidly, but I strongly believe that new companies with significant defensibility can still emerge. While it's true that incumbents are moving quickly, they inherently focus on creating closed-loop systems, and not all closed-loop experiences are good.

Although these platforms are taking steps to cut costs in order to rationalize operational and capital expenses, their path ahead to maintain their distribution moats is not as straightforward. The key dilemma here is how to take advantage of some of the generation-defining opportunities that may arise in the near future - which by all means can be expensive in the short term. Most of us are aware about the expensive play-books that these companies have run to ensure their stronghold over distribution.

1. For eg. Alphabet pays Apple a whopping ~US$20B annually to ensure that Google Chrome is the default browser on Apple devices.

2. Microsoft executes its bundling strategy with finesse, making high-quality companies seem like features in comparison to its suite of products (Remember Slack vs. Teams - Microsoft won that war a long time ago - They offered the product almost for free!)

These companies encounter a diverse array of opportunities (as well as challenges), ranging from potential disruptions in the search business model to the emergence of lucrative acquisition prospects. To maintain their competitiveness, especially in controlling distribution, these platforms must remain adaptable and constantly search for new ways to innovate and expand. However, this process is likely to be costly. Moreover, the financial performance of these companies is currently lackluster. Figures relating to revenue growth and profitability are less than impressive, which exacerbates this dilemma.

Alphabet

In Q4'22, Alphabet's revenue increased by 1% YoY. However, the company's overall net profit margin decreased significantly from ~27% to ~18%. The Search business that represents ~56% of the revenue saw a decline of ~2% YoY. The losses from other bets continued to increase. These bets included initiatives such as Deepmind. In an effort to control costs, the Alphabet team announced a reduction of approximately 12,000 jobs and a change in the useful life of their servers and network equipment costs. In January 2023, Alphabet completed an assessment of the useful lives of their servers and network equipment. This resulted in a change in the estimated useful life of their servers and certain network equipment to six years (from five years). They expect this change to reduce depreciation by approximately $3.4 billion for the full fiscal year 2023 for assets in service as of December 31, 2022. The reduction will be recorded primarily in the cost of revenues and research and development (R&D) expenses. Google indicated that it will take time to execute on cost rationalization, more clearly seen in 2024 (than in 2023).

Meta

Meta's revenue for Q4 '22 declined by ~4%, and its net profit margin dropped from ~32% to ~14%. To optimise costs, Meta has lowered its Capex guide for 2023 from US$34-37B to US$30-33B. This reduction in capex reflects their updated plans for lower data center construction spending as they shift to a new, more cost-efficient data center architecture that can support both AI and non-AI workloads. The current capex is due to building out AI infrastructure, and future capital expenditure outlook will depend on the ROI of the AI projects. The best manifestation of the dilemma (of either being the Best Technology Platform vs the Most Efficient Technology Platform), which we will discuss later in the essay, is represented through Zuck's statement below.

“And I do want to continue to emphasize the dual goals here of making the company a better technology company and increasing our profitability.

I think it's also really important to focus on the first one of just making it a better company because that way, even if we outperform our business goals this year, I just want to communicate, especially the people inside the company that we're going to stick with this”

Amazon (AWS)

AWS experienced a significant decline in year-over-year (YoY) growth, dropping from ~40% in Q1 to ~20% in Q4'22. The margin profile (net profit margin) also significantly degraded from ~30% to ~24%. While one of the core values of cloud solutions is their ability to scale up or down as needed, this level of cyclicality is sort of unprecedented, resulting in a high deceleration of growth. One particularly noteworthy metric is the incremental operating margin, which has been shockingly dropping. In Q4 '22, incremental operating expenses exceeded incremental revenue, indicating that AWS may have overestimated demand. Andy Jassy - Amazon’s CEO - also reiterated the same thing in his defence during the earnings call.

“It’s one of the advantages that we’ve talked about since we launched AWS in 2006 of the cloud, which is that when it turns out you have a lot more demand than you anticipated, you can seamlessly scale up. But if it turns out that you don’t need as much demand as you had, you can give it back to us and stop paying for it. And that elasticity is very unusual. It’s something you can’t do on-premises, which is one of the many reasons why the cloud and AWS are very effective for customers.”

While Capex intensity has nearly doubled over the past five years, it is difficult to draw any conclusions from this, as much of it may be due to other investments whose economics are difficult to ascertain.

I think at this point it’s safe to say that most of the large platforms are facing efficiency challenges, and these are likely to persist. So what exactly is the conundrum?

The potential imploding dynamics of Search Business | Distribution vs Margins

All of us are fairly attuned with Microsoft’s partnership with Open AI to integrate Chat GPT (and its other products) into its gamut of offerings including Bing. As we had discussed earlier, the search business represents roughly ~56% of Alphabet’s revenue and even limited change in share could lead to increase in Microsoft’s Bing revenue by a couple of billions of dollars. Most of the people have seen the tweet about Satya Nadella talk about making the Gorilla (aka Google) dance. However, as analysts of markets and businesses, we should be aware that this will have a significant impact on gross margins (There’s a reason we don’t quite often see a Gorilla dance, coz it’s freakin expensive!). This is not just for Microsoft, but also for Alphabet if it chooses to integrate ChatGPT, like LLM, into its search capabilities. The real choice here (for platforms such as Google or Microsoft) is to prioritise either “Defending the Distribution”, which may adversely impact the margin dynamics in the short to medium term, or “Focusing on protecting the Margin Dynamics”. The reality is that both goals cannot be achieved within a short time frame.

The process of feeding data points into a machine-learning model to generate an output is known as Machine Learning Inference. However, the cost of inference far exceeds the cost of training a machine learning model. In the case of ChatGPT, the inference costs exceed the training costs on a weekly basis. Dylan Patel has done a great job of estimating the cost of integrating chatGPT-like LLMs into search. He estimates that doing so (as it is) would cost anywhere north of US$30B. To put things in perspective, Google's net annual income would drop from US$55.5B to US$19.5B. Obviously, this is not going to happen, but it does imply that the gross margins of the Search business are likely to be impacted in the short to medium term. There was a slight hint of exorbitant inference costs for ChatGPT when Elon Musk asked about the cost of running ChatGPT and Sam Altman, CEO of OpenAI, responded by saying a couple of cents per chat.

Google runs approximately 320,000 searches per second. In 2022, its search business segment had a revenue of ~US$162 billion. This means that, on average, each query generates 1.61 cents in revenue. Google incurs huge expenses for computing, web crawling, model deployment, and other services. Google's services business unit enjoys a relatively high operating margin of ~34%. When this margin is applied on a per-query basis, the average cost to run a query is ~1.06 cents. Therefore, the cost of a query with an LLM must be significantly less than 0.5 cents; otherwise, the search business will become unprofitable. Implementing an LLM hastily would increase this cost to 1.42 cents per query, thereby completely altering the margin dynamics for the search business. On top of this, basis Dylan’s model, deploying ChatGPT into Google would require ~512,820 A100 servers with a total of ~4.1M A100 GPUs. The cost of these servers and networking equipment alone exceeds ~US$100B. In hindsight, Satya Nadella's (below) statement during the earnings call makes a lot of sense.

Obviously, one can argue that not all searches are suitable for a chat interface (One may want recommendations in image forms if they are searching about touristy places!). However, the focus is not on estimating the exact potential margin impact but rather on how to protect the margin profile / net income. Companies such as Alphabet will need to react to ensure that the margins of their search businesses are not affected in the long term. Below is a basic scenario analysis of the impact on gross margin based on the percentage of searches conducted through a chat format.

There is another issue to consider: as search quality improves, there is a probability that search volume may decrease. This is because LLMs may be able to provide exact responses to user queries, which could potentially reduce the number of search queries in the first place. Additionally, deploying ads in a chat format is very challenging. With so many issues, it is not surprising that Google has been cautious in its launch of Bard, its own LLM-powered search engine. Bard is built on a significantly lightweight version of LaMDA and requires much less computing power - hence, lower inference costs. This allows them to protect their margins, as they cannot afford to deploy really large models that would significantly erode their search earnings. There are two implications of going with this decision (of going with the lightweight model)

1. Lower Latency (i.e. coming with output quickly, where Google has an advantage): Most LLMs experience significant latency in generating output. Models trained on larger batch sizes, while still maintaining efficient GPU utilization, are likely to suffer from even greater latency. Google’s Bard would likely have an advantage in this regard - compared with ChatGPT.

2. Comprehensiveness of Answers to Queries (where Google is at a disadvantage): Sequence length refers to the amount of context that an LLM can analyze. For instance, ChatGPT’s architecture allows it to review up to 2000 to 4000 words to build the context. Based on all estimates, the sequence length of Bard is approximately half of it, which implies that it is not as comprehensive.

In summary, Google has an advantage in quickly providing answers, but Microsoft Bing's ability to process more information potentially makes it more comprehensive (doesn’t mean it’s more accurate!). Moving forward, Google's goal is to provide highly accurate search results, process more information, and deliver answers with limited latency, all while maintaining healthy gross margins. The jury is out on how long it will take Google to achieve all of these goals, particularly given that some of these goals are significantly divergent. Having said that It’s never a good idea to underestimate Google - My sense is that there will be margin pressure and an increase in capex commitments in the short to medium term. However, claiming that Microsoft has completely changed the game seems far-fetched.

The end of zero interest rates creates opportunities for acquisition for Platforms

When it comes to cloud, most of the 1st phase focused on migrating workloads to the cloud, Phase 2 is now all about managing the cloud operations - securing, operating, and managing the cloud to make it more efficient. As spending becomes more rationalized, enterprise clients may try to consolidate their software vendors. Large hyperscalers, such as AWS, GCP, and Azure, may pose significant platform risks for companies built on top of them. The mentality (of hyperscalers) to cater to 80% of problems of 80% of the customers gave rise to really large software companies. The hyperscalers never bothered due to the inherent tailwinds towards cloud migration (even if that meant leaving large value pools to relatively smaller companies). However, the shift towards the end of zero interest rates has the potential to change the ball game.

Long-term valuations are a function of discounted rates. When interest rates rise, discounted rates increase, resulting in lower potential valuations. Most SaaS companies rely on the services of cloud providers such as AWS, GCP and Azure. Companies without a product-market fit are likely to close, while those that have one are likely be asked about the size of the market opportunity. All these factors combined together do present a lucrative opportunity for these large platforms for high-quality acquisition prospects (at a much sober price). If platform risks are indeed real, then this maybe one of the years in which it can play out.

Demand side softness for small and large companies is taking many forms. For instance, software sales cycles are now longer, and more approvals are required to close deals. Discounts are also becoming more prevalent, and payment terms are more flexible, which will negatively impact cash collections. For software companies with usage-based models, customers are trying to ration usage. Companies, which now have less capital and are operating at a higher cost, will need to either rationalize costs and "do more with less" across the board in order to survive or get acquired.

Case Studies on Dropping Valuations

At this point, I would be stating the obvious if I were to explain the chart below, which shows the percentage decline in NTM revenue multiples of public software companies. But even if we were to take Snowflake as case study, which by all means was a darling of public markets and trades at the highest multiple of any SaaS company - with ~US$1.5B of ARR, growing at ~102% YoY at that scale, Net Dollar Retention Rate of ~178%, Magic Number of 1.3, and Rule of 40 of ~109% - the multiples are down by ~73% from the peak.

Miro is another interesting case study - As of October 2022, the company has raised $476 million in cumulative funding from firms such as Accel, Atlassian, and Salesforce Ventures. Their Series C funding resulted in a valuation of $17.5 billion, which represents a multiple of 58x their 2021 revenue of approximately US$300 million. In the public markets, collaboration companies such as Zoom, Asana, and Monday.com have seen their revenue multiples collapse from a high of 119x, 89x, and 84x, respectively, to their current levels of 3.5x, 6.5x, and 5.7x as of October 2022. If Miro starts to experience a slowdown in growth similar to what other remote work companies have experienced, its high valuation will be called into question.

Good Tooling is No Longer Enough: Enterprises Need a Bundled Strategy

Battery recently concluded a survey of top CXOs across large enterprises to understand the spending priorities of companies and one of the interesting takeaways was that while bottoms-up technology adoption is still happening, there are some interesting nuances - Although the hurdle to enter the enterprise sandbox is relatively low, it is considerably more difficult for new technologies to expand into the production environment.

1. ~75% of the enterprises allow developers to self select the tools for dev/test environments

2. But only ~14% allow such procurement in the production environment

Moreover, most CXOs are interested in streamlining vendor usage rather than reducing headcount. Those companies that can execute the bundled strategy well may have a strong advantage - In cases like these, platforms would have an undue advantage over companies that specialize in specific problem-solving.

Based on the observed behavior, I anticipate that following outcomes are likely to occur in one way or another. We will revisit this at the end of the year to see if any of these events have occurred.

1. I believe that the Gross Margins and Operating Margins of these platforms will be negatively affected in the next two to three quarters before stabilizing (especially for companies such as Alphabet). While the impact may not be as significant as the math we did above, there is likely to be some downward pressure on margins. The improvement in margins over the long term will be driven by both hardware and software improvements. While Google’s TPU (hardware) is generally considered better for inference workloads and can reduce the cost significantly, software improvements such as Sparsity, Pruning, Confident Adaptive Learning Model (CaLM) can lead to additional improvement by magnitudes. Unlike many market commentators, I personally believe that Alphabet's response to Microsoft so far has been measured in a good way. With the pressure to rapidly roll out products, companies are forgetting to perform basic due diligence on the output that their models produce. In contrast, Alphabet's response has been more responsible. It’s another story that Google had a PR disaster (and Microsoft did not) due to erroneous outputs of their Models.

2. Yes, the tailwinds for lucrative acquisitions exist, however these platforms - and even large companies - are likely to be cautious with their strategies, given that many have revised their forward-looking capex estimates downward. I believe they will wait for efficiency metrics to kick-in before prioritizing acquisitions in categories where they believe large value pools can be accrued (and enterprise clients are looking to bolster their spending) and consider those categories as part of their future growth strategy. For eg. Acquisition of a Cybersecurity company by a Data Observability Platform such as Datadog would make a lot of sense given the whole shift left movement of Cybersecurity during the software development process. Below are a couple of areas, where CXOs of large enterprises want to spend a significant amount of budget and resources.

3. Additionally, companies built on top of hyperscalers, such as AWS, GCP, and Microsoft Azure, may collaborate on co-selling engagements together. For example, Elastic, a data observability platform, has been strengthening its ties with cloud providers. Following is the excerpt from its earnings call touching the same subject

   “Like other independent software companies, Elastic has been deepening their relationships with the hyperscalers recently. A few years back, this was not the case, as the hyperscalers (particularly AWS) rolled their own versions of popular open source projects. Partially due to licensing changes, but primarily due to customer preferences, the hyperscalers have reversed their competitive positioning with Elastic and now embrace the Elastic sales team in co-selling engagements. Elastic has forged strong partnerships with AWS, Azure and GCP at this point.”

Please feel free to reach out if you would like to discuss anything. I will try to respond within 48 hours.

It goes without saying that all opinions expressed are personal! 🙂

Most likely, if you are reading this essay, you are likely to hold different opinions about the companies I have mentioned or maybe none at all.

1. You're likely to associate NVIDIA with gaming, but it actually specializes in machine learning.

2. There is a high probability that you were unaware of the impact of TSMC's (Taiwan Semiconductor Manufacturing Company) contribution globally (before the China Chip ban). From iPhones, and GPUs to fighter jets, its microchips power most computing devices.

3. About ASML - well, let's say it's the most sophisticated technology company we've never heard of.

Even before we dive deep into the subject, I'd like to spend some time discussing why we are covering this subject in the first place. One of the interesting things that has happened in the past few months is a Cambrian explosion of Generative AI applications. This is obviously driven by the fact that the availability of computing power has led to an increase in model size by ~15.5KX in the last three years. The big breakthrough in deep learning was the invention of Transformer models. These are a type of models that are very good at processing and understanding natural language, such as the type of language that people use to communicate with each other. Now guess the hardware that powers such models - It's the GPUs built by NVIDIA (and obviously a few others). They are the workhorses of the AI computing industry and are usually considered the gold standard. The company that builds the chips that go into NVIDIA's GPUs is Taiwan Semiconductor Manufacturing Company (TSMC). I won't waste your time with another trivia, as you may already be able to guess the answer. ASML is the only company that builds EUV machines that power the photolithographic operations of the chip-making process in TSMC's Fabs. Photolithography is the most critical process of chip making. Below is a good illustration of the most unique supply chain we have ever (or probably never) heard of.

NVIDIA - The Machine Learning (and also the gaming) company

There was a huge backlash in the gaming community post NVIDIA’s GTC keynote last year. A part of the frustration came from NVIDIA’s ever-increasing graphics card prices, but an even bigger part came from their strong focus on the Datacenter business - the one that caters to AI and machine learning applications. Let me throw an interesting stat around this - Jensen Huang, CEO of NVIDIA, spent precisely 20 minutes on the gaming division of the 97-minute-long presentation. This is when the gaming division still contributes to ~45-50% of the revenue that NVIDIA generates. Interestingly there is a reason towards this sharp tilt - the Datacentre division has grown ~3X in terms of annual revenue in the last three years and is expected to contribute a lion's share as AI/ML applications get more integrated. So how did NVIDIA reach here?  

NVIDIA came up with the first programmable Graphics Card and CUDA (NVIDIA’s software system to program GPU), making it incredibly easy for programmers to write embarrassingly parallelized software (as all modern AI software is). It ensures forward and backward compatibility so that older chips can still run newer software and vice versa. Let me explain this in simpler terms - Typically the CPU (the power horses of computers) fed information to Gaming Cards and pixels came out as the product. In reality, the GPU was doing a tremendous amount of basic computation but the computation was completely married to the specific game library. As the games/library evolved, those graphics cards were rendered useless, so NVIDIA created a programmable graphics card. Interestingly NVIDIA had to create a programming language and the whole architecture around this called CUDA (because obviously, no one had imagined this). It was probably after six years of inception that NVIDIA realised that there was another use case for these programmable graphics cards - i.e. powering AI and ML infrastructure. 

In simplest terms, the CPU of the computing device is the brain, which can execute highly versatile tasks and assign tasks. However, it has a limited number of cores implying it cannot do multiple things at a time (try running slack and working on 1GB excel file in parallel and then come chat with me). GPU on the other hand can do very specific tasks but a lot of them in parallel. Most of the CPUs have 4 cores (and the most powerful of them has probably 48 of those) but NVidia’s top-of-the-line GTX 1080 Ti GPU has over 3,500 shaders running at nearly 1.5 GHz., implying that it can execute over 3,500 individual programs at speeds approaching CPUs. One of the best examples of an embarrassingly parallelised problem statement is how a GPU can be used for encryption. If you want to discover a key for encryption, you probably pick up a combination and check if it is right or not. Now CPUs are really smart and can handle all sorts of logic but most of the CPUs in the world have 2 to 4 cores - entailing that it can just guess 4 combinations at a time. While a GPU can perform a narrower set of tasks - a GTX 1080 can guess 3500 combinations at a time. GPUs are just better at repetitive tasks. It just so happened that a lot of machine learning is just like that (and yes Ethereum mining as well!).

As a result of this development, NVIDIA built the overall infrastructure for GPU computing (and hence AI infra) - a higher-level language, APIs, and compilers for data centre operators. NVIDIA also built the programming platform CUDA to program shaders (The Cores of a GPU). Using the CUDA platform, programmers can access the parallel processing power of NVIDIA's GPUs without having to learn graphics programming languages (such as OpenGL). The reason it is necessary is that parallelizing machine learning is highly essential. If you love the subject - I highly recommend reading Ben Thompson’s essay (Link) that he published last year. The essay talks about the three levels of the NVIDIA Stack and how it enables the most computationally intensive processes - i.e. from training trillion parameter AI models to Computational Biology

TSMC - A generational defining company that transformed the chip-making industry

We have talked a lot about NVIDIA, but let's just say that an NVIDIA-like company exists only because of TSMC. Earlier companies such as Intel designed, manufactured, and sold chips as part of a full stack approach. Getting into chip-making was extremely hard because of huge barriers to entry driven by insane capital intensity dynamics. TSMC unbundled the whole process and became “Chip Manufacturer as a Service” for Fabless companies such as NVIDIA, AMD, Apple and others. These companies designed their own chips and worked in conjunction with companies such as TSMC to manufacture them at scale. (Fun fact: NVIDIA to date has only raised US$20M as part of private funding). Last month, Apple's CEO Tim Cook and US President Biden inaugurated one of TSMC’s newest Fabs in Arizona, where the company plans to invest over US$40B in the coming years (Fab is the actual building where semiconductors are made; Foundries are companies that sell that building’s production as a service). Additionally, you may be aware that the Biden administration enacted sweeping export controls to undermine China's ability to manufacture leading-edge nodes (aka the most advanced chips). Companies such as TSMC, ASML, Lam Research Corp were either directly or indirectly impacted due to these restrictions, requiring them to halt any shipments related to advanced chips. So how did we reach here?

Manufacturing Chips and Replicating Semiconductor Supply Chain is incredibly hard

It's probably one of the few industries where suppliers command huge power due to the unique capabilities required to manufacture chips. Obviously, this is coupled with the mind-boggling capital expenditure required to create the infrastructure to produce advanced nodes (aka chips). But you just can’t buy your way into making chips. Below is a chart that lists the number of companies that can produce lagging edge nodes (say chips that go into say your vehicles | >60NM) vs leading-edge nodes (say chips that go into your Apple Macbook, NVIDIA’s GPUs). Lesser the gate length, the more the number of transistors that can be packed into a chip, enabling it to be much more powerful. For eg. a 5NM chip is much more powerful than a 28NM chip. The number of players with leading-edge manufacturing capabilities is reducing with time driven by the complexity of building high-quality semiconductors. Fun fact for you: TSMC is the only company that can produce 5NM chips at high-quality throughput (more on this later). Additionally, TSMC, most likely, will be the only company that will be able to build 3NM chips in the coming years (and maybe only followed by Intel which is typically 3-5 years behind). Let's take a closer look at some of the reasons for this phenomenon. 

Its Freakishly Expensive

Fabs are incredibly expensive to build (north of ~US$12B in Capex for 20K wafers per month). Although venture capital was used for the first time to fund chip companies, it seamlessly transitioned to the software industry. As a rule of thumb, the entire Fab facility depreciates over the span of five to ten years until the cost of producing a chip reaches a marginal level. For Fabs to be economically viable, they should be operated for as long as possible. For eg, Fab 2 of TSMC (that is a 150NM facility) was built in 1990 but is still up and running. it’s among 6 of the 26 facilities that are over 20 years old. It is true that the 150NM chips don't sell for a high price, but almost all the revenue generated through sales is pure profit since the Fab has fully depreciated and the marginal cost of producing a chip is negligible. ASML in its annual report published the Capex that will be incurred by Chip Manufacturers (such as TSMC) on the upcoming fab facilities (and buoy it’s expensive!). The price tag of US$12B on the Arizona facility is only the first phase, which will produce 20K wafers a month. As the project progresses, more capacity will be added      

The Capex can be divided into the key broad categories - The construction of the building itself along with the Clean Room (where the actual manufacturing happens), The Purchase of Equipment, which is super expensive, The utility costs and The Labor costs in building the whole facility (along with the utilities). Let’s further try to estimate the CAPEX profile.

1. Equipment Costs: Photolithographic equipment accounts for most of the equipment costs. The EUV machines are responsible for printing transistors on the wafers. There is only one company that builds such equipment (ASML). Each piece of equipment can cost up to US$300M and a Fab may need around 15-20 of these. Production cycle time determines the number of machines. The more machines there are, the shorter the production cycle time. As TSMC caters to the world's largest fabless chip companies (such as NVIDIA and others), they need to have lower cycle times.

   1. Deep Ultraviolet Machines costs - US$100M to US$150M a piece

   2. Extreme Ultraviolet Machines Costs - US$200M to US$300M a piece depending on the node 

2. Labour Costs: Construction of a Fab of this capacity typically takes 4-5 years. There will be an additional two years of work to set up the tools, plan utilities, portion clean rooms, etc.

3. Building Costs: This involves the construction of Clean Rooms, which is the actual place where semiconductors are manufactured. This typically costs ~US$1B to US$1.5B. Additionally, one needs ~2-4M Gallons of water per day to produce wafers. So a huge recycling facility is needed in order to reuse the water. There needs to be a facility to convert raw water into ultra-pure water for the process to work. Setting up this facility can cost more than US$800M on top of incurred expenses on construction.

4. Utility Costs: This involves maintaining the classification of the clean room - No dust or temperature, and adhering to special installation requirements during the installation of the machine. Additionally, all other resources when the construction phase going on

One may argue that one can buy their way into building this infrastructure but it’s nearly impossible (especially advanced nodes) to replicate the supply chain needed to produce Chips at scale.

Almost impossible to replicate Supply Chain

It’s probably one of the very few industries where there are far fewer suppliers than the customers (Note that these customers include the likes of Apple, NVIDIA and Others). Manufacturing advanced nodes is a long learning process. It's one of those industries, where customers invest in their suppliers. Like ASML (the company that manufactures machines installed in Fabs) required a heavy co-investment from their customers (TSMC, Intel, Samsung), Fabs require co-investment and risk sharing of capacity with their customers. The semiconductor industry has moved towards the NCNR model (Non-Cancellable, Non-Refundable Model) due to the expensive nature of the business. This kind of environment just cannot be created by spending tons of money. For eg. both ASML and TSMC have to work together constantly to ensure that the chip production just works fine. There is a concept of risk production phase when a node is launched. In this phase, which typically lasts for 3 - 4 years,  the yield is extremely poor and it is improved constantly. By most estimates, this hovers around ~25% to 30%.   Typically many customers (such as NVIDIA, AMD and others) are supposed to give small prototype orders to TSMC. The processes (and steps) are optimised and pieces of equipment are modified (by suppliers such as ASML) in order to improve the yields and make the yield reach up to at least 80%. It's very interesting that customers are also expected to assist foundries like TSMC in risk production phases. The company generates revenue from prototyping (paid for by its customers). The revenue from this phase is not as high as the revenue from HVM (High Volume Manufacturing). This is due to the fact that the number of wafers being ordered will be lower. This process requires considerable collaboration between TSMC and its vendors (such as ASML). This makes it incredibly difficult to replicate this supply chain even when countries like China are capable of high CAPEX commitments. The value chain for the leading edge node is characterized by incredible supplier concentration. Below is a flow chart that depicts the broad value chain.

Several aspects of the value chain are serviced by only one supplier across the globe because of the sheer complexity of technology. For eg. ASML is the only company that builds EUV Machines (Extreme Ultra Violet Machines) required to print transistors in an advanced node. To give you a more detailed view, here is a simplified map of suppliers by each part of the value chain.

Software used to design the chips 

There are only two companies (Cadence or Synopsys) in the world that create software solutions for chip designing. Fabless companies use these software solutions to design their chips. Furthermore, both companies are based in California, so your supply chain is always dependent on U.S. goodwill.

Photolithography Process powered by ASML (The most innovative company you have never heard of)

There is only one company in the world that manufactures extreme ultraviolet (EUV) lithography equipment. It's called Advanced Semiconductor Materials Lithography (ASML). Photolithography is a process of using light to print transistors on silicon wafers. However, there is one fundamental problem - The wavelength of visible light is 193NM, so how does one print a transistor with gate length <= 5NM? It is ASML that builds these EUV machines that can do this. The technology is so complex that it only became commercially viable after 30 years of conceptualization (in 1986). To put things into perspective, a EUV machine is made up of more than 100K parts and It takes 4 Boeing 747s to transport one machine. The ASML crew works in a deeply integrated way with the Foundries to install and make these machines work. Below is the process of lithography described by the Brookings.

“An extreme ultraviolet lithography machine is a technological marvel. A generator ejects 50,000 tiny droplets of molten tin per second. A high-powered laser blasts each droplet twice. The first shapes the tiny tin, so the second can vaporize it into plasma. The plasma emits extreme ultraviolet (EUV) radiation that is focused into a beam and bounced through a series of mirrors. The mirrors are so smooth that if expanded to the size of Germany they would not have a bump higher than a millimeter. Finally, the EUV beam hits a silicon wafer—itself a marvel of materials science—with a precision equivalent to shooting an arrow from Earth to hit an apple placed on the moon. This allows the EUV machine to draw transistors into the wafer with features measuring only five nanometers—approximately the length your fingernail grows in five seconds. This wafer with billions or trillions of transistors is eventually made into computer chips.

EUV lithography technology has been in development since the 1980s but entered mass production only in the last two years. Other companies make older generations of lithography machines that don’t use EUV and can only make older generations of less cost-effective chips. These companies include venerable firms such as Nikon and Canon. They have the experience, expertise, and market discipline that comes from decades of profitability in a competitive industry under extreme technological demands. If these companies could make EUV machines, they would—it would make them billions of dollars. This is also why, after more than 30 years of development and billions of dollars in R&D, ASML still faces such a backlog of orders: They are hard to make. EUV machines are at the frontier of human technological capabilities.”

Such is the complexity of technology that there is only one company in the world that produces mirrors to deflect Ultraviolet Radiation. The name of the company is Zeiss (and it is based out of Germany).

A country like China (which is well capable to commit insane amounts of capital) will need to create a whole ecosystem of chip manufacturing processes, including really high-quality talent with hyper specialisation. This ecosystem would consist of TSMC, ASML, Lam Research, Applied Materials, and Tokyo electronics. They may have to go a level deeper and create an ecosystem of companies such as Zeiss.

So what’s the future?

The proliferation of AI across civilian and defence use cases entails that there will be likely a huge expansion at the compute layer. Most of the tech monopolies are trying to follow a vertical approach. These companies are trying to create their end-to-end AI infrastructure (like the one created by NVIDIA, powered by TSMC and ASML).

Google, too, has invested in hardware - the TPU (Tensor Processing Unit) has been designed specifically to work with Google's TensorFlow network. TPUs are an effort to capture end-to-end AI stack. TPUs are application-specific integrated circuits that accelerate machine learning workloads - providing AI services on Google Cloud. To ensure that Google can capitalise on this technology, these TPUs can only be rented with Google cloud services. 

AWS is also building AI infrastructure. Although they are further behind the race, they acquired Annapurna Labs - an Israeli semiconductor company to build their own hardware meant to provide an end-to-end AI stack in conjunction with the AWS platform.

Countries are pushing for semiconductor sovereignty and trying to replicate the supply chain that we earlier talked about. While I believe there is a potential to localise this supply chain for lagging edge nodes (used in vehicles and other equipment), we are still further away from localising the supply chain for leading-edge nodes, which will be necessary for powering AI/ML applications. Below is the chart showing capital commitments by different countries.

The jury is still out on whether this hardware infrastructure stack can get replicated or not. Nevertheless, we can expect significant government investments to localize this hardware stack, and who knows, perhaps new companies will emerge in these environments.

Please feel free to reach out to me in case you want to debate some of these ideas!

Subscribe now