A few weeks ago I was talking to a team at a large financial company that has spent the last year building AI into one of its core operational workflows.

They had made real progress. They’d broken the process into individual checks, used LLMs to automate parts of the review, and were already seeing work move faster. Now they wanted to take the next step and let the system make some decisions without a person reviewing every case.

That’s where they hit a problem.

Some of their checks were easy to make deterministic. If you need to verify a fact against a database, you can look it up and compare the result. But other checks required reading an unstructured document and deciding whether it satisfied a policy. An LLM could do that well, but not with the consistency they needed to let it make the final decision.

So they were going back through the entire workflow, check by check, asking where they could replace an LLM with deterministic logic and where they actually needed the judgment of an agent.

I’ve had versions of that conversation across compliance, underwriting, due diligence, and research workflows over the past three years.

The prototype is rarely the hard part. Problems appear when teams run the same workflow hundreds of times and need consistent results, traceable decisions, and predictable costs.

By mid-2026, roughly 79% of large enterprises were piloting AI agents. Only 8.6% to 14% had an agent operating at true production scale, handling real task volume with monitoring and incident response behind it.

That range comes from our review of seven major surveys covering more than 125,000 respondents, including research from Deloitte, Gartner, and Sinch. Each survey defines production differently, but the gap between pilots and durable deployments is consistent.

Meanwhile, model performance improved sharply between 2024 and 2026. On WebArena, which tests agents on multi-step web tasks, the top score rose from 14.4% to 71.6%.

You’d expect those improvements to translate into more agents making it into production. They haven’t.

We examined two independent sources of production evidence. Clyro collected 591 documented agent failures, while Presenc AI analyzed public evaluations and deployment instrumentation from more than 60 enterprise agent customers. In Clyro’s dataset, 88% of failures involved infrastructure around the model. Hallucinations and other model-quality failures accounted for around 10%.

That finding matches our experience pretty closely.

Read the full Grep AI whitepaper on Why Agents Fail in Production here.

One way to understand the problem is through some simple math. Say an agent has to complete a 20-step workflow and is 90% accurate at each step. A 90% success rate sounds pretty good in isolation. Across all 20 steps, though, the probability of completing the entire workflow successfully falls to 12.2%.

At 50 steps, the probability falls to 0.5%.

Production workflows also contain dependent errors. If an agent makes a bad tool call at step three, that result enters the context for step four. The agent makes its next decision using bad information, then carries that decision into the following step. It can execute a coherent plan built on a mistake it made ten minutes earlier.

When someone shows me an agent performing an impressive task, I want to know what happens on the hundredth run, then the thousandth. I want to know how it handles an API timeout halfway through a task, contradictory sources, or a context window crowded enough to obscure an instruction from the beginning of the run.

We started Grep AI in compliance, which gave us an unforgiving place to learn how to build agents. If an agent reviews a business for money-laundering risk, “usually right” provides little value. Analysts need to know what it found, where it found it, how it used the information, and what happens when it lacks enough evidence to decide.

Our first products used GPT-4-era models. We abandoned a free-running agentic loop in favor of a pre-orchestrated agent, which we called an agent on rails. We wrote about that architecture in Agents aren’t all you need. The rails improved consistency because we defined the sequence of steps in advance and isolated the model calls.

That architecture also limited what the agent could do. Each workflow required hand-written logic, which restricted us to a small number of compliance use cases.

We added configurable steps so customers could change the order and behavior of a workflow. That made it easier to create new agents while preserving the controls we needed in production.

As models improved, the rails began to constrain the quality of the work. An investigative agent assessing negative news could open the articles it had received, extract evidence, and classify the result. It could not decide to search for another source, follow a connection between two companies, or investigate a person mentioned deep inside an article. Each step remained an independent model call because we had designed the system to contain compounding errors.

That led us to move from agents on rails to an agent in a sandbox, initially built on the Claude Code harness. We described the architecture in Claude in a box. The agent could plan its own work, use tools, write code, and pursue evidence across multiple sources. It opened up deep research, due diligence, legal analysis, and other high-stakes workflows that our previous architecture could not support.

The new agent was far more capable. It was also too expensive to run.

Earlier this year, we reviewed our Anthropic bills and found that we were spending more on tokens than the agent generated in revenue. We had to reduce the cost of each run for the business to work.

That pushed us to build our own harness around a few operating requirements.

First, the harness needed to reuse proven workflows. General-purpose coding agents plan each task from scratch because each input may require a different approach. Enterprise agents often perform the same workflow thousands of times. Once an agent discovers a reliable sequence, it should reuse that work instead of paying to rediscover the process on every run.

The harness also needed to learn operational facts. If an agent discovered that one data provider lacked business-registration coverage in the Bahamas while another provider covered it, future runs should use the right provider from the start. Some steps may no longer require a model once the system has learned a deterministic procedure.

We also added verification during execution. A check at the end of a 20-step run arrives too late if step three contaminated everything that followed. The system needs to inspect intermediate work, catch unsupported claims, and correct errors before later steps inherit them.

Finally, model selection became the most important cost and reliability control. Complex reasoning may require a frontier model. Summarizing a search result, extracting a date, or formatting an output may require a smaller model or a piece of code. The harness can choose between them at each step rather than sending the entire workflow to the most expensive model available.

The new harness is in testing, and we plan to roll it out to our largest enterprise customers. Early tests show cost reductions of 80% to 90%, with accuracy holding steady or improving. We still need more production volume before treating that range as a durable benchmark.

Technical architecture covers only part of the work. Customers ask different questions once an agent moves from helping someone make a decision to making the decision itself.

They want to know how we tested it, why it reached a conclusion, and which sources it used. They ask how it handles conflicting evidence and whether a model update changed its behavior. They want to run the agent against cases where they already know the correct answer.

Customers have pushed us to build those controls.

One customer automating due diligence for an underwriting workflow wanted to verify every material claim the agent made. We exposed citations alongside each claim, showed the underlying tool outputs, added confidence information, and made the execution trace easier to inspect.

Another customer found that asking the agent to do more work improved accuracy and auditability. It also produced reports several times longer than their analysts wanted to read. Report length looked like a product preference until analysts stopped reviewing the work. We had increased accuracy while reducing the practical value of the output.

Production teams have to balance accuracy, consistency, latency, cost, auditability, and the amount of human attention each run requires. More investigation may improve coverage while increasing cost and review time. Tighter controls may improve consistency while preventing the agent from following useful leads.

We expose the agent’s reasoning, tool calls, sources, and generated files so customers can inspect its work. We combine that visibility with versioned agent configurations, in-product evaluations, and cost controls.

Future model improvements will help. They will not repair an unreliable API integration or define the right permissions for a workflow. Teams will still need to detect policy drift, roll back changes, and decide where a person should review the work.

The financial services team I spoke with was asking the right questions. They were deciding where an agent’s judgment added value, where software could produce a certain answer, and where a person still needed to review the result.

Putting agents into production requires teams to define the workflow, constrain access, verify intermediate work, monitor changes, and make the economics hold at real volume. More capable models expand what teams can build. The reliability of the finished system depends on the choices they make around them.

You can read the full Grep research report on Why agents fail.

About the author:

I’m AJ Asver, co-founder and CEO of Grep. I’ve spent the past three years building and deploying AI agents for high-stakes work inside large companies. I write The Operator’s Guide to AI to share what we learn when those systems meet real workflows, customers, costs, and production constraints.

You can follow me on X and LinkedIn.

This piece is a snapshot of how Grep works today (mid-March 2026) and where we think this architecture goes next.

For historical context, we originally took everything we learned at Parcha and turned that into a harness for doing corporate due diligence. We wrote about that in Claude in a Box. Over time, we realized that the concept of due diligence for businesses was really a special case of something broader: there was a well-established industry forming around AI-assisted, in-depth research. So we evolved our harness into a general researcher that could operate across almost any domain.

If you want an agent to do its best work, you have to give it the right environment to work in. That means more than a list of tools or MCPs. It means giving the agent a full compute structure and the ability to take on the work required by the task in front of it, while also managing context efficiently at every step.

This industry is moving extremely quickly, and this system may look completely different in a couple of months. But we wanted to stop and document what we have, how we built it, and what we think comes next.

How GREP Works at 40,000 Feet

The easiest way to understand GREP is as one agent that morphs across stages of the research process - and when the work demands it, splits into parallel copies of itself.

GREP has multiple modes, and depending on the mode and the effort level, the shape of the system changes. Here I am describing the higher-effort mode: the one that can go deep for hours, doing as much research as needed.

This is not a multi-agent system in the typical sense. We do not have a zoo of specialized agents negotiating with each other. Instead, we have one agent that reshapes itself depending on the stage, and can spawn sub-agents that are clones of itself with narrower focus. Think of it less like a team and more like cell division - one researcher that multiplies when the problem requires it, then converges back into one voice.

The second most important point is that execution is not a single pass. It is a two-layered loop:

• Inner loop: The researcher spawns sub-agents, evaluates their results, identifies gaps, and spawns more sub-agents until it is satisfied with the coverage and quality of the research.

• Outer loop: After the report is written, reviewers and fact-checkers evaluate the entire output. If the report does not meet the quality bar - if claims are unsupported, sections are thin, or the research has gaps - the system goes back to the research phase and the inner loop runs again.

The agent decides when it is done. Not a timer, not a token budget. The agent reads the review, compares it against the adequacy criteria from the plan, and either ships the report or goes back to work.

The flow starts with the planner, which takes the user’s question, any supporting information, files, and additional context. It synthesizes all of that and begins to shape the research process. That plan is both a proposal for how the research should be carried out and an opinionated view of how the system should operate.

The planner is a researcher too. It may need to gather context through web search before it can write a good plan - very much like a coding agent exploring a codebase before taking on a large implementation task. It explores the topic, asks the user clarifying questions if needed, and then writes an elaborated plan aligned to the capabilities of the expert that will execute it.

Planning

In difficult research, the quality of the plan often determines the quality of the outcome.

Planning matters because difficult research is not just a matter of searching more. It is a matter of spending the right amount of time understanding the question, clarifying the user’s intent, identifying what is known and unknown, and deciding how the problem should be broken apart.

The planner is not just routing work. The planner is doing research. Careful planning requires context, and in some cases it requires interacting with the user to clarify the question, the user’s intent, or the context behind the request. A good planner gathers enough information to shape the rest of the system without overcommitting too early.

Expert Shaping

One of the most important things the planner does is shape what the expert will become for the task ahead. We do not think of GREP as a set of separate agents negotiating with each other. We think of it as one super-agent that morphs over time. Planning is its own phase, with its own skills and objectives, because that phase determines what the later researcher will need to be.

If the task is about financial analysis, the planner has to identify that the researcher will need the skills, tools, and ability to work with financial datasets, run analysis, and rely on reliable and up-to-date data sources. That is especially important because language models do not know everything, and more importantly, they must know what they do not know. Planning is where we decide how the system will complement the model’s general knowledge with web access, tools, command-line interfaces, and computation.

Skills, Tools, and What the Model Doesn’t Know

Planning is also where we decide which skills from our broader skill library should be loaded into the expert without overwhelming the context window. The planner selects tools, MCPs, CLIs, or specialized capabilities that might help the researcher operate more effectively. That might mean specialized news and real-time search, or it might mean coding tools, data analysis tools, or something even more specific.

The output of planning is a detailed file, and in some cases additional files with supporting structure. That file shapes the rest of the run. It includes an opinionated view of how the research should proceed, which capabilities should be loaded, what the major tasks are, what good outcomes look like, and what the final review checklist should contain.

Execution

Once the plan exists, execution becomes a problem of task shaping, context preservation, and structured synthesis. This is where the two-layered loop comes to life.

The first thing the researcher does is load the skills, tools, MCPs, and CLIs prescribed by the planner. But the researcher still has agency. It can adapt the plan, reshape it, and make decisions about how to pursue the goal. The plan is guidance, not a hard-coded script.

The Inner Loop: Research Until Satisfied

The first major execution step is turning the plan into a set of tasks. We use tasks heavily, and they are extremely powerful because they allow us to define work in a way that is specific, independent where possible, and suitable for concurrency.

This is where our architecture begins to look map-reduce-like. If the planner identifies five subtopics that can be researched independently, the researcher can spawn five sub-agents to go pursue those lines of inquiry. If a task needs a different type of expert - like a data analyst - it can spawn one with a different configuration, tools, and objective.

The inner loop does not stop after one round of sub-agents. The orchestrating researcher evaluates the results, identifies gaps in coverage or quality, and decides whether to spawn more sub-agents, deepen existing lines of inquiry, or move on. This loop continues until the researcher is satisfied that it has enough material to produce a strong report.

This is not a fixed number of iterations. The agent reads the evidence, compares it against the objectives in the plan, and makes a judgment call. Some questions resolve in one round. Others require three or four rounds of increasingly targeted research.

Context is Precious

The inner loop works because the main agent is disciplined about preserving its own context window. The orchestrating researcher does not try to hold every detail of every branch of research in its active context. Instead, it coordinates, synthesizes, and writes the final research while sub-agents use their own context windows to do the deeper work.

The file system is what makes this practical. Sub-agents do not send all of their work back through one giant message. They write files. In fact, they often write multiple files: the long analysis itself, and then smaller supporting documents that point the main agent to the most important sections. That way the orchestrator can read the summary or pointer file, decide which sections matter, and only read the relevant portions of the longer analysis.

This is one of the central ideas in GREP: context is precious. Effective use of context matters more than simply having a large context window. You need to preserve attention for synthesis, judgment, and writing - not waste it on moving large quantities of intermediate text around the system.

Tasks also help with persistence and recovery over long-running jobs. When research runs for a long time, the agent may compact and lose some active memory of the objective or the plan. The plan is always available in the file system to be read again, but tasks and task dependencies also preserve structure. They act as a durable representation of what needs to happen, what has already happened, and what depends on what.

Writing the Report

Writing the final report is more complex than it looks. Very long outputs are hard for language models to produce reliably in one pass. So instead of asking the agent to write a hundred-thousand-character report from scratch in one shot, we use what we think of as a pixelated-to-picture-perfect process.

The writer starts by generating a table of contents into a file. Then it writes thesis sentences or early scaffolding for sections and subsections. Then it expands section by section, gradually refining the report into something complete and cohesive. We do not use multiple writers stitching whole sections together because that often reads like multiple authors pasted into one document. The main writing agent owns the voice and cohesion from start to finish.

The Outer Loop: Review and Iterate

After the report is written, reviewers and fact-checkers take over. The final checklist defined in planning is read by reviewer agents, which perform their own validation, support claims with evidence, and identify areas where more research or correction is needed. They write their reviews into files.

Here is where the outer loop kicks in. The main agent reads the reviews and makes a decision: is the report good enough, or does it need to go back to the research phase? If the reviewers flag unsupported claims, thin sections, or missing perspectives, the system does not just patch the text. It goes back into the inner loop - spawning new sub-agents, gathering more evidence, and synthesizing again before rewriting the affected sections.

The agent decides when it is done. It reads the review, compares it against the adequacy criteria defined during planning, and either ships the report or goes back to work. This is what makes the system fundamentally different from a single-pass pipeline: the quality bar is enforced by the agent itself, not by a predetermined number of steps.

Benchmarks

We did not build GREP to win benchmarks, but the benchmarks made it clear we had built something unusual. They give us a way to understand how far we have come on certain dimensions and how we compare against the very companies and teams defining the category.

DRACO

DRACO is Perplexity’s benchmark. It measures research quality across 100 questions in 10 domains - law, finance, medicine, technology, and more. Each answer is evaluated on factual accuracy, breadth and depth, presentation, and citation quality.

Our high-effort agent scored 78.6%, outperforming every entrant. Perplexity’s best configuration came in at 70.5% - on their own benchmark. That’s an 8.1 percentage point lead.

The domain breakdown tells a more interesting story. Grep wins 9 of 10 domains. The only loss: Personalized Assistant, by 1.5 points.

The largest gaps are in areas that require careful research methodology: Needle in a Haystack (+12.4pp), UX Design (+14.8pp), Shopping/Product (+10.5pp), and General Knowledge (+10.1pp). These are exactly the kinds of tasks where having a structured planning and execution pipeline - rather than a single-pass search - makes the biggest difference.

Grep also leads on all 4 rubric axes: Factual Accuracy (+7.5pp), Breadth & Depth (+7.2pp), Presentation (+3.0pp), and Citation (+14.5pp). The citation gap is particularly notable - our architecture’s emphasis on evidence tracking and source verification through file-based context pays off directly.

DeepSearchQA

DeepSearchQA is Google’s benchmark. It evaluates multi-hop search problems where the system must follow several steps of reasoning and information gathering to arrive at a specific answer. 896 questions, each requiring factual correctness.

Grep scored 84.5% factual correctness. That puts us ahead of Perplexity (81.9%, the unofficial SOTA before us) and well ahead of Google’s own Gemini Deep Research Agent (66.1%), which created the benchmark.

What makes that result more interesting is that we did not achieve it with our highest-effort agent. We achieved it with our medium-effort agent. In GREP, effort does not simply mean doing more work. It often means being more focused, more deliberate, and less excessive. In benchmarks like this, excessive retrieval or overly verbose answers can actually hurt performance. GREP’s focus helped here.

The thing worth noting here is the floor, not the ceiling. The lowest-performing category, Finance & Economics, still hits 79.5% on 132 questions. The spread from best to worst is only about 13 points. That matters because it means GREP is not gaming any particular question type - the architecture generalizes. Most deep research systems have a spike in one domain and a crater in another. Ours stays flat because the planning step adapts the research strategy per question rather than relying on a fixed retrieval pattern.

DeepResearch Bench

DeepResearch Bench is an independent benchmark built by researchers at the University of Science and Technology of China and Metastone Technology. It includes 50 PhD-level questions in English and 50 in Chinese, evaluated with the RACE scoring framework across comprehensiveness, insight, instruction following, and readability.

Grep scored 56.27, edging out Cellcog Max (56.13) and a competitive field of research systems. The margins are tight at the top - 0.14 points separating first and second.

The most surprising detail: our reports were even higher quality on the Chinese portion (56.42) than on the English portion (56.13), despite the fact that we had spent effectively zero time optimizing specifically for Chinese.

That result says something important about the architecture. When you give a very strong base model the right structure, the right tools, and the right environment, you often do not need to over-index on narrow prompt tuning or specialized optimization for every domain. You give the agent expertise, you give it capabilities, and then you let it work.

Note: As of publication, the official leaderboard lists our previous harness version in second place (56.09). Evaluators are currently confirming our latest results (56.27), which are published in full at our benchmarks repo.

What’s Next

The story of GREP so far is really the story of discovering what an expert needs in order to do excellent work.

Experts Need Skills

The first thing we learned is that an expert cannot exist without specialized capabilities. That means tools, skills, command-line interfaces, and increasingly the guided ability to write code. It is not enough to hand an agent a static list of APIs. You need to give it actionable capabilities that match the work in front of it.

Experts Need Computers

Giving an agent a computer turned out not to be a convenience but a requirement. It needs to write and run programs. It needs a place to hold data, previous research, code, notes, and intermediate outputs. The file system is not just storage - it is a practical extension of the model’s working capacity.

One useful mental model is to think of the context window like L1 or L2 cache in a computer: fast, precious, and limited. The file system is more like a larger and slower memory layer. It is not as immediate, but it is much more durable and much more available. That distinction has become increasingly important in how we think about agent design.

The computer does not have to be one thing. For some customers, it is a very simple sandbox where untrusted code can be run safely. For others, it can be much more powerful. We already have customers exploring what happens when research agents have access to GPU clusters and can perform significantly more computationally expensive operations.

Experts Need Brains

We are increasingly realizing that an expert needs a brain. It needs to remember previous work, preserve useful information across runs, accumulate context over time, and eventually improve by learning from prior research. Language models by themselves do not do this well. They do not naturally become better at a job just because they have done it many times. So one of the areas we are most interested in now is building abstractions on top of them that let them remember and deepen their expertise over time.

General Tools Over Narrow Interfaces

On the skills side, we are also learning that it may be less important to keep adding handcrafted capabilities than it is to let the agent write code, execute code, and use general-purpose tools well. Unix, Bash, Python, package managers, data tooling, and the file system are often more powerful than an ever-growing library of narrow interfaces.

The theme, increasingly, is simple: give the agent guidance, give it agency, and let it cook.

A Personal Note

One of the most fun parts of this project is that it has largely been built by two people, using the system itself to run the company around it.

Parcha and GREP are down to two people - AJ and myself. Not exactly by design. Startups are very hard. But the two of us have been able to build all of this, run the business, move customers onto it, and continue innovating by using GREP itself.

We have GREP experts helping with go-to-market. We have GREP experts helping with finance. We have coding agents fixing bugs. We have a growing internal army of experts helping us research, operate, and build. In practice, it is the two of us working alongside an army of GREP experts.

Building the company this way has been incredibly fun. It also feels like a preview of what is coming next for companies more broadly: smaller teams, leaner organizations, and significantly more output per person.

If you want to learn how we are doing this, or partner with us to optimize how you and your company work with GREP, get in touch. We would be happy to share notes and give you access to some of the latest things we have been building.

We’re just getting started.

Miguel Rios is co-founder and CTO of Grep. Previously Head of Platform Engineering at Brex and Head of Consumer Data Science at Twitter. @miguelrios

Almost four years ago, right after ChatGPT launched, I started writing this newsletter, The Hitchhiker’s Guide to AI, to learn about the then-nascent AI space and share what I learned with other curious builders and operators in my network. You subscribed then because you were as curious about AI as I was.

At the time, most of us were trying to figure out generative AI, large language models, and AI agents for the first time. What could these models do? Where was this going? How much of the hype was real?

I kept writing for a while, and that eventually led me to the conviction to found an AI startup. Writing took a back seat to building.

For the last three years, I’ve been deep in deploying AI agents into production for some of the world's leading fintechs, like Airwallex, Coinbase, IG, and Worldpay. We started with AI for compliance, where the stakes are high, and mistakes have real consequences. Today, with our latest product, Grep.ai, we’re applying what we learned to a much broader category: high-stakes, repetitive knowledge work.

A few weeks ago, a customer asked me how other companies were approaching AI. What strategies were working? How were they rolling agents out? Where were they getting stuck?

I realized I answer versions of these questions all the time. I talk to teams building their own agents, executives deciding where to deploy them, and operators responsible for making them work once the demo is over. I’ve just never written much of it down.

So I’m starting now.

I’m also renaming this publication The Operator’s Guide to AI.

When I started writing, we were all still trying to understand AI. Today, many of you deploy it. You have agents in production, teams trying to scale them, a board asking about ROI, and your name on the line when something goes wrong.

How do you decide which work is ready for agents? How do you evaluate them? Where do humans stay in the loop? What does reliability look like when an agent is doing thousands of tasks a day? How do you know if any of this is creating real value?

That’s what I want to write about here.

I’ll share what we’re learning from running agents in production at Grep, what I’m seeing from companies doing their own rollouts, and the practices that seem to hold up once AI meets real work.

After three years of building, I have a lot to write down!

The first edition, which will land later this week, will focus on why AI agents fail in production, the most common pitfalls, and why it’s often the harness, not the model, that is the culprit.

If you’re deploying AI into production, automating mission-critical work with agents, or leading AI transformation at your company, I hope you find this newsletter helpful!

About the author:
I’m AJ, the CEO and Co-Founder of Grep.AI, a platform for deploying AI agents for repetitive high-stakes knowledge work. We work with global enterprises like Airwallex, IG, and Worldpay to automate mission-critical manual workflows across compliance, onboarding, go-to-market, corporate development, and more. If you want to learn more about Grep, you can schedule a call with me here.

You can also follow me on X and LinkedIn.

Welcome to The Guide To AI by Parcha!

I'm AJ, CEO of Parcha. In this newsletter, members of the Parcha team share our insights into AI and how it’s changing the world around us. Occasionally, we also share updates about what we’re building at Parcha. Today, we're diving into a topic that's been on my mind lately: how AI fundamentally changes the economics of SaaS and why I believe it will usher in a new era of usage-based pricing.

The shift to usage-based pricing, spurred by AI-native products, has enormous implications for founders, product leaders, and anyone involved in the SaaS industry. So, let's break it down and explore what it means for the future of software pricing.

The AI Pricing Paradox

Here's the fascinating paradox at the heart of this shift: As AI becomes more integral to SaaS products, the cost to serve each customer increases with usage. This flies in the face of traditional SaaS economics, where marginal costs typically decrease as you scale.

Why is this happening? It all comes down to compute power. AI models, especially large language models (LLMs), are incredibly resource-intensive. Every time a user interacts with an AI feature, it costs the provider real money regarding GPU usage. This creates a major problem for the subscription-based pricing model that's dominated SaaS for the past decade. Suddenly, your power users become your least profitable customers. It's simply not sustainable.

The Rise of Usage-Based Pricing

The solution to this paradox is usage-based pricing. It aligns costs with revenue, ensures profitability, and provides a better experience for customers who only pay for what they use. We're already seeing this play out in the market:

1. OpenAI's pricing for GPT-4 is purely usage-based

2. Google's new AI features for Workspace will likely come with usage limits

3. Even GitHub Copilot, while subscription-based, has usage caps

But it gets fascinating here: We believe this shift will extend far beyond "AI-first" products. As AI capabilities become table stakes across all SaaS categories, usage-based pricing will become the norm.

Implications for the SaaS Industry

As more and more products add AI features, this shift has enormous implications for the entire SaaS ecosystem:

1. Pricing Strategy Overhaul: SaaS companies will need to rethink their pricing strategies completely. The days of simple "per user, per month" models are numbered.

2. Evolution of Sales and Marketing: Say goodbye to "unlimited" plans. Sales teams will need to learn how to sell value based on usage, not just seats.

3. Customer Success Becomes Critical: Driving profitable usage becomes even more important. Customer success teams will play a crucial role in ensuring customers are extracting maximum value.

4. Product Development Focus: Product teams will need to prioritize features that drive valuable usage, not just engagement for engagement's sake.

5. Financial Modeling Challenges: Finance teams will need to get comfortable with more variable revenue streams and potentially more complex forecasting.

What This Means for Founders and Product Leaders

If you're building a SaaS product today, especially one with AI features, it's crucial to start thinking about how you might transition to a usage-based model. Here are a few steps to consider:

1. Analyze Your Costs: Understand the true cost of serving each customer, especially as it relates to AI feature usage.

2. Model Different Scenarios: Experiment with different pricing structures and see how they impact your unit economics.

3. Talk to Your Customers: Gauge how receptive they might be to a usage-based model. You might be surprised by the positive response.

4. Start Small: Consider introducing usage-based elements alongside your existing pricing model before making a full transition.

5. Invest in Usage Analytics: You'll need robust systems to track and bill accurately based on usage.

The Road Ahead

The shift to usage-based pricing won't happen overnight, and it won't be without challenges. At Parcha, we’ve been experimenting with usage pricing from day one. It requires some upfront education to help our customers understand that our product should be compared to their headcount costs, not their SaaS budget. But we think it’s worth it because usage-based pricing better aligns with our costs and allows us to expand with our customers without upselling more seats. In return, customers have more control over their spending and will only pay for the value they extract from our product. For SaaS companies, it's an opportunity to build more sustainable, profitable businesses aligned with the actual costs of delivering AI-powered software.

What do you think? Will AI drive a usage-based pricing revolution? Or am I off base here? I'd love to hear your thoughts in the comments.

Until next time, thanks for reading Parchans!

-AJ

The Promise of AI Agents

We founded Parcha in early 2023, when "AI agents" built with language models emerged. We were inspired by demos showcasing how LLMs and code could dynamically create and execute complex plans using agentic behavior. Frameworks like AutoGPT and BabyAGI emerged, and people started applying this technology to numerous use cases. The demos looked promising, and the excitement around autonomous AI agents was palpable. Soon after, we saw startups emerging for horizontal (AI Agent platforms) and vertical (AI Agents for specific industries)  use cases like Parcha..

Starting Parcha: The Thesis of AI Agents

We founded Parcha with the thesis that AI agents could automate many operational processes in fintech and banks. Our initial approach was to take a standard operating procedure (SOP) that a human follows to perform an operational process and have an AI agent read and execute those steps autonomously. We developed an agent that would read the SOP and assume the role of the operations expert performing this process today, such as a compliance analyst.

The Parcha agent would generate dynamic plans based on the SOPs provided by our design partners. These plans would guide the agent through each process step, utilizing various tools and commands to execute tasks and record outcomes in the scratchpad. Once it has performed all the steps, the agent will use the SOP and the scratchpad to make a final determination, such as approving or denying a customer, and deliver the result to the end customer. We aimed to create a flexible and adaptive system capable of handling many workflows with minimal human intervention.

We were not building in a vacuum. From early on, we engaged with design partners — fintech and banks- with complex fintech workflows and compliance processes that agreed to work with us and test our agent. These companies saw potential in our technology to achieve their tasks without scaling up their workforce. We used their SOPs and even shadowed sessions where their teams performed these processes manually. We used our platform to build agents for Know Your Business (KYB), Know Your Customer (KYC), fraud detection, credit underwriting, merchant categorization, Suspicious Activity Reports (SARs) filings, and other processes.

The feedback was very encouraging. The demos quickly impressed our partners, and we felt like we were on the path to building a general agent platform to cover multiple processes. However, as we transitioned from demos to production, we found ourselves (a tiny team, by design) spending too much time building the "agent" platform and not enough time building the product and solving the problems our design partners wanted us to address.

While the concept of agentic behavior was promising, building reliable agentic behavior with large language models (LLMs) was a massive endeavor. Creating general-purpose "autonomous agents" could have taken us years. During that time, we wouldn’t solve the problems our customers cared most about with a product that directly addressed their needs. Our customers required accuracy, reliability, seamless integrations, and a user-friendly product experience—areas where our early versions fell short. They would much rather have a solution that was very accurate and reliable for a subset of tasks than a fully autonomous solution that could automate a workflow end-to-end but worked only 80% of the time. We needed to choose between building the agent or building the product.

The Complexity of Putting Agentic Behavior in a Box

As we ran our agents in actual production use cases, we began to uncover the inherent complexity of agentic behavior within a system. From systems theory, we know that as the number of components and interactions within a system increases, so does the complexity and the likelihood of failure. Our approach, which relied on dynamically generated plans and a shared memory model (a scratchpad), made it difficult to decouple and measure subsystems effectively. Each run produced different plans, leading to an ever-shifting landscape of tasks and interactions, which increased the unpredictability and potential for errors.

To put this into perspective, if an AI agent carries out a workflow consisting of 10 tasks autonomously but has a 10% error rate per task, the compounded error rate over the whole workflow is 65%.

Imagine a factory where the assembly line is rearranged dynamically with every run. In such a factory, the placement and order of machinery and tasks change constantly, making it impossible to establish a consistent and reliable workflow. This is akin to what we faced with our agentic behavior model. The dynamic generation of steps—where the agent created an execution plan on the fly—meant that no two executions were identical, leading to an ever-shifting landscape of tasks and interactions. This constantly evolving setup made it exceedingly difficult to instrument and evaluate the system, further complicating our ability to ensure reliability and performance.

The scratchpad, intended to facilitate the agent's observations and results, compounded these issues. Instead of simplifying the process, it created a tightly coupled system where decoupling tasks became nearly impossible. Each task depended on the preceding tasks' results recorded in the scratchpad, leading to a cascade of dependencies that we found challenging to isolate and test independently. Moreover, these dependencies made parallelization impossible. Tasks that could run in parallel had to wait, as the agent needed to evaluate the scratchpad before deciding the next steps.

This tightly coupled system with dynamically generated steps is inherently prone to failure for several reasons:

1. Unpredictability: Each execution path varied significantly, making predicting the agent's behavior and outcomes difficult. This unpredictability is a hallmark of complex systems and a primary source of failure.

2. Complex Interdependencies: Changes or errors in one part of the system could propagate through the scratchpad, affecting subsequent tasks and leading to systemic failures. The reliance on shared memory created a cascade of dependencies that amplified the impact of any single issue.

3. Artificial Performance Bottlenecks: Tasks that could run in parallel ran serially because the agent needed to evaluate the scratchpad constantly. This bottleneck reduced efficiency and increased processing time, countering one of the critical advantages of using AI agents.

4. Evaluation Challenges: The constantly evolving setup made instrumenting, testing, and evaluating the agent's accuracy difficult. Determining the root cause of errors was challenging. Was the issue with the dynamically generated plan, a misunderstanding of the SOP, a malfunctioning tool, or a flaw in the decision-making process? This complexity hindered our ability to ensure the agent delivered accurate and reliable results, ultimately impacting our ability to meet customer needs effectively.

These challenges underscored how impractical it was to rely solely on autonomous agentic behavior for complex workflows. Building a reliable, robust, and stable AI agent would require significant investment. Solving these challenges would have required all our resources for an extended period. Following that path would have prevented us from building our product, generating revenue, and iterating based on customer usage and feedback.

The agent “on rails”

As we spoke to more customers, we realized that for a specific use case like carrying out Know Your Customer compliance reviews, most companies used a variation of the same process and repeated that same process repeatedly. That meant the agent didn’t need to develop a new plan every time it ran.  We could move faster while leveraging what we built by putting the agent "on rails." We removed the dynamic generation of the plan and the shared scratchpad and instead developed an orchestration framework based on static agent configurations. For example, a KYB agent needs to perform the following tasks (simplified for illustration):

1. Verify business registration

2. Verify business ownership

3. Verify the web presence of the business

4. Verify if the business operates in a high-risk industry

5. Verify if the business operates in a high-risk country

This plan started as a configuration file! Now that the plan was static, we could parallelize the process. Tasks 1, 2, and 3 can be done in parallel, while tasks 4 and 5 depend on task 3 as we scan the business's website to understand where and in which industry it operates.

This approach allowed us to break down SOPs into discrete, manageable tasks, now automated with specialized tools. Previously, our "tools" were simple integrations, such as searching for an address on Google Maps. Now, Parcha tools handle complex yet decoupled parts of the process. For example, instead of having an AI agent figure out how to verify a business's registration, we use a sequence of tools that parse a PDF document, perform OCR on it, extract relevant information about the business, and check if it matches what the customer entered in the application. Each tool works standalone and can be reused in multiple workflows.

Focus

Since we no longer rely on an AI to determine a plan based on its SOP, we can only automate a process if we have a list of steps to perform it and the tools we need to do the job. Hence, we decided to hyperfocus on one area we knew well: Know Your Business/Customer (KYB/KYC). This allowed us to build intelligent workflows that could seamlessly integrate into our customers' existing processes, providing clear and measurable benefits quickly. We also placed a strong emphasis on the user experience (one of our values is to "Make it Dope!"), ensuring our product is easy to use and requires minimal integration and training. This new approach resonated well with our design partners, who, by this point, were paying customers.

Where Parcha is Today

The decision to focus on one workflow, executed as a simple set of steps, enabled us to build the best product for our customers. This strategic focus allowed us to explore and leverage generative AI more effectively. Today, we have LLM-powered workflows in over a dozen KYB and due diligence checks, covering document intelligence (e.g., proof of address verification), web-based due diligence (e.g., high-risk country/industry checks), and advanced search (e.g., adverse media, sanctions, and screening). These checks extensively use large language models and the concepts behind agentic behavior. For example, one Parcha KYB agent can perform more than a thousand LLM calls.

So, how do we use LLMs?

When we decided to focus, we also built our AI tools as decoupled, reusable blocks that we could extend and independently evaluate. We orchestrate these subsystems to perform tasks ranging from extracting specific pieces of information from documents to complex reasoning-driven functions like making an approve/deny decision-based on the output of various checks. We built abstractions and primitives for these checks. For example, a "data loader" is responsible for extracting information from one or more sources and structuring it appropriately, while a "check" uses that information to pass judgment. This modular approach allows us to evaluate every part of this pipeline independently. Did we extract the correct information from the document? Was the decision, based on the extracted data, correct? This approach ensures we can focus on each task without worrying about too many external factors.

For example, consider the task of screening news articles to determine if any of them mention a particular person of interest. Rather than building an agent that constructs a plan and autonomously fetches and analyzes each news article, we handle the process in a structured manner:

1. Extraction: We first extract relevant information from each news article, including what happened, when it happened, who was involved, and other pertinent facts. Our extraction process handles formats like HTML, PDF, or plain text and extracts the information from one article at a time. We make tens or sometimes hundreds of concurrent LLM calls to an inexpensive model to extract this information (e.g., a person with a common name will have multiple hits with their name on them). Each language model call focuses on this task, guided by a thoroughly tested prompt.

2. Clustering: Then, we use the extracted features to cluster the articles into "news events." A news event (e.g., reporting a corporate financial crime) may contain multiple news articles, some with unique information to help the judgment process. We use a more complex model for this process since clustering requires more reasoning than a simple extraction.

3. Partial Judgement: Once we have news events and all the information surrounding them, we use a top-of-the-line model to determine if the person of interest is the perpetrator mentioned in the article based on their name, age, address, and other available information about their profile. The language model independently provides judgments on each article, ensuring that the reasoning process is constrained to limited and structured information without noise.

4. Final Decision: Finally, we aggregate the partial judgments' results. We then synthesize these judgments into a final pass-or-fail decision (e.g., "we found adverse media about this person"), a detailed report about what we found, and a human-readable explanation of the thought process that customers can incorporate into their audit logs.

Benefits of this Approach

This approach, building on the agent "tool" (or what we call "check") as a primitive noun in our system, has enabled us to move faster while reliably delivering value to our customers.

Here are some of the benefits we have observed so far:

1. Modularity and Reusability: We can extend and independently evaluate each component by building AI tools as decoupled, reusable blocks. This modular approach allows us to maintain flexibility and adaptability while ensuring high reliability. Each subsystem, whether extracting information from documents or performing complex reasoning tasks, functions independently and can be upgraded or replaced without affecting the entire system.

2. Scalability and Parallelization: Our workflows can handle large volumes of records efficiently by parallelizing tasks. For example, we make tens or hundreds of concurrent LLM calls to inexpensive models to extract information in screening news articles. This ensures we can perform complex workflows that may take a human 30 minutes to an hour, in minutes.

3. Accuracy and Reliability: We independently evaluate each part of the workflow to ensure accuracy. For instance, the "data loader" fetches and extracts information, and the "check" uses that data to pass judgment. We have datasets and experiments in each part of the pipeline, so a team member can improve extraction, test, evaluate, and measure performance. At the same time, someone else does the same with the judgment component.

4. Cost-Effectiveness: Our approach allows us to optimize using LLMs based on task complexity. We use inexpensive models for more straightforward extraction tasks and reserve more robust models for complex reasoning and judgment tasks. This strategic use of resources helps us control costs while maintaining high-quality results.

5. Transparency and Auditability: We provide a clear and human-readable explanation of the thought process behind each decision. This transparency is crucial for audit logs and compliance reporting, ensuring our customers can trust and verify the results. The final aggregation and decision-making process includes detailed reports outlining the reasoning and data used, making the process transparent and auditable.

Where we think AI agents are useful

While compliance workflow automation may not be the ideal use case for AI agents today, they have potential in other areas. One successful application is in customer support, where AI agents answer questions based on a knowledge base, reducing the likelihood of failure. Companies like Intercom and Klarna have seen success with this approach. Additionally, AI agents can be effective when strong feedback loops are present, such as coding agents that must verify their code runs correctly before proceeding.

Conclusion

Our journey at Parcha has taught us several critical lessons about building production-ready products using AI. It has also taught us the value of moving quickly and not shying away from change. In just a few months, we completely changed our approach to focus on reliability and accuracy instead of autonomy, resulting in workflows that are now three times more reliable.

We recognized that building reliable agentic behavior with large language models (LLMs) would have been a massive endeavor, requiring most of our resources and focus. Yet we knew we could use LLMs and many of the concepts behind agents today and provide significant value to our customers.

Today, our focus on modular and scalable LLM-powered workflows has enabled us to deliver reliable, efficient solutions with unit economics that make sense to Parcha and our customers. We have accelerated our customers' largely manual processes and enabled them to grow faster, and since we can move fast, we can quickly improve our product based on their feedback. Ultimately, they did not ask us to build them an AI agent; they wanted us to solve a problem effectively and efficiently.

Happy New Year! We hope 2024 is off to as a great a start for you as it is for us here at Parcha. Here’s a recent interview Parcha CEO AJ Asver did with Brett Gibson, Managing Partner at Initialized Capital and host of the High Bit podcast. In Brett’s podcast, he does deep dives with technical founders on the engineering challenges they had to overcome to build their startup and product. In the interview AJ shared his experience at Parcha with automating KYB manual reviews using AI and specifically LLM-powered AI Agents.

Check out the interview below!

P.S. If you want to learn more about how Parcha can help your company onboard more customers, faster, with stronger compliance, sign-up for a demo here:

Request a Demo

Interview with Brett Gibson

You can also listen to the interview here:

Apple Podcasts:

Spotify Podcasts:

Google Podcasts: Scaling AI Agents for Real-World Tasks with Parcha CEO AJ Asver

It’s been a while since we’ve posted an update in this newsletter as we’ve been heads down working on our new startup, Parcha. You may have also noticed a few changes in this newsletter which is now officially part of the Parcha brand. We will be cross-posting here and on our company blog moving forward.

In this update, we’re going to share a deep dive on what we’ve learned as we brought our AI product from demo into production. Before we jump in, a quick reminder to subscribe to our newsletter to get updates on our work at Parcha and how AI is changing the way we live, work and play.

Subscribe now

For the past six months at Parcha, we have been building enterprise-grade AI Agents that instantly automate manual workflows in compliance and operations using existing policies, procedures, and tools. Now that we have the first set of Parcha agents in production for our initial design partners, here are some reflections and the lessons we have learned.

What do we call an “AI Agent”?

An agent is fundamentally software that uses Large Language Models to achieve a goal by constructing a plan and guiding its execution. In the context of Parcha, these agents are designed with certain components in mind. Let's explore them in more detail.

The agent specifications and directives

This is the agent's profile and basic instructions about how they operate. Agents have expertise in a particular topic or field; they have a role and means to do a job. They also have tools and commands they can use to perform their job. Here is a simple example of the specifications of an agent:

• Profile: You are an Expert Assistant, an AI assistant trained by Parcha.ai to help users answer questions and complete tasks that require subject-matter expertise. You are assisting a Know Your Business (KYB) operations specialist in this conversation.

  • Directives and constraints: You share your thought process as you answer a question; you always use commands available to answer questions. If you are unsure how you previously did something or want to recall past events, thinking about similar events will help you remember. Do not make things up. If you don't know the answer or you don't have sufficient information, immediately say so via a clarification question.

  • Tools or commands: these can be other calls to language models (e.g. to summarize a document) or API calls to third parties (e.g. verify an address using the Google Maps API). For example, these are the commands for an agent responsible for performing address verification:

Commands:
1. address_verification

Command description: This tool is used by agents to verify a business physical address (nor URL). The address could be physical, virtual, a PO Box, etc.

args: "tool_input": {'type': 'string'}

2. company_detailed_risk_profile_tool

Command description: This tool is used by agents to assess the risk profile of a company.

args: "website": {'title': 'Website', 'description': 'the domain or name of the company', 'type': 'string'}

The scratchpad

This is a space in the prompt to the language model where the agents add results from tools and observations as they execute their plan. These observations are used as tool inputs to guide the execution plan or the final assessment.

Example of the scratchpad for an agent that performed one of the commands above:

System: Command kyb_applicant_data_tool returned:

{'answer': "ACME CORP is a personal finance platform founded in 2015. It has over 250 employees and is headquartered in Chicago, Illinois. It offers investment, borrowing and spending tools to hundreds of thousands of customers and manages over $6 billion in assets. The company's website is acme.com and it can be contacted at support@acme.corp"}

A Standard Operating Procedure (SOP)

An SOP is a set of instructions the agent needs to perform to complete a task. The agent uses the SOP to construct a plan using the available tools and to assess if it has all the information needed to make an assessment.

Here is an example of a simple SOP used to perform Know Your Business in a customer:

When a new company is onboarding onto BaaS’s Issuing platform, we are required to complete a KYB (Know Your Business) check on the customer based on FinCEN regulations. To complete the check we must carry out each of the following steps:

Company information: Get all the company information you have for the given company you wish to complete onboarding for. This information should include company address, beneficiary owners, website address etc.

Verify Business Registration: Confirm that the business is registered in the state where they claim to be. This can be done by checking with the Secretary of State of the state where the business is registered or using an SoS verification tool.

Verify Business Address: Confirm the business's operating address or addresses. The operating address may be different from the registered address. The business address can often be confirmed through business lookup tools like Google Places.

Check Watchlists and Sanctions Lists: Verify that the business is not listed on any watchlists, sanctions lists, or are otherwise involved in criminal activity. This can be done by using a business watchlist tool.

Check Business Description: Verify that the description of the business provided by the customer matches the actual business operations. This can be done by reviewing the business's website and comparing it to the description provided.

Check Card Issuer Rules: Review the use case that the business describes for Using our platform and confirm that it is compliant with the card issuer rules.

Final assessment instructions

Our agents have specific instructions that dictate the output of the agent. This could be an assessment of approving/denying/escalating to humans an application, writing a report based on the observations retrieved, or a specific output the user wants to action on.

Example of the Know Your Business final assessment:

Provide Detailed Report: If any of the checks do not pass, prepare a detailed report outlining which checks did not pass and why. Provide a recommendation on whether to approve the customer or whether further review is needed based on these findings. The final answer should include the detailed report and a recommendation.

How we started

Our initial approach to building agents was fairly naive. Our objective was to see what was possible and validate that we could build AI agents using the same instructions humans would use to perform the task. The agent was simple: we used Langchain Agents with a standard operating procedure (SOP) embedded in the agent’s scratchpad. We wrapped custom-built API integrations into tools and made them available to the agent. The agent was triggered from a web front-end through a websocket connection, which stayed open, receiving updates from the agent until it completed a task. While this approach helped us get something built quickly to get validation from our design partners, the approach had multiple setbacks we had to improve over time to prepare our agents to perform production-grade tasks.

Some of the particular setbacks of this approach include:

• Websocket connections caused many reliability issues and ended up not being the best tool for communication between our agents and their operators. We initially envisioned our agents as bi-directional, able to have a two-way conversation. In practice, after the initial interaction (when an operator asks an agent to perform a task like “do a KYB process on X customer”), the agent updates the customer until it completes the process. We may still want to correct an agent or ask for follow-up tasks. Still, there are simpler ways to handle communication for this mostly uni-directional interaction. Our customers didn’t need a chatbot; they needed an agent to complete a job.

• As we started testing our agents with our design partner’s SOPs, we realized it would take more than embedding the full text of instructions into one agent and hoping for the best. The agent would confuse tools or skip tasks. As more steps were performed and results were added to the scratchpad, the context window would be full of noise that the agent would not parse well. Finding the SOP or parsing results that would be useful in the next steps became a challenge.

• Similarly, we relied on the scratchpad as a simple means of “memorizing” information. Many times, the agent would not pick up the right piece of information from it and run a tool more than once so that it would gather input for a new step. This would make the agent way slower than it should be and inefficient.  

• Our agents may take minutes to perform a complex task. Some tasks require doing OCR in multi-page documents, web crawling to research a particular topic, or calling multiple domain-specific APIs to cross-correlate information before concluding the SOP. Our initial approach had no recovery mechanism: if, after 3-4 minutes, a step would fail, the connection to the customer would drop, and the agent would have to be rerun. This, even for a POC, was a pretty poor customer experience.

• LLMs are stochastic, and as such, they can hallucinate, causing the agent to pick a tool that doesn’t exist or provide incorrect input to a tool. This would result in the workflow breaking and the task erroring out before completion.

• Finally, we were custom-building each agent without putting much thought to reusability. Tools were tightly coupled with their agents, and for the most part, each new agent we built required a new set of tools and API integrations we had to build from scratch.

Lessons learned

Agents as async, long-running tasks.

We now run our agents as long-running processes asynchronously. A web service can still trigger agents, but instead of communicating bi-directionally through WebSockets, they post updates using pub/sub. This helped us simplify the communication interface between the agent and the customer. It also made our agents more useful beyond a synchronous web service. Agents can still provide real-time status through server-sent events. They can still request actions from the customer, like asking for clarifications or waiting to receive a missing piece of information. Furthermore, agents can be triggered through an API, followed through a Slack channel (they start threads and provide updates as replies until completion), and evaluated at scale as headless processes. Since agents can be triggered and consumed through REST (polling and SSE), our customers can integrate them with their workflows without relying on a web interface.

Overview of Parcha’s agent architecture (October 2023)

Divide and conquer

After evaluating multiple real-world SOPs and shadow sessions with design partners, we realized instructions could be decoupled into multiple SOPs. We developed the ability for agents to trigger and consume the output from other agents. Now, instead of adding a very complex SOP into one agent’s scratchpad, we have a coordinator <> worker model. The coordinator develops an initial plan using the master SOP and delegates subsets of the SOP to “worker” agents that gather evidence, make conclusions on a local set of tasks, and report back to the coordinator. Then, the coordinator uses all the evidence workers gather to develop a final recommendation.

For example, in a KYB process, it’s common to perform tasks like verifying the applicant's identity, verifying if a certificate of incorporation provided as a PDF is valid, and checking if the owners are on any watchlist. These are multi-step processes (checking the certificate involves performing OCR in the document, validating it, extracting information from it, and comparing it with the information provided by the applicant (usually fetched from an API). Instead of having one agent performing all these steps, a coordinator triggers a worker agent to perform each and report back. The coordinator would decide if - per the SOP - the customer should be approved or not. Since each agent has its scratchpad, this helped us steer them to complete tasks with less noise in the context window.

Context windows on single agent vs. coordinator/workers.

Don’t rush to judge

We faced a similar challenge when asking our Agents to verify information from a document. In the KYB example above, the customer enters information (e.g., company name) into a form and provides some evidence about it (e.g., an incorporation document). Our initial approach to a verification task using an LLM was to enter the text extracted from the document using OCR and the self-attested information into the context window and ask the LLM to verify if the information matched. Since documents are long and full of potentially irrelevant information, the results of such a task were not great. We made this significantly better by separating the extraction process from judgment. We now ask the LLM to extract the relevant information from the document (e.g., is it valid? What’s the company being mentioned, and which state/country is it incorporated in?). Then in a second trip to the LLM, we ask it to compare the information (removing the unnecessary elements from the document) from the self-attested information (e.g., the company information entered in the form). This not only worked way better but also didn’t increase token count or execution time significantly since the second step has a significantly reduced amount of prompt tokens in it.

Context window length, the complexity of the extraction, and judgment

Keep it simple

The coordinator <> worker agent model introduced a challenge: since each agent has a scratchpad, how do we share information? We had agents who needed the same information for two tasks, and initially, they both had to execute the same tools. This was inefficient. The obvious answer was to incorporate memory. There are a lot of mechanisms for memory in agents or LLM apps out there: using vector DBs, adding a subset of the previous steps into the scratchpad, etc. Rather than complicating our stack or risking polluting the scratchpad with noise, we decided to leverage an in-memory store we were already using for communication, Redis.  We now tell the agent pieces of information they have available (the keys to the information in Redis) and built our tool interface to incorporate pulling inputs from the in-memory store. By only adding relevant memory to the scratchpad, we save on LLM tokens, keep the context window clean, and ensure any worker agents pick the right information whenever needed.

Human: Here are memories that you can provide to tools if they need them. These memories SHOULD NOT impact what step you pick next in the plan

{'identity_verification_api_full_name': 'John Doe', 'data_loader_tool_application_documents': '[{"name": "ID_VERIFICATION-ID-272905.pdf", "url": "http://example.com/file.pdf"}], 'digifi_data_loader_tool_self_attested_full_name': 'John Doe'}

...

"command": {

    "instruction": "Verifying identity document",

    "name": "identity_verification",

    "args": {

      "doc_url": "http://example.com/file.pdf",

      "self_attested_values": {

        "full_name_1": "John Doe"  

      }

    }

  }

Example of relevant memory injected into the prompt as needed

Mistakes happen

Each Parcha agent interacts with multiple homegrown libraries and third-party services. Since the workflows are complex, there are cases where a step fails. An HTTP connection to an API may return an internal error, or the agent may choose the wrong input for a tool. Given how our agents were configured, any error would cause the agent to fail with no opportunity to recover. We now treat agents as asynchronous services with multiple failover mechanisms. They are queued and executed by worker processes (we use RQ for this), and we get alerted when they fail. More importantly, we are now leveraging well-typed exceptions in our tools to feed them back to the agent. If a tool fails, we send the exception name and message to the agent, allowing it to recover independently. This has helped us reduce the number of catastrophic failures significantly.

System: Command digifi_data_loader_tool returned:

Unable to execute command due to an error:

1 validation error for DataLoaderInput

display_id  field required (type=value_error.missing)

Human: The tool returned an error. If the error was your fault, take a deep breath and try again. If not, escalate the issue and move on.

Example of self-correction prompt sent to the agent with exception name and message

Building blocks

After taking multiple weeks to develop each of our first set of agents, we decided to focus on reusability and speed of building. We want to enable our customers to build their agents quickly. To that end, we developed our agent and its tools interface, focusing on composability and extensibility. We now also have enough workflows from design partners to understand which building blocks we need to invest in. For example, many of our customers do some document extraction. We now have a tool that can easily be applied to any document extraction task. We built this tool once but already use it in most workflows. Our customers can use it to extract information from an incorporation document and validate its veracity or to calculate an individual's income from a pay stub. The work to adapt the tool to one specific, new workflow is minimal.

Our document extractor tool can be easily extended into multiple use cases

What's Next

• Webhook triggers to run an agent and perform actions on its final assessment, enabling end-to-end automation for our customers.

• Implementing our in-house robust agent benchmarks, assessing the capabilities of our agents to Plan, Execute, be Accurate, and Reasoning (PEAR, we’ll write about it).

• Deploy agents and tools as micro-services. By training our agents to create and perform complex execution plans (essentially DAGs), we could orchestrate them as asynchronous micro-services, which will increase composability and enable agents to use language-agnostic tools while enabling our tools to be compatible with agents outside of Parcha.

Want to build with us?

We are hiring a full-stack founding engineer in San Francisco. This person will be responsible for leading the architecture and development of the platform powering our AI Agents. They will partner with our founders and AI engineers to build bleeding-edge technology for real enterprise customers. Learn more about this role here and contact miguel (@miguelriosEN on X or miguel@parcha.ai).

I’m excited to share another interview from my podcast, this time with David Kossnick, Product Manager at Coda. Coda is a collaborative document tool combining the power of a document, spreadsheet, app, and database.

Before diving into the interview, I have an update on Parcha, the AI startup I recently co-founded. We’re building AI Agents that supercharge fintech compliance and operations teams. Our agents can carry out manual workflows by using the same policies, procedures, and tools that humans use. We’re applying AI in real-world use-cases with real customers and we’re hiring an applied AI engineer and a founding designer to join our team. If you are interested in learning more, please email founders@parcha.ai.

Also don’t forget to subscribe to The Hitchhiker’s Guide to AI:

Subscribe now

Now, onto the interview...

Interview: Supercharging your team with Coda AI | David Kossnick

I use Coda daily to organize my work, so I was thrilled to chat with David Kossnick, the PM leading Coda’s AI efforts. We discussed how Coda built AI capabilities into their product, and their vision for the future of AI in workspaces, and he gave me some practical tips on how to use AI to speed up my founder-led sales process.

Here are the highlights:

• The story behind Coda’s AI features: Coda started by allowing developers to build “packs” to integrate with their product. A developer created an OpenAI pack that became very popular, showing Coda the potential for AI. At a hackathon, Coda explored many AI ideas and invested in native AI capabilities. They started with GPT-3, building specific AI features, then gained more flexibility with ChatGPT.

• Focusing on input and flexibility: Coda designed flexible AI to work in many contexts. They focused on providing good “input” to guide users. The AI understands a workspace’s data and connections. Coda wants AI to feel like another teammate—able to answer questions but needing to be taught.

• Saving time and enabling impact: Coda sees AI enabling teams to spend less time on busywork and more time on impact. David demonstrated how Coda’s AI can summarize transcripts, categorize feedback, draft PRDs, take meeting notes, and personalize outreach.

• Tips for developing AI products: Start with an open-ended prompt to see how people use it, then build specific features for valuable use cases. Expect models and capabilities to change. Focus on providing good "input" to guide users. Launching AI requires figuring out model strengths, setting proper expectations, and crafting the right UX.

• How AI can improve team collaboration: David shared a practical example of how AI can help product teams share insights, summarize meetings and even kick-start spec writing.

• Using AI for founder-led sales: David also helped me set up an AI-powered Coda template for managing my startup's sales process. The AI can help qualify leads and draft personalized outreach emails.

• The future of AI in workspaces: David is excited about AI enabling smarter workspaces and reducing busywork. He sees AI agents as capable teammates that understand companies and workflows. Imagine asking a workspace about a project's status or what you missed on vacation and getting a perfect summary.

• From alpha to beta: Coda’s AI just launched in beta with more templates and resources. You can try it for free here: http://coda.io/ai

David’s insights on developing and launching AI products were really valuable. Coda built an innovative product, and I'm excited to see how their AI capabilities progress.

Episode Links

Coda’s new AI features are available in Beta starting today and you can check them out here: http://coda.io/ai.

You can also check out the founder-led sales CRM I build using Coda here: Supercharging Founder-led Sales with AI

Transcript

HGAI: Coda AI w/ David Kossnick

Intro

David Kossnick: ,One of our biggest choices was to make AI a building block initially. And so it can be plugged in lots of different places. There's a writing assistant, but there's also AI, you can use in a column. And so you can use it to fill in data, you can use it to write for you to categorize, for you, to summarize for you and so forth across many different types of content.

David Kossnick: Having that customizability and flexibility is really important. I'd say the other piece more broadly is there's been a lot of focus across the industry on what, how to make good output from AI models and benchmarks and what good output is and when do AI models hallucinate and lie to you and these types of things.

David Kossnick: I think there's been considerably less focus on good input. And what I mean by that is like, how do you teach people what to do with this thing? It's incredibly powerful, but also writing natural language is really imprecise and really hard.

AJ Asver: Hey everyone, and welcome to another episode of the Hitchhikers Guide to ai. I'm your host, AJ Asver and in this podcast I speak to creators, builders, and researchers in artificial intelligence to understand how it's going to change the way we live, work, and play. Now, You might have read in my newsletter that I just started a new AI startup

AJ Asver: since starting this startup a few months ago, a big part of my job has been attracting our first set of customers. I love talking to customers and demoing our product, but when it comes to running a founder-led sales process, prospecting, qualifying leads, And synthesizing all of those notes can be really time consuming, and that's exactly why I decided it was time to use AI to help me speed up the process and be way more productive with my time.

AJ Asver: And to do that, I'm gonna use my favorite productivity tool, Coda. Now, if you haven't heard of Coda, it's a collaborative document editing tool that's a mashup of a doc, a wiki, a spreadsheet, and a database.

AJ Asver: In this week's episode, I'm joined by David Kossnick, who's the product manager that leads Coda's AI efforts. David's going to share the story behind Coda adding AI to their product. Show us how their new AI features work, and give me some tips on how I can use AI in Coda.

AJ Asver: By the way, I've included a template for the AI powered sales CRM I built in the show notes, so you can check it out for yourself.

AJ Asver: But before I jump into this episode, I wanted to share a quick update on my new startup At Parcha, we're on a mission to eliminate boring work. Our AI agents make it possible to automate repetitive manual workflows that slow down businesses today.

AJ Asver: And we're starting with FinTech in compliance and operations. Now, if you're excited by the idea of working on cutting edge autonomous AI and you're a talented applied AI engineer or designer based in the Bay Area, we would love to hear from you. Please reach out to founders@parcha.ai if you wanna learn more about our company and our team.

AJ Asver: Now, let's get back to the episode. Join me as I hear the story behind Coda's latest AI features in the Hitchhikers Guide to AI.

AJ Asver: hey David, how's it going? Thank you so much for joining me for this episode.

David Kossnick: It's going great. Thanks for having me on today.

What is Coda?

AJ Asver: I am so excited to, go deeper into Coda's AI features with you. As I was saying at the beginning of this episode, I've been using Coda's AI features for the last month. It's been kind of a preview and it's been really cool to see, it's capable of. I'm already a massive Coda fan, as you know. I used it previously at Brex. I used it to organize my podcast and my newsletter, and most recently it's kind of running behind the scenes at our startup as well for all sorts of different use cases. But in this episode, I'd love to jump in and really understand why you guys decided to build this and what really was the story behind Coda's AI tools and how it's gonna help everyone be more productive.

AJ Asver: So maybe would you describe and what exactly it does?

David Kossnick: Coda was founded with a thesis that the way people work is overly siloed. So if you think about the most common productivity tools, you have your doc, you have your spreadsheet, and you have your app. And these things don't really talk to each other. And the reality is often you want a paragraph and then a table, and then another paragraph, and then a filter view of the table, and then some context in an app that relates to that table.

David Kossnick: And it's just really hard to do that. And so you have people falling back to the familiar doc, but litter with screenshots and half broken embeds. So Coda said, what if we made something where all these things could fit in one doc and they worked together perfectly? And that's what Coda is.

David Kossnick: It's a modern, document that allows you to have a ton of flexibility and integrate with over 600 different tools, uh, for your team.

AJ Asver: Yeah, I think that idea of Coda of being able to one, integrate with different tools would also be both a doc that can become a table and then have a mashup of all this different type of data is something I've really valued about it. I think, especially when I was at Brex and we used to run our team meetings on Coda, it was really great to be able to have like the action items really formatted well in the table, but also have the notes and more freeform and then combine that with kind of follow ups.

AJ Asver: And we even had this crazy table my product team where we would post like weekly photos and that's like really hard to do or in an organized way in a doc, and you'd never wanna do that in a spreadsheet. So, um, I love the fact that Coda enables you to combine all that different type of data together. So, Coda has that. And then it also has packs, which you mentioned too, right? And these are these integrations that allow you to like take data from lots of different places and put it all together.

Story behind Coda AI

AJ Asver: And from what I understand, Coda's AI products started as a pack, right? It was like this pack that someone had, Coda had built more as kind of like a hack project to get people to use the OpenAI capabilities inside Coda.

AJ Asver: And I'd actually tried this too, and that's kind of how you guys decided to actually build this into a real, uh, native integration.

David Kossnick: Yeah, totally. I'd say, you know, the first bet Coda made was on packs as a platform. And so maybe about a year ago now, we released an SDK where anyone can build a pack for Coda in their browser, in JavaScript and compile it and publish it. And so it can pull data from places, push data to places. And it's really been incredible to see people do this for all sorts of use cases we never even thought of.

David Kossnick: And it made possible people, starting to experiment with AI in a much more effortless manner. And so someone did a kind of weekend project and published the first OpenAI pack and it really took off starting to see it get used for all sorts of different cases. And it got us inspired for thinking, Hey, you know what?

David Kossnick: If we did something native that you didn't have to think about authenticating to external services, what if it could go much deeper in what context it had about your doc and your workspace in order to help you save time?

AJ Asver: One of the things I really loved here is kind of a product management thing is like seeing this nascent behavior right, happening on the platform and then deciding that it was something worth investing in further. So what point did you guys decide like, oh, this is becoming big enough or popular enough where we should make an investment in, and how was that decision made?

AJ Asver: Is that like a decision that like the CEO makes or is it more like kind of bubbled up from the bottom where like there was a team that saw this happening and was like, hey, we'd like to invest in this further. I'm really curious. Like give us a bit of like the inside baseball of how that happened.

David Kossnick: There were a few moments. Uh, the first moment was kinda the weekend hackathon where a few people, I think it actually started on Friday afternoon. Some of the DALL-E API had been released by OpenAI, and they really wanted to start generating images in a doc. And so it took like two hours, uh, to basically create a new pack from scratch and have it fully workable inside of a doc.

David Kossnick: And then we had a weekend blitz to basically ship it on Product Hunt. And Reid Hoffman, who's on Coda's board, big fan of OpenAI, was actually kind enough to hunt it on Product Hunt for us. Um, a bunch of really cool templates people had quickly built for it. And then about a month and a half later, we had a company-wide hackathon. This must have been back in December. And there was a ton of enthusiasm on about AI, partially from, um, the OpenAI pack. And so we explored like a dozen different ideas. Um, I think we won the People's Choice Award, the company votes on all the different hackathon pitches at the end of it.

David Kossnick: And then, uh, at January we had, a board meeting and we showed off some of the, thoughts from the hackathon as well as some of, what people had already been doing in the community. And the board was really excited about it, and so we started up a much bigger effort within the company.

AJ Asver: It's a really cool story to hear that, you know, what started as like a project that became like a hackathon, that, that became like a pack that was put together really quickly and just kind of an experiment then became like this big investment for the company. Right? And now when I look at the product, which, you know, you're gonna talk a bit more about, I can see how like it's, it could really become like a core kind of primitive of the Coda experience, just like a table and a dock and a canvases as well.

AJ Asver: So, that to me is like really inspiring, especially for other folks that are like working in product that, companies that are like Coda stage that you can really like come up with these experimental ideas and they can end up becoming products. And also I think it was really encouraging to see that you guys kind of explored AI and like integrating into the product pretty early, right?

ChatGPT and Coda AI

AJ Asver: Like this was like pre-chat g p t hype, like, or just like as ChatGPT was becoming popular, but before GPT-4 came out, Christmas at least, I, I feel like the hype was just starting to simmer them, but not nec necessarily boil over like it is right now. talk me through what it was like developing the AI product into what it is today and how you guys kind of built into the product and, and how it works.

David Kossnick: I look back on those early days and it's sort of, uh, amazing how much chaos there was in the market. You know, ChatGPT had just come out and it was incredible, and we were dying to get our hands on a chat based API but at the time, the only backend available to us was GPT-3.

David Kossnick: They hadn't released an API for ChatGPT. There was nothing else really at that quality level on the market. And so I remember going and chatting with every major developer, API, AI company, every platform and being like, Hey, like, what have you got? What can we use? And we had a bunch of different ideas on UX, but we were kind of bottlenecked on what's possible.

David Kossnick: So we started by building something with GPT-3 actually. And we said, okay, Chat's gonna come at some point. We don't know if it's a month out or a year out. Uh, we, you know, we gotta start moving now. Um, and we did a few experiments on it. Some just straight outta the box. And some that were very specific.

David Kossnick: So actually one of my favorite was Coda has a formula language. It's incredibly powerful. People love it. It's also, um, a little complicated for people who are starting to learn it. It's a lot like Google Sheets or Excel. And so we had said, what if you could have natural language that just turned into a Coda formula?

David Kossnick: And so we, um, we collected a data set for that. We actually crowdsourced it within the company. We took all of the company's internal staging environment, data of quota formulas, and we had people annotate what the natural language equivalent was. and we fine tuned GPT-3 for it.

David Kossnick: And we built a little thing that would basically, you know, convert text to formulas. We were like, wow, that's actually pretty good. We realized, you know, some of the hard cases are really hard, but some of the average cases it does quite well on. Um, but it was definitely a mode where, because there was no sort of generic chat backend, we had to think like, feature by feature, what would we do for this exact scenario?

David Kossnick: What are the prompts we would create, uh, and so forth. Um, and we got, you know, decently far down that path. Uh, at which point, you know, ChatGPT's API, which was called Turbo 3.5, was released and unlocked kind of a whole set of other use cases.

AJ Asver: I think for people that, you know, may have forgotten by now, cause it happened so quickly, right? GPT wasn't available through chat until November. So you basically had to just provide a prompt and it would do a completion, but it wasn't fine-tuned in the same way it was right now. It wasn't, um, basically it wasn't as good at following instructions right as it is now. And so you had to do a lot more work to get it working, and then of course chat landed. Did it feel like you kind of were given this like gift where you'd be like, oh, this is gonna make it way easier. And then how did that kind of lead to where the product ended up?

David Kossnick: was definitely a gift. We were super excited. It's also, as I'm sure you know, is like, a very double-sided, uh, sword. Um, you know, prompt engineering is hard. It's brittle. You sort of make a tweak and move. Sideways in, forward and backwards all simultaneously for different set of things.

David Kossnick: Um, and so there's definitely a new muscle on the team as we moved into sort of turbo and GPT-4 on, how do we really evaluate which things it's doing well on, which is doing poorly on how do we make it really good for those use cases, both by setting user expectations and by, by changing the input, when we actually want GPT-3 with something fine-tuned.

David Kossnick: And so it sort of opened up a whole kinda, new worms in the problem space, which was super exciting. And I think one of the things that, that got me really revved up about, uh, you know, Coda specifically as a uniquely great surface for AI is there's so many different ways people use Coda. So many different personas, so many scenarios.

David Kossnick: It's an incredibly flexible tool. And so having a backend like, ChatGPT is really, really useful for a fallback. For any sort of long tail and unusual, surprising request, cuz it does really well at the random thing. And one thing we've discovered is, you know, for the very narrow set of most common things, it does pretty well too, but not as good as the more specialized thing.

Making Coda AI work for lots of use cases

AJ Asver: So as you were talking about, Coda being used for lots of different use cases, I noticed that because there's so many different templates in the gallery and so many different ways Coda has been used, there's like pretty big community of Codens, right, that are building these different types of Coda docs. How did you think about it when it came to adding AI into Coda to make sure it's versatile, versatile enough to be used in many different ways. much did that impact kind of the end design and the user experience?

David Kossnick: you know, One of our biggest choices was to make AI a building block initially. And so it can be plugged in lots of different places. So you'll see as we get to a demo a bit later, there's a writing assistant, but there's also AI, you can use in a column. And so you can use it to fill in data, you can use it to write for you to categorize, for you, to summarize for you and so forth across many different types of content.

David Kossnick: Having that customizability and flexibility is really important. I'd say the other piece more broadly is there's been a lot of focus across the industry on what, how to make good output from AI models and benchmarks and what good output is and when do AI models hallucinate and lie to you and these types of things.

David Kossnick: I think there's been considerably less focus on good input. And what I mean by that is like, how do you teach people what to do with this thing? It's incredibly powerful, but also writing natural language is really imprecise and really hard.

AJ Asver: Mm-hmm.

David Kossnick: We had a user study early on. I remember it was super surprising.

David Kossnick: Someone asked, our AI block how much money was in its bank account when it was in the person's bank account. And I was just blown away that they, it felt so knowledgeable and powerful. They assumed. Know that even though they never authenticated their bank account, they just like forgotten. It just felt like something that they, that we would expect it to do.

David Kossnick: Um, and so how do you remind people sort of the universe of what's possible or not, or what it's good at or not? We have something very simple. You know, a lot like Google and Auto Complete as you type, you get suggestions underneath it. But a surprising amount of effort went into that piece in particular.

David Kossnick: What do we guide people towards? Which specific types of prompts, how do we make the defaults really good? How do we expose the right kinds of levers to show people what's possible and make them think about those? Um, and I think as an industry, we're still pretty early on that for these large language models, like I think we're gonna see a wave of innovation on how do you teach and inspire people how to interact to to have good input in order to get the good output.

AJ Asver: So we've talked a lot about that story behind Coda and AI, and it's really interesting to hear how you guys developed kind of the thesis around it and put into the product. Um, I think for folks that aren't familiar with Coda especially, I'd love to just jump in and for you to show us a little bit about how Coda works with AI and, and what it can actually do.

Demo: superchaging your team with Coda AI

David Kossnick: That sounds great. Yeah. Maybe I can walk you through a quick story about a team working together in a team hub with AI. And so this team hub is a place where different functions come together on a project, and have shared context.

David Kossnick: So, a super common way people use Coda is collecting feedback. Um, all sorts of feedback, support, tickets, customer feedback, uh, sales calls. Um, and so we have lots of integrations that do this. They pull in Zoom transcripts and content looks a lot like this. It's really rich.

David Kossnick: There's so much context in here, but it's really hard to turn this into something that's valuable for the whole organization. Um, I've spent many hours of my life writing up summaries and tagging things, and so wouldn't it be great if AI could just do this for me? Uh, and so here's a quick example. I'm gonna ask AI to summarize this transcript in a sentence, and here we go.

AJ Asver: That was really cool. And I think what was really magical for me about it is not just that you can summarize, because obviously you can take the transcript, you can put it in GPT-4 and summarize. And there are other tools that do summaries too. But I think what's magical is when you combine that AI in the column, but with the magic of Codar's existing integration.

AJ Asver: So like you have connected it to, I think it's Zoom, right? There's like a zoom

AJ Asver: pack that's already outputting all the transcripts. So you've automated that bit and then you create this formula that basically runs on that column and then every time a new transcript comes in, I presume it just automatically summarizes it.

AJ Asver: So that like piece of like connecting all those dots together, that's why I love Coda and that's why I think this is a really cool example of where kind of Coda shines.

David Kossnick: That's exactly right. And one of my favorite pieces here for dealing with large amounts of data is just categorizing things. There's so many times I'm going through a large table of data, picking a category, so wouldn't it be great if based on this transcript I could just automatically tag. What type of feature request it was and boom, there it is.

AJ Asver: where's it getting those feature requests from,

David: Yeah,

AJ Asver: tags? Is it like kind of making those or?

David Kossnick: In this case, I already have a, a schema for what are the types of feature requests that I've seen, and so it's just going ahead and tagging all those things.

AJ Asver: That, that's a really interesting feature too there, by the way, because now what you're doing is you're taking kind of the open-ended feature tagging problem where really GPT could like generate any feature tag at once and you're constraining it with this select list. And that's another good example of where if you did this in ChatGPT, you may end up with lots of different types of feature tags, but by constraining, you now end up with this format when you can now I, I assume go and like, organize these transcripts by feature tag because they're like actual each of those little data chips, right? And so it's now like segmentable, like

David Kossnick: Yeah, and one of the cool things about it is you'll see this one is blank. That's actually intentional. That means it, it either couldn't find a match or didn't know what a match was. As there's plenty of cases where there's, you know, there's no real feature request or doesn't really know what to do with it, and it just won't tag anything either.

AJ Asver: Very cool. Very cool. Okay, what What else you got?

David Kossnick: So very common. Maybe the support team does an amazing job summarizing all the tickets, the feedback that's coming in, even tags things for you. And then as a PM you have to sit and read it all and think about it and say, okay, how should this influence my roadmap? Wouldn't it be nice if you could use AI to get started?

David Kossnick: And so imagine you say, create a PRD for new image editor based on the problems in all this user feedback. Here we go.

AJ Asver: Okay. No need for PMs

David: Yay.

AJ Asver: what are you gonna do after? this goes into production?

David Kossnick: Well, of course it's a first draft. You know, you should always, uh, proofread. You should always change it. And so maybe AI can help with that too. Say, make this a little longer, um, and have it help me edit this PRD before I send it off. Um, there we go.

AJ Asver: One of the things I love about this is for me, often when I was PMing, it's that cold start problem. It's like you are at like Tuesday, you know, you've got your no meeting Wednesday and you've gotta write this PRD in time for like a review deadline on Thursday because it's gonna go in product review on Friday.

AJ Asver: Right? And you've gotta start it and you just keep putting it off cuz you've got back to back meetings. And then you get to Wednesday and you're staring at a blank screen and then you're like, Oh, maybe I need to go check my email. Right. I just think like more than anything else, this will just solve that cold start problem of just getting something down that you can start iterating on so you can just make progress faster.

AJ Asver: And now what would've taken you three or four hours to get from like blank screen to PRD first draft is now probably gonna be a couple of hours because you've got that first version, you can kind of iterate on it from there. So I think this is gonna be a huge help for PMs. And, the cool thing about it is you're taking all this structured data right, from different places and bringing it into one place.

AJ Asver: And one of the things we often did when we used Coda at Brex is like we would have, you know, like you had like kind of place that had customer feedback, Or you'd be aggregating different feature ideas in like a brainstorming, section and then you're kind of bringing them in here and turning them into a PRD.

AJ Asver: So that's pretty cool.

David Kossnick: Thanks. So another super common scenario is you have a team meeting at Coda. We do these inside of a doc with structured data, which I really love. They let people vote on which agenda item to talk about first, and you can even have notes attached to it here about what's happening. But again, what do I do after the meeting?

David Kossnick: As a PM I spend a ton of time writing up next steps, but oh yeah, I can do that for me. That's awesome. Uh, one of the other things I do all the time is write up summaries. Um, what if instead I could ask ai, AI to do that too? And then of course I can send that out to Slack direct from Coda.

David Kossnick: So outreach. One that we've already started doing at Coda is personalizing messages to key accounts. Um, and so let's say we have this, uh, launch message about this new image editor feature. We wanna tailor it based on the title and the company.

David Kossnick: Uh, we can go ahead and get a, an easy first draft to start with here. Boom. Let's say we don't like one of these. I'm just gonna refresh this one. We'll get another example. Um, and maybe I want to go in and change this a little bit. Um, hope your family is doing well and using our Gmail integration. I'll just go ahead and send that email.

AJ Asver: And that's actually gonna send that email now to the, person that you wanna do outreach to. And you just basically generated that email based on kind of the context of the person's job

David Kossnick: Totally. And one of the really fun parts of this is it's super flexible. So imagine you had another column that's like family member names, or hobby or other things like that that you're not gonna find in, your favorite, uh, sales tool. Um, AI is really good incorporating that context. So having all your stuff here in the team hub, being able to pull that context in, um, is really powerful.

AJ Asver: Does it generate tables too? How how does

David Kossnick: You know, we showed a simple example here, which generates the table of target audiences. Um, but actually one, maybe I'll just show real quick, um, that I've been doing in my personal life. So very different kind of template. Um, we use meal planning, so I'm a vegetarian, so meal planning can be sometimes a bit of a pain with two kids, uh, making sure everyone gets exactly what they need.

David Kossnick: Um, and so I made a quick template. Everyone in my family can go in and add. Their favorite ingredients and get out both a bunch, a bunch of ideas as well as, uh, specific dishes. And so this is an interesting case where it's just a normal table here. And I'll say, uh, AJ, what's your favorite ingredient?

AJ Asver: Well, you know what? My kids love

David Kossnick: Broccoli. Wow. Nicely done. cool. And then I'll go ahead and, uh, or auto update it here. Uh, so it has a bunch of different meal ideas and yeah, let's say I'll take, uh, spinach and cheese quesadilla. Um, and I'll add that one in here. Spinach and cheese quesadilla. Um, and AI is gonna start generating, uh, what ingredients are needed for that as well as a, a recipe and about how long it would take.

AJ Asver: That's a really, really awesome hack. I think I need to start doing that, as well to just use it to generate ideas for, meals as well and meal planning. That's, that's very, very cool. And this is like a good example of also like how it can be helpful in like a personal setting too.

AJ Asver: Right?

David Kossnick: For sure.,

Using AI to speed up lead qualification

AJ Asver: I was not gonna bring you onto this podcast without you helping me with something. So one of the reasons I wanted to bring you on here is because now that we've started this startup and we're trying to get everyone to be very excited about our AI agents, I am doing what's called founder-led sales, which is where we kind of work out, okay, who are the companies that we wanna target?

AJ Asver: And then at the very early stages we're trying to find like five design partners that we can work with. And they're kind of like enterprise FinTech companies, similar to like Brex where I used to work. And my job is to do the sales. Cause there's only two of us right now and Miguel's busy building the product. And so I gotta work out which companies will be a good fit and then we work out, okay, how do we get an intro to them? Maybe it's through an investor, maybe it's through a mutual contact. Maybe we, outbound to them because you know, maybe it's a company that we worked at before or something. And so I've been trying to work out how to use Coda to help me do this.

AJ Asver: And I was wondering if you might be able to help me make my Coda CRM better with AI.

David Kossnick: Sounds amazing. Let's do it.

AJ Asver: I'm very excited about this because this is gonna save me a lot of time.

AJ Asver: Over here is my Coda CRM and very high level. I have this list that I've got, and I think I got it from Crunchbase, um, with just a bunch of FinTech companies. And some of them might be a good fit, some of them might not. Oh, that's definitely not a FinTech company. Let's take that one out. Um, and I, and I'm trying to work out kind of one, how big are these FinTech companies? Either they kind of size where they would be an interesting fit for our, for our product, we generally try to focus on kind of growth stage FinTech companies, and then two, Would they qualify or not? Based on do we think they might need the product we're trying to build? And so I was wondering, maybe the first thing is I have this kind of list of the different, um, tiers of, of customers that I might want or the different types of companies that I might want to organize 'em into.

AJ Asver: So for example, there might be growth stage FinTech companies, there might be early stage FinTech companies. And I wanna work out how I can take these FinTech companies that I have in this list and kind of categorize them by that tier. maybe we could, start there.

David Kossnick: Yeah, sounds great. What kind of categories are you thinking about?

AJ Asver: I think maybe we can start with kind of early stage FinTech growth stage FinTech and a financial institution. So maybe first I need to add like a column, right? Is that correct?

David Kossnick: Yeah, sounds good.

AJ Asver: Okay. So we can go do that, have a column after this one. And then do I make this a select list? Is that

David Kossnick: Yeah, that's exactly right. So you could add, uh, select list

AJ Asver: Great.

David Kossnick: sort of a, a list of types or a list of items.

AJ Asver: Okay, so then let's, let's pick a few different stages. So it's growth stage, FinTech, um, maybe series C +, and then maybe early stage FinTech Seed Series C and then maybe kind of traditional financial institution. Cause I think there's a few of those in this list. And then I know there's probably some crypto startups in here too, so maybe I'll put like, crypto is another one too. Okay. So I've got my select list. What do I need to do next?

David Kossnick: Um, cool. So the, uh, actually before you forget, yeah, maybe rename the, the column there.

AJ Asver: So we call this customer segment. Great.

David Kossnick: So next, if you could just go to, uh, right click on that column and do add AI. Um, so the question is, what do you think would determine it? Is it, you know, one thing we could do is just give it the company name. many that are well known, it might do actually a pretty good job.

David Kossnick: Um, you could also try giving it the, um, the description as well. And you could say basically, you know, what, what category does this belong to?

AJ Asver: I kind of like the idea of doing it by description. That seems like a good way of doing it. So then would I just do this like, um, pick is like kind of pick the correct customer segment based on the company description

David Kossnick: Mm-hmm.

AJ Asver: provided?

David Kossnick: That's perfect.

AJ Asver: that work?

David Kossnick: And then if you do at, and then the column name should pull it in for you.

AJ Asver: Okay. And then let's see, what's this company

David: I think it was info.

David Kossnick: Yeah.

AJ Asver: Yeah. Great. And then how does it know which one to pick? Oh, it just kind of knows from what's available, right? Is that how it

David Kossnick: Yeah. The AI knows in a select list or look up the AI knows to use that set of options.

AJ Asver: Ah, smart. All right. So let's, let's do it. Okay. Wow. so it's already started. Um, but what happens if we think one of them might be wrong? So, for example, Affirm. I don't know if maybe it needs a bit more data.

AJ Asver: So I'm wondering if a good one could be that we look at funding.

AJ Asver: And then last funding round will probably help. Um, and then maybe like employee count. I feel like this should all help work it out.

David Kossnick: Yeah. Sounds great.

AJ Asver: Okay. So then if we do that, um, then we need to go back to here and go to this segment thing. We close this. Okay. And then we go, okay. Provided. And then this is kind of my prompt engineering now I usually like to do it like this. So company, maybe we'll do company name cuz that might be helpful too, right?

AJ Asver: Might know some of this stuff already. Company description, just info. Last funding round. Total funding. Okay. And now let's give this another try.

David Kossnick: All righty.

AJ Asver: Um, Fill.

David Kossnick: Wow. Very different result.

AJ Asver: Great. That's way better. Right . Now it like correctly categorized firm, which is awesome. um, and Agora and Airwallex Crypto for Anchorage, which is awesome. Apex, this is great. Now I have some qualification.

AJ Asver: I just ran the qualifier and it's actually started qualifying these leads, which is really cool. And I guess in the case where it doesn't have information, it'll tell me, if the company's a traditional financial institution, might work that out. So I guess I could go through these and check these all later. But the cool thing is I now have some qualified beads that I can start, um, working out which ones to focus on for my sales.

David Kossnick: That's awesome.

AJ Asver: And reach out.

David Kossnick: Nice.

Coda's long term strategy with AI

AJ Asver: I am really excited to start, um, using my new. AI powered lead qualifier cuz I'm a one man sales team and I'm already thinking of other ideas. Like earlier when you showed me generating emails, I think I'm gonna start trying that as well, like generating outbound emails or even introduction emails to, to the right customers, based on who the right investor connection is.

AJ Asver: And that stuff always just takes a lot of time. And I'll say like when I'm in front of a customer and pitching, I'm like the most energized. And I think through the sales process when I'm like in the CRM and trying to write emails and then qualifying leads, I'm like the least energized. So having AI helped me there is gonna be really great to, to gimme a bit of a boost. Um, I'm curious, where do you guys see the long-term kind of impact of AI being for Coda and how you guys think about it for long-term strategy and also when will this be available for other people to try out?

David Kossnick: We are launching, the beta very, very soon. The beta will be very similar, but there'll be a lot more templates and resources, um, and we'll be letting in way more folks and so would love feedback.

David Kossnick: Please try it out. And in terms of where we're going with AI, I'm really excited about it. I mean, as I mentioned at the start, Coda has kind of a uniquely great place for AI because it brings so many different pieces of a team together in one place. And that context is so helpful for AI. And so things that get me really excited is being able to ask your workspace a question about how a project is going, and it's able to just answer because it knows all the tools you're connected to, all the notes people are taking about every project.

David Kossnick: Um, you know, imagine you come back from being off for a week on vacation, you're like, what did I miss? And you get the perfect summary and you can drill into more detail and granularity on anything that you're curious about. Um, you know, imagine it felt more like having a teammate when you used AI.

David Kossnick: Who was able to engage on, um, on your projects and give feedback and have details. And obviously it's not exactly a teammate and you're gonna have to, to teach it about each, each case. Um, but I think the, the vision of having less busy work and more impact is really exciting.

AJ Asver: I, I think that potential of, AI in Coda that you mentioned beyond just a single doc, but when you're using AI to really run your company is gonna be a really, really, really powerful one. Because if you have kind of sold on using AI as your wiki and to run your projects and your docks and your spreadsheets, then you guys basically have all the information as you mentioned that's required in kind of an internal knowledge base to answer these complex questions.

AJ Asver: Like what's the status of a project, who is working on what parts of the project? And so. personally, that's, uh, something I'm very excited before because we use Coda for everything and we're a very small team right now. But I imagine as we get bigger and more people get involved, being, being able to ask those questions and being able to answer 'em is really, really cool.

AJ Asver: And so it's gonna be in beta soon, which is really, really awesome. And how are you, as a PM thinking about, you know, launching this Beta and what it means to kind of bring this into production? And do you have any tips for other PMs that are working on AI products? Because you are, you, you've been at this now for six months, right? Um, which puts you, I would say, in like the early percentage of PMs that are working with, you know, GPT. Um, curious what your tips are for, for other folks trying to bring a product from that hackathon to a beta and then a GA.

David Kossnick: know, I'd say a lot of people start, like we did, of just sort of, you know, throwing AI behind a prompt, a prompt box in your product. I think it's a great starting point to learn and see what people are doing. And I think as you develop a sort of deeper sense of what use cases are really valuable, um, you're gonna build something much more specific for them.

David Kossnick: One of the really fun things about working with AI is you don't know exactly how it's gonna go. You know, the models are a moving target in terms of what they're really good at, what they're really bad at, what the, what people expect of them as well. And so, uh, you know, a very common process I've seen a lot of teams do, us included is have some generic prompt box, some entry point into AI in their product and sort of see what people use and gravitate towards in their scenario.

David Kossnick: I think that's a great starting point to learn. As you have deeper conviction, building purpose-built things for those use cases is really, really valuable cuz it's just a lot less effort at the end of the day. Writing prompts is really helpful for a really specific thing, but it's a lot of work for something you just want done quickly.

David Kossnick: And so some of the stuff we've been working on at the last month or two is, exacto knives, really specific things to open up exactly what you want in one scenario. and people have been loving them, which is great.

AJ Asver: I have one feature request, which is please automatically work out what type of column I need. Based on like the description of the column That seems like an easy one for AI to, to solve.

David Kossnick: You know, that's an interesting one cuz we, we actually do do it today, but in a subtle way actually. And yeah, this is like really in the weeds, but the text column type in Coda is actually an inferred column type. If you put a number in there or any other kind of structure piece of data, it will actually be able to, you know, operate on that data as if it were that type.

AJ Asver: Well, I am very, very much looking forward to seeing more AI in my Coda docs and also very excited to see where this all goes. think what you guys are doing with Coda and AI is really, really, really, really cool and also just very helpful. Like it saved me a lot of time and I think other people too. Um, once it's available in beta, I'm sure there'll be lots of new use cases that we haven't even seen yet.

David Kossnick: I did have one last question for you as well, which is, uh, you know, since you're working on AI every day on agents in particular, I'm curious how you think about sort of the future of agents in productivity. Like, what do you imagine agents are gonna do in finance and in every vertical, uh, you know, in collaboration with people?

AI agents

AJ Asver: That is something I spend a lot of time thinking about, um, in between like building actual agents. And right now I would say that we are vastly underestimating what's possible based on what we've been able to achieve at departure with our agents and the fact that they can just follow a set of instructions from a Google doc and carry out, you know, a very complex compliance task, like,

David Kossnick: It's not a Coda doc? AJ!

AJ Asver: That's true. We should be using Coda docs. Yes. A Coda doc, sorry. being able to follow the instructions and just carry out a task with the tools given. That's like a very novel and pretty awesome, uh, thing to be able to do because now you can automate a lot of repetitive tasks that have to be done very manually today. And so I imagine we're gonna see a lot of. This idea of intelligent automation as we've been calling it, where you aren't just doing the kind of robotic process automation or like workflow automation that you did before where you're connecting things together with conditional rules. But you're actually now using essentially prompt engineering to automate a task fully with lots of different steps and lots of different tools being used just by providing the instructions.

AJ Asver: In a way, you're, you are used to doing it already if you are solving this manually with the team, which is essentially a user manual, a set of, um, kind of procedures. And so you are gonna think about the many different use cases for that, not just in finance, but in processing forms in healthcare or insurance or in all kinds of other places where these manual, workflows are being done.

AJ Asver: I think it's gonna free up people to have a lot more time to do more interesting creative, strategic work, than doing this more repetitive kind of, Tedious work that exists today. So we're very excited to see where that goes. And we're at the very early stages of it today, but, um, I I think it's gonna move very quickly.

David Kossnick: Amazing.

Wrapup

AJ Asver: David, really appreciate it. you for taking the time to demo Coda AI, for talking to us a little bit about the kind of story behind how this feature came about and then for also helping me be more productive, with my Coda workflows as well. I'm really excited to see, product launch soon, beta and just a reminder for everyone where can they find, AI if they wanna sign up for it.

David Kossnick: coda.io/ai.

AJ Asver: Awesome. Well thank you very much David, and you everyone for listen in to this week's episode of the Hitchhikers Guide to ai and I will see you on the next one.

​

First, welcome to the new subscribers who hitched a ride with us this week! Last week I shared the news that I’m starting a new startup Parcha in the AI space and said I would follow up with more details soon. As promised, this week, I wanted to share the story behind Parcha and why we’re so excited about building AI agents to supercharge businesses.

Before I jump into this week's post, I have an exciting announcement:

On June 6th, The Hitchhiker’s Guide to AI and Parcha are co-hosting an AI dinner with our friends over at Hireflow.ai. If you’re based in SF and are either actively building something in AI or are interested in meeting other engineers exploring AI, we would love to meet you! We have limited spots, and are prioritizing AI hackers/engineers. If you’re interested, RSVP for the dinner here ASAP and we will get back to you.

RSVP

On to this week’s update…

In this post, I will talk about the problem our new startup Parcha is solving, how AI enables a solution, and how we plan to differentiate from the competition. Let’s jump in!

Fintech’s scaling problem

When I started this newsletter just under six months ago, my goal was to help people that was to share my insights with other people that were new to the space, like myself. My inspiration for writing was from Paul Graham, the founder of YC and prolific essay writer who once said that “writing is a form of thinking.” Writing about AI would help me think about the space and where I might fit in.

You may not know that before writing this newsletter, I had been in product management for over a decade, most recently in fintech at Coinbase and Brex. I joined Coinbase as the second PM hired to work on the consumer app at the height of the 2017 crypto bull run. During that time, I saw Coinbase 5X in size, experienced the depths of the crypto winter, and learned how money movement, compliance, and fraud prevention worked at a hypergrowth fintech company.

Even though you might think of Coinbase as a crypto company, most transactions on the platform at the time were between fiat and crypto. That meant money moving between credit cards and bank accounts through the traditional financial system, which was and still is far less technologically sophisticated than you might expect. In fact, every day, there was a team of operations folks at Coinbase that would manually process incoming wires in Coinbase’s bank account to make sure the money matched up to the correct Coinbase customer accounts to fund their crypto purchases!

Later when I joined Brex, a fintech company providing corporate credit cards and cash management accounts for startups, I was surprised to see a similar story. This time it was teams of operations folks manually reviewing customer applications, making underwriting decisions for credit limits and manually processing customer’s check deposits. In fact, in the early days of Brex Cash, the bank-account replacement product I helped launch, if you wanted to deposit actual cash into your account, your best bet was it mail it to HQ!

We did try to automate as many of these manual workflows at possible at Brex. My co-founder Miguel and I led a team of ~200 PMs, engineers, designers, data scientists, and cross-functional partners tasked with overhauling Brex’s onboarding, compliance, fraud, and underwriting systems. We made a lot of progress and helped the company scale its customer base 10X in a year. However, many manual workflows remained and were eventually outsourced to a third-party contractor or Business Process Offshoring (BPO) service. The challenge with these services is their agents churn a lot, they require a lot of upfront investment in training and you need to give the vendor access to internal tools and sensitive data.

Isn’t this just an excellent example of Y Combinators’ mantra of “do things that don’t scale?” After all, both Brex and Coinbase were YC companies. But in reality, in the zero-interest rate environment, we were in for the last decade, this approach of having armies of operations people doing a lot of manual work did scale. Many fintech companies like Brex and Coinbase have grown aggressively with armies of operations people working tirelessly behind the scenes to keep money moving smoothly, mitigate risk, and manage compliance.

The problem with this approach is that, unlike software, these operations backends in the fintech industry scale linearly at best. The more customers you have, the proportionally larger your team needs to get.

Fast-forward to today, these same fintech companies are under immense pressure to lower costs and extend their runway to ride out the current economic uncertainty, ideally by becoming profitable. That usually means that headcount has been frozen or reduced even as customer growth continues. But as I said earlier, these teams scale linearly, so at some point, you hit a ceiling on how many customers you can serve.

In my post on navigating the AI landscape to where I identified opportunities for AI startups, I said the following:

“There might be industries that are both fragmented and have knowledge workers who carry out repetitive software-based workflows. These industries are probably ripe for disruption because (1) there's no dominant player that has a structural advantage, and (2) repetitive workflows can be quickly augmented with AI.”

From my own experience, I haven’t seen any dominant players today that own workflow automation in compliance and operations in fintech and, more broadly, in finance. This is a perfect example of the type of problem that I think is an excellent opportunity for an AI startup. Then last month, a critical innovation in AI unlocked the solution: autonomous AI agents.

AI Agents unlock intelligent automation

The emergence of large language models, such as GPT-4, has enabled AI models to perform chain-of-thought reasoning. This means an AI model can follow multiple steps, create an internal model of what is happening in a scenario, and reason through it to get to a solution (see example below).

This chain of thought reasoning is so impressive that we can now create autonomous AI agents that can complete tasks in sequence, using tools to solve problems, as I covered in my previous posts on autonomous AI agents like BabyAGI:

“I believe there is a more profound implication for these types of iterative, LLM-powered autonomous AIs that can use other tools/models to complete tasks. It feels much closer to how humans interact with the world, whereby we have additional sensory input beyond our brain’s intelligence, and we use tools to complete our objectives.”

Many knowledge workers today perform repetitive work that has to be done by humans due to nuances in language, analysis, and steps that are hard to automate, even if they are very repetitive. The emergence of AI agents will change how these tasks are done. AI agents can carry out tasks like manual reviews in fintech, which would take humans several minutes or hours to complete, and can now be completed in seconds. But it’s not just time being saved. AI agents are more scalable (up and down), easier to audit, faster to train, and cheaper to run than humans carrying out the same tasks. They can make existing human teams far more productive too.

At Parcha, we are excited to be at the forefront of building these agents. Our platform empowers existing operations teams by supercharging them with AI agents. This will enable them to do more work and no longer scale linearly. One person can now manage dozens, if not hundreds, of agents. Our platform is unique because how you create and deploy agents is approachable to nontechnical folks. All you have to do is give our agents the same policies, procedures, and training docs you give a new team member and access to the same tools. That’s it!

Once you have deployed these AI agents, you can monitor, manage, scale up, and scale down your digital workforce from one central dashboard. Because all the agents share context, you connect them together to complete larger, more complex tasks too, much like you run a multi-layered organization today.

Open-source models and enterprise

The final piece of the puzzle that makes Parcha possible is building an enterprise-grade product that prioritizes privacy and data security, a critical feature for finance use cases. The reality is, most enterprise companies are not comfortable with handing over their data to OpenAI.

Enter open-source models…

Open-source versus closed-source models are a hot topic in AI and very relevant to enterprise use cases. At an AI event I attended, Greg Brockman, CTO of OpenAI, recently shared his belief that closed-source models will continue to be most centralized and at the bleeding edge, while open-source models will be about one to two years behind. In a previous post, I also thought there would be a divergence between open-source and closed-source models:

Over time though, open-source will probably lag behind proprietary models for three primary reasons:

1. Cost to train: As models get bigger and bigger, they will become more expensive. Open-sourced models will therefore need more well-funded sponsors to develop them. Today StabilityAI might be able to afford to sponsor a GPT-3 equivalent model that costs $10M to train, but next year they might not be able to afford to support a GPT-5 equivalent that costs $100M to train.

2. Proprietary datasets: Well-funded companies like OpenAI license additional private data sets to improve their models. This will make it harder for open-sourced models to compete on performance and they may lag one or two generations behind proprietary models. For many customers, the price and versatility of open-sourced models will be the deciding factor. Using a lesser performant open-source model will be a path many developers take to get started cheaply before developing their own models or switching to proprietary ones once they have enough scale.

3. Closed research: Many contributions that have enabled open-source AI models to come from private companies, like Google’s Transformer architecture. That might change as the need for Big Tech to compete in AI takes a higher priority over publishing research.

After seeing the sheer pace of innovation in the open-source AI community since Meta’s release of LLAMA though, I’m now far less convinced that open-source will lag far behind proprietary closed-source models. It's no longer clear to me that just having more data at this point and more parameters will solve the problem. There may be other innovative and exciting ways that we can improve language models beyond that, which the open-source community may find and experiment with just as quickly as a closed-source community in the big research labs.

In fact, it seems increasingly likely that language models will follow a similar path to open-sourced generative image models like Stable Diffusion. This collective momentum of the AI community, along with many companies now releasing their own models like MosaicML and Databricks, is really accelerating open-source innovation. This is reflected in a recently leaked memo by a Google employee titled “We Have No Moat, And Neither Does OpenAI":

While our models still hold a slight edge in terms of quality, the gap is closing astonishingly quickly. Open-source models are faster, more customizable, more private, and pound-for-pound more capable. They are doing things with $100 and 13B params that we struggle with at $10M and 540B. And they are doing so in weeks, not months.

While close-source model creators like Google still have a huge advantage in distribution and existing ecosystems, open-source models will be an excellent option for startups developing AI products for end consumers with stricter privacy or security requirements. Furthermore, open-source models can the fine-tuned to better fit a particular vertical use case, with far fewer parameters.

Open-source models, therefore, become an excellent option for startups building for enterprises in regulated industries, like fintech and finance. Being able to control the data around the models, optimize their output, and deploy them on custom infrastructure that prioritizes privacy and security makes open-source models a great fit.

At Parcha, we’re developing fine-tuned models based on open-source LLMs that can easily be deployed in an enterprise setting. We believe this approach will be a differentiator in the long term because we will have better control over our agent’s capabilities, and most importantly, our customers will have better control over their data.

Lastly, there is timing…

When starting a startup, there’s one factor you often can’t control: timing. I will never forget a piece of advice I once heard from a partner at Sequoia Capital at a YC dinner I attended over a decade ago when I was doing my first startup. The advice went something like “If there’s a wave that can give your startup momentum, you should ride it.” The idea is that it’s hard enough doing startups as it is, and you should take advantage of any tailwinds you can get. AI is the most important technology wave we have seen since mobile, and I believe the magnitude of its impact will far surpass any predecessor. Coupling this with the need for enterprise companies to be under pressure to manage costs and find more efficiencies, it feels like an opportune moment for AI-powered automation.

I always knew I wanted to start a startup again at some point in the future. After spending six months on the shore watching the AI wave gaining momentum, I knew it was the right time to grab my surfboard, and luckily, my cofounder Miguel was ready to paddle out with me too.

Thanks for reading this update. If you’re considering starting an AI startup, I would love to hear from you - I’m happy to share what I’ve learned so far. If you’re curious about joining an AI startup, we’re hiring at Parcha and would love to hear from you too!

Next week we will go back to our regular programming, breaking down how the latest developments in AI will change how we live, work, and play.

P.S. Don’t forget to subscribe!

First, a huge welcome to the 100+ new subscribers that hitched a ride with us in the last two weeks. In this week’s post I’m sharing an interview I did recently with Josh Albrecht, CTO and founder of Generally Intelligent. Generally Intelligent is an AI research team focused on building general-purpose AI agents that can be safely deployed in the real world.

Before I jump into the interview, you might be wondering why I skipped a week. Well, I have some exciting news: After 6 months of exploring AI, I’ve decided to start a startup with my friend and former engineering partner at Brex, Miguel Rios Berrios.

Miguel and I worked together with an amazing team to grow Brex's customer base 10X in just a year by making onboarding, compliance, risk management and underwriting more automated. Despite 200+ people working on this problem there were still many workflows that required humans in the loop. We believe AI can help.

At Parcha, we’re building enterprise-grade AI agents that supercharge your business. We are starting with fintech companies that can use our AI agents to accelerate their manual workflows in operations and compliance. Over the coming weeks, I’ll be sharing more about why we started Parcha and what our core thesis is.

We’re hiring a small team in San Francisco and San Juan, Puerto Rico. We currently have open roles for an senior full stack engineer, applied AI engineer who has experience working with LLMs and a UX designer.

If you’re interested in learning more about what we’re doing or would like to join our team please email us via founders@parcha.ai or reply directly to this email.

As for this newsletter, I still plan to post as often as I can because I find writing these updates to be a great way to keep up with the AI space. If you are reading (or listening) to this for the first time, please don’t forget to subscribe:

Subscribe now

On to the interview…

Interview: AGI and developing AI Agents with Josh Albrecht, CTO of Generally Intelligent

I’ve been spending a lot of time researching, experimenting and building AI agents at Parcha. That’s why I immediately jumped at the chance to interview AI researcher Josh Albrecht, who is the CTO and co-founder of Generally Intelligent. Generally Intelligent’s work on AI Agents is at the bleeding edge of where AI is headed.

In our conversation, we talk about how Josh defines AGI, how close we are to achieving it, what exactly an AI researcher does, and his company’s work on AI agents. We also hear about Josh’s investment thesis for Outset Capital, the AI venture capital fund he started with his co-founder Kanjun Qui.

Overall it was a really great interview and we covered a lot of ground in a short period of time. If you’re as excited about the potential of AI agents as I am or want to better understand where research is heading in this space, as I am this interview is definitely worth listening to in full.

Here are some of the highlights:

• Defining AGI: Josh shares his definition of AGI, which he calls Human-level AI, a machine’s ability to perform tasks that require human-like understanding and problem-solving skills. It involves passing a specific set of tests that measure performance in areas like language, vision, reasoning, and decision-making.

• Generally Intelligent: General Intelligence's goal is to create more general, capable, robust, and safer AI systems. Specifically, they are focused on developing digital agents that can act on your computer, like in your web browser, desktop, and editor. These agents can autonomously complete tasks and run on top of language models like GPT. However, those language models were not created with this use case in mind, making it challenging to build fully functional digital agents.

• Emergent behavior: Josh believes that the emergent behavior we are seeing in models today can be traced back to training data. For example being able to string together chains of thought could be from transcript of gamers on Twitch.

• Memory systems: When it comes to memory systems for powerful agents, there are a few key things to consider. First of all, what do you want to store and what aspects do you want to pay attention to when you're recalling things? Josh’s view is that while it might seem like a daunting task, it turns out that this isn't actually a crazy hard problem given that we know how to do this already for non AI systems.

• Reducing latency: One way to get around the current latency when interacting with LLMs that are following chains of thought with agentic behavior is to change user expectations. Make the agent continuously communicate updates to the user for example vs. just waiting for to provide the answer. For example, the agent could send updates during the process, saying something like "I'm working on it, I'll let you know when I have an update." This can make the user feel more reassured that the agent is working on the task, even if it's taking some time.

• Parallelizing chain of thought: Josh believes we can parallelize more of the work done by agents in chain of thought processes, asking many questions at once and then combining them to reach a final output for the user.

• AI research day-to-day: Josh shared that much of the work he does as an AI researcher is not that different from other software engineering tasks. There’s a lot of writing code, waiting to run it and then dealing with bugs. It’s still a lot faster than research in the physical sciences where you have to wait for cells to grow for example!

• Acceleration vs deceleration: Josh shared his viewpoints for both sides of the argument for accelerating vs decelerating AI. He also believes there are fundamental limits to how fast AI can be developed today and this could change a lot in 10 years as processing speeds continue to improve.

• AI regulation: We discussed how it’s challenging to regulate AI due to the open-source ecosystem but that some form of regulation is probably required.

• Universal unemployment: Josh shared his concerns that we need to get ahead of educating people on the potential societal impact of AI and how it could lead to “universal unemployment”.

• Investing in AI startups: Josh shared Outset Capital’s investment thesis and how it’s difficult to predict what moats will be most important in the future.

Episode Links:

The Hitchhiker’s Guide to AI: http://hitchhikersguidetoai.com

Generally Intelligent: http://generallyintelligent.com

Josh on Twitter: https://twitter.com/joshalbrecht

Episode Content:

00:00 Intro

01:42 What is AGI?

04:40 When will we know that we have AGI?

05:40 Self-driving cars vs AGI

07:10 Generally Intelligent's research on agents

09:51 Emergent Behaviour

11:07 Beyond language models

13:17 Memory and vector databases

15:25 Latency when interacting with agents

17:13 Interacting with agents like we interact with people

19:08 Chaing of thought

19:44 What do AI researchers do?

21:44 Accelerations vs. Deceleration of AI

24:05 LLMs as natural language-based CPUs

24:56 Regulating AI

27:31 Universal unemployment

Hi Hitchhikers!

I’m excited to share this latest podcast episode, where I interview Charlie Newark-French, CEO of Hyperscience, which provides AI-powered automation solutions for enterprise customers. This is a must-listen if you are either a founder considering starting an AI startup for Enterprise or an Enterprise leader thinking about investing in AI.

Charlie has a background in economics, management, and investing. Prior to Hyperscience, he was a late-stage venture investor and management consultant, so he also has some really interesting views on how AI will impact industry, employment, and society in the future.

In this podcast, Charlie and I talk about how Hyperscience uses machine learning to automate document collection and data extraction in legacy industries like banking and insurance. We discuss how the latest large-scale language models like GTP-4 can be leveraged in enterprise and he shares his thoughts on the future of work where every employee is augmented by AI. We also touch on how AI startups should approach solving problems in the enterprise space and how enterprise buyers think about investing in AI and measuring ROI.

Finally, I get Charlie’s perspective on Artificial General Intelligence or AGI, how it might change our future, and the responsibility of governments to prepare us for this future.

I hope you enjoy the episode!

Please don’t forget to subscribe @ http://hitchhikersguidetoai.com

Episode Notes

Links:

• Charlie on Linkedin: https://www.linkedin.com/in/charlienewarkfrench/

• Hyperscience: http://hyperscience.com

• New York Times article on automation: https://www.nytimes.com/2022/10/07/opinion/machines-ai-employment.html?smid=nytcore-ios-share

Episode Contents:

00:00 Intro

01:56 Hyperscience

04:52 GPT-4

09:41 Legacy businesses

11:13 Augmenting employees with AI

15:48 Tips for founders thinking about AI for enterprise

20:34 Tips enterprise execs considering AI

23:49 Artificial General Intelligence

29:41 AI Agents Everywhere

32:12 The future of society with AI

37:44 Closing remarks

Transcript:

HGAI: Charlie Newark French

Intro

AJ Asver: Hey everyone, and welcome to the Hitchhiker Guide to ai. I am so happy for you to join me for this episode. The Hitchhiker Guide to AI is a podcast where I explore the world of artificial intelligence and help you understand how it's gonna change the way we live, work, and play. Now for today's episode, I'm really excited to be joined by a friend of mine, Charlie Newark, French.

AJ Asver: Charlie is the CEO of hyper science, a company that is working to bring AI into the enterprise. Now, Charlie's gonna talk a lot about what hyper science is and what they do, but what I'm really excited to hear Charlie's opinions on is how he sees automation impacting our future.

AJ Asver: Both economically, but as a society, as you've seen with recent launch of G P T four and all the progress that's happening in AI, there's a lot of questions around what this means for everyday knowledge workers and what it means for jobs in the future. And Charlie, has some really interesting ideas about this, and he's been sharing a lot of them on his LinkedIn and I've been really excited to finally get him on the show so we can talk. Charlie also has a background in economics and management. He studied an MBA at Harvard and previously was at McKinsey, and so he has a ton of experience thinking about industry as a whole, enterprise and economics and how these kind of technology waves can impact us as a society.

AJ Asver: If you are excited to hear about how AI is gonna impact our economy, our society, and how automation is gonna change the way we work, then you are gonna love this episode of The hitchhiker Guide to ai.

AJ Asver: Hey Charlie, so great to have you on the podcast. Thank you so much for joining me.

Charlie: Aj, thank you for having me. I'm excited to discuss everything you just talked about

AJ Asver: maybe to start off, one of the things I'm really excited to understand is how did you end up at Hyper Science and what exactly do they do?

Hyperscience

Charlie: Yeah, hyper Science was founded in 2014. It was founded by three machine learning engineers. so We've been an ML company for a long time. My background before hyper science was in late stage investing. Had sort of the full spectrum of outcomes there.

Charlie: Some why successful IPOs, some strategic acquisitions, and then a lot of miserable, sleepless nights on some of the other areas. I found, hyper science, incredibly impressed with, their ability to take cutting edge technology and apply it to real well problems. We use machine vision, we use large language models, and we use natural language processing, and we use that those technologies to speed up back office process.

Charlie: The best examples here are a loan origination, insurance claims processing, customer onboarding. These are sort of miserable long processes, a lot of manual steps, and we speed those up. With some partners taking it down from about 15 days to four hours.

Charlie: So all of that data that's flowing in of this is who I am, this is what's happened, this is the supporting evidence. We ingest that. It might be an email, it might be a document. It's some human readable data. We ingest that, we process it, and then ultimately the claims administrator can say, yes, pay out this claim, or no, there's something.

AJ Asver: Yeah, so what, what you guys are doing essentially is you had folks that were previously looking at these documents, assessing these documents, maybe extracting the data out of these forms, maybe it was emails, and entering those into some database, right? And then decision was made, and now your technology's basically automating that. It's kind of sucking up all these documents and basically extracting all that information, helping make those decisions. My understanding is that with machine learning, what you're really doing is you've kind of trained on this data set, right, in a supervised way, which means you've said like, this is what good looks like.

AJ Asver: This is what, you know, extracting a, a, a data from this form looks like now we're gonna teach this machine learning algorithm how to do it itself. Now what what I found really interesting is that, That was kind of where we made the most advancements, really in kind of AI over the last decade, I would say.

AJ Asver: Right? It's like these deeper and deeper neural networks. They could do machine learning in very supervised ways, but what's recently happened with large language models especially, is that we've now got this like general purpose AI that, you know, GPT-4, for example, just launched this. and there was an amazing demo where I think the CTO of OpenAI basically sketched on like the back of a napkin, a mockup for a website, and then he put in in GPT and it was able to like, make the code for it.

AJ Asver: Right. So when you think about a general purpose, let large language model like that, compared to the machine learning, y'all are using do you consider that to be a tool that you'll eventually use? Do you think it's kind of a, a threat to like the companies that have spent the last, you know, 5, 6, 7 years, decades, maybe kind of perfecting these ma machine learning tools or, you know, I, is it something that's gonna be more like different use cases that won't be used you know, by your customers?

GPT-4

Charlie: Open ai ChatGPT, GPT-4. Anything that's been, the technology you're speaking about has really had two fundamental impacts. There's been the technology. It's just very, very cutting edge, advanced technology. And then you've got the adoption side of it. And I think both sides are as interesting as each other.

Charlie: On the adoption side, I sort of like to compare it to the iPhone that there was a lot of cutting edge technology, but what they did is they made that technology incredibly easy to use. There's a few things that Open AI has done here that's been insanely impressive. First, , they use human language. Um, humans will always assign a higher level of intelligence to something that speaks in its language.

Charlie: The other thing, it's a very small thing, but I love the way that it streams answers so it doesn't have a little loading sign that goes around and dumps an answer on you. It's like, it's almost like it's communicating with you. Allow you to read in real time and it feels more like a conversation.

Charlie: Obviously the APIs have been a huge addition. It's just super easy to use, so that's been one big step forward. But it's a large language model. It's a chat bot. I don't wanna underestimate the impact of that technology, but my thoughts are AI will be everywhere. It's gonna be pervasive in every single thing we do.

Charlie: And I hope that chatbots and large language models aren't the limitation of ai. I'd sort of like to compare chatbots and large language. To search the internet is this huge thing, one giant use case that if you ask people what is the internet? They think it's. Google, And that's the sort of way I think this will play out with AI and the likes of a whichever large language model and chatbot wins to be the Google of that world, which at the moment appears very clearly to be open ai.

Charlie: But there's some examples of stuff that. Certainly right now, that approach wouldn't solve. I'll give you a few, but the, this list is tens of thousands of use cases long. We spoke about autonomous vehicles earlier. I suspect LLMs are not the approach for that physical robotics. Healthcare detecting radiology diseases, fraud detection.

Charlie: I'm sure if you put in like a fake check in front of GPT-4 right now it was written on the napkin, it might be able to say, okay, this is what the word fraud means. This is what a check looks like, but you've got substantially more advanced ai AI out there right now that looks at. , what, what was the exact check that Citibank had in nine, in 2021?

Charlie: Does this link up with other patterns that should be happening with this kind of transaction and so, I think that you are gonna have more dedicated solutions out there than just sort of one chat or interface to rule them all, would be my guess. Yeah. Hyper science. There are things that chat G p t does, or g p t four does right now that we do.

Charlie: Machine vision is one that's an addition that appears to be working alongside their large language model. So they're combining different technologies versus just a large language model, is my guess. I obviously don't have work ins. But we build a platform here at hyper science that builds workflows, that enriches data, validates data, compares data looks at data that's historically come into an organization that might not be accessible to sort of public chat bots or large language models.

Charlie: I think the question that you sort of said at the beginning, Could we be using chat, G p t or g p T four? Absolutely. And I think that a lot of startups could, but I suspect that, that you, what you'll see here is a lot of the startups that spin up leveraging this and building something far greater from a user experience for a very specific use case versus open AI solving all the sort of various small problems along the way, if that makes.

AJ Asver: Yeah, I think that makes a lot of sense. And it's one thing I've been thinking a lot about. I actually wrote a blog post recently about this as kind of how these foundational models are gonna become more and more commoditized and it's gonna create this massive influx of products built on top of it.

AJ Asver: What I find really interesting is that you know, GPT, you can actually use that transformer for a lot of different things that aren't necessarily just a chatbot.

AJ Asver: Right. So you mentioned the fraud use case. If you send a bunch of patents of fraud to a large transformer, its ability to actually remember things makes it very good at identifying fraud. And in fact, I was talking to a friend that, that worked at Coinbase in their most recent fraud control mechanism.

AJ Asver: They went from kind of linear aggressive models to deep learning models, to eventually actually using transformers and it, and it was far more, far more effective. So I guess coming back to the question, do you see a world where instead of building these. Focused machine learning models for particular use cases like you know, ingesting documents or maybe making sense of data and extracting that data and tabulating it into a, into a database that you might and one day end up actually just having a general pre-trained transformer that you are then essentially fine tuning with one shot. Kind of tuning me. Like, this is how you extract a document for one of our clients. This is how you you know, organize this information into like loan data. Is that a world we could move in? That's probably different from where we are today and, and maybe a different world of hypo sciences too.

Legacy businesses

Charlie: Look, it would be a very different world. I think the next five, 10 years are leveraging the, the. Of technology that OpenAI is building and maybe that specific technology, as you sort of say of commod, some commoditized layer and building. Workflows on top of that, I'll give you the, just the harsh reality of what the world looks like in reality out there.

Charlie: Right? So this isn't just a single use case that I go and type something in as a consumer on on the internet at a bank in the uk they have a piece of cobalt that it was written in the 1960s that is still live in their mortgage processing.

AJ Asver: Wow.

Charlie: Rolling out, even just from a compliance level, any change to that mortgage processing that isn't piecemeal fashion, that doesn't about the implications, that doesn't think about customer interactions in a a week timeframe or a a year or three year timeframe is just not dealing with the reality of the situation on the ground.

Charlie: These are complex process. People get fired if you take a mortgage processing system down for minutes. And they're complex. So do I think that's a possibility in the future? It's absolutely possible. I think the best use of GPT-4 right now is to go and build the extensive workflows that require a little bit of old-fashioned AI, as well as cutting edge AI to, to have an end-to-end solution for a specific problem versus assuming that we're ever gonna get something. But you just say, okay, I'd like to know, should I give this customer this mortgage?

Charlie: And you get an answer back. That, to answer that question is still a very complex process.

Augmenting employees with AI

AJ Asver: Yeah, and I think we, especially in the technology industry, especially someone like me that spends so much time thinking about, talking about reading about AI, kind of forget that a lot of these legacy businesses can't move as fast as we think. I mean, we see like Microsoft moving quickly and slack moving quickly for example.

AJ Asver: But those are all like very software focused consumery businesses that you know, necessarily touching like hard cash and stuff like that where there's a lot more compliance and regulations. So that makes a lot of sense. So then what we really are thinking about is like you kind of have humans that can be, as you've put it before in some of your predictions around ai, augmented, right?

AJ Asver: These, this idea of like an augmented employee that can use AI to, to help them get things done, but we're not necessarily replacing them straight away. Like, talk to me about what, what you see as a future of kind of augmented employees and, and kind of co-pilots as they're also called.

Charlie: Totally. So the augmented employee is a phrase that I've been using. For about 10 years, it's been a prediction for a while now. It didn't used to be a particularly popular one. You would get a whole load of reports even from the like big consultancy groups that say these five jobs are definitely gone in five to 10 years.

Charlie: That five to 10 years has come and gone over that period of time, or I'll give you a longer period of time. Over the last 30 years, we've added 30 million jobs here in the us about a on average. Obviously, it's been a bit of fluctuation. There's no good sign on a short term decision making time horizon that jobs are gonna be wildly quickly displaced.

Charlie: There's very little evidence. That's my. What do I mean by short term horizon? I really mean by the when, what a large enterprise, which is what I, my company serves and what I'm interested in serving, makes decisions 5, 10, 20, maybe even as that's the sort of edge of where I think things start to really change.

Charlie: Fundamentally you should make decisions around software. And AI in this case, substantially helping people do a better job. The, the, the first time I read this getting sort of a mainstream idea was about a year ago. And by mainstream, I mean outside of the tech industry New York Times wrote an article where the title was something like in the.

Charlie: Fight between humans and robots. Humans seem to be winning. I think that was just a very interesting change of thought. And there was a line in there that says the question used to be, when will robots replace humans? The better question is, which I absolutely love this phrasing of it, when will humans that leverage robots replace humans that don't leverage robots?

Charlie: And I think that's the right way to think about it. I, I'll give you a couple of examples. One with sort of, non-AI OpenAI technology and then chat. G PT Speci specifically, or, or G P T. Radiology is something that's been talked about for a while. This was a giant step forward where software AI could detect most.

Charlie: Cancers most diseases, basic diseases better than humans could just had higher accuracy. And the prediction for five, seven years was, this is the end of radiologists. We've seen no decrease in radiologists. If you want to go and get a cancer screening now, you're gonna probably look at a six to nine month wait.

Charlie: I don't have any issues, but I'm waiting for a cancer screen right now. Just a nice safety check that I want to my own benefit and cause of the sheer backlog of work. , I can't get that done. I can't get it done for a while. So is the, the future for me is in two or three years time, there's not fewer radi.

Charlie: There's just much higher accuracy and much shorter wait, wait times. And maybe the future, as I say, which I'm sure we'll speak about 20 years down the line is is I can just go to a website. They can do some scan of me, and then they can give me the answer within seconds. I, I, I can't wait for that, but it's just not here today.

Charlie: And I'll give you one aj one quick open AI example. When ChatGPT came out there was so many sort of, this is not ready for mainstream things that went round. And the, the way that I thought about it is, if you want ChatGPT then to write you a sublease because you want to lease your apartment and you want it to be flawless, you just want to click send to the person that's doing the sublease on. It's nowhere near medi ready for mainstream. If you wanna cut down a law legal person's work by 90% because the first draft is done. They're gonna apply local laws, they're gonna look at a few nuances. They're gonna understand the building a bit then it's so far beyond ready for mainstream. It should be used by everybody for every small use case it can. So I think it's human augmentation for a while. I think that jobs don't go away for a while, and I sort of like to compare it to the internet a little bit here, which is we use the internet today and every single thing we do and it makes our jobs substantially easier to do. It makes us more effective at them, and that's what I think the next sort of 10 years at least looks like for AI within the work.

Tips for founders thinking about AI for enterprise

AJ Asver: The thing you mentioned there, I find to be really fascinating is this idea that, you know, we're not gonna replace humans immediately. That's not happening. But people thought that for a long time. Right. And it almost feels like with every new wave of technology, there's this new hum of people saying like, we don't replace humans, we're gonna replace humans.

AJ Asver: Right. But at the same time, I, I kind of agree with you, having spent a lot of time using chat JBT and working with it, I found that it certainly augments my life, in writing My substack in fact, in this interview preparing for this interview, I actually asked it to help me think about some interesting questions to ask you based on some of the posts you'd written.

AJ Asver: Because I'd read some of your posts on, on, on LinkedIn fairly regularly, but I couldn't remember all of them, so I actually asked the Bing ai chat to help me. Right. And then when you think about these especially regulated environments where you. The difference between right and wrong is actually someone's life or a large sum of money or breaking the law, then it really matters. And in that case, augmentation makes a lot of sense. Now, the reality is, AI, especially kind of large language models in building on top of open AI is a fairly low barrier to entry right now. That's why we're seeing a lot of companies in copywriting, in collaboration, in presentations, and the challenge with that is if there's an incumbent in the space, That already exists. It's very hard to beat them on distribution right now. Where I did see an interesting opportunity is exactly what you are talking about, is like going deep into a a fairly fragmented industry, which maybe has a lot of regulation or a lot of complexity, maybe disparate data systems.

AJ Asver: You mentioned kind of the. 30 year old like cobalt data system, right? Like that is a perfect place where you can go in and really understand it deeply. Now, as someone that's running a company that does that, I'm curious, like what advice do you have to founders or startups? I wanna take that path of like, Hey, we're gonna take this AI technology that we think is extremely powerful, but go apply it into some deep industry where you really have to understand the ins and outs of that industry, the regulation, the, the way people operate in that industry and in the problems.

Charlie: Absolutely a few thoughts. Firstly, make sure that you are trying to solve a problem. This is just general advice for setting up a business. What is the problem you're trying to solve? What is the end consumer pain point? For us here at Hyper Science, it's that people wait for their mortgage to be processed for six weeks.

Charlie: No good reason why that's happening. People wait for their insurance to be insurance claims to be paid out sometimes for two years. No good reason for that to be happening. So always start with the customer pain point, and then decide does the current technology, which is AI in this case, allow you for solving it?

Charlie: And then that gets you to the, does it allow you to solve for it? And what I've looked for here is, the more open AI can do it or G p t four can do it a whole load of diverse stuff, but your highest value things are gonna be what's just happening time and time again. If for us, like there is just a whole load of mortgages, that process not right now or there is just a whole load of insurance things that are processed and they're.

Charlie: Relatively similar, although we think of them as different. They've got a lot of, certainly to a machine similarities. So I'd look for volume. You can think of this as your total addressable market in terms of traditional vc, non-AI speak. But this is, is the opportunity big enough? And then the, the next thing I'd look for is repetitive tasks.

Charlie: So the more repetitive it is, the easier it's. You can go out and solve something really, really hard with a large language. But there's probably even easier applications that you can solve that are just super repetitive and you can take steps out. So I think that's it. Have the problem in mind.

Charlie: Think about volume, think about repetitive natures, and then one of the key things, once you've got all of that set and you've got, okay, this is an industry that's right for customer pain, right, for disruption. This is definitely a good application of where AI is today. I would think about ease of use above everything.

Charlie: My, my thinking is, and I spoke about this with open ai, one of the biggest things they've done is they've just taken exceptional technology, but made it so, so simple for someone that doesn't know AI to interact with. And the question I always get asked is the CEO of enterprise software, AI company is how can we upscale all of our employees?

Charlie: The answer to that is you shouldn't have to. This software should be so easy to engage with that your current employees should seamlessly be able to do it. There should be, if there is rudimentary training needed needed, your software should do that. And again, I like to compare this to the internet. We use the internet day in, day out.

Charlie: There has been 20 years of upskilling, but it's not really been that hard. Like I think if you took today the internet and you gave it to somebody 20 years ago, it might be a little bit advanced for. , but we've made software, internet software, so easy to work with that you don't need to know how the internet works.

Charlie: The number of people that know how the internet works, even the number of people that know how to truly code a website. Absolutely tiny fraction of the number of people that use the internet to improve their daily lives. So I'd say ease of use for AI is possibly as important as the technology.

Tips enterprise execs considering AI

AJ Asver: I love those insights by the way. And just to recap that you said go for a problem that has a lot of volume, whereas solving a real problem to end users, but there's clear volume or like, you know, a large addressable market. The other thing you mentioned was make sure it's repe repetitive tasks, like with l LM says it's temptation to go after these really complex problems, but like repetitive tasks are the ones that are most.

AJ Asver: That's probably the most incentive to solve as well. Right. And then the last thing you mentioned is like, it should be really intuitive for a, for an end user to use to the point where they don't have to feel like they need to be upskilled. Now, if you are a founder or a startup going down this path, the other thing you're thinking about is like, how do you sell into these companies?

AJ Asver: So maybe taking the flip side of that, if you are in the enterprise and you're getting approached by all these AI startups, they just got funded this year being like, we're gonna help you do this. We're gonna help you do that. We're gonna automate this. How do you decide when it's the right time to make that decision?

AJ Asver: How do you decide? Kind of, the investment on that and whether it's worth it. Like what, what are your thoughts on that?

Charlie: My thoughts on that are linked directly to the economic cycle we're in right now, which is not a pretty one. Somewhat of a maybe a mild recession, maybe the edge of a recession. And I see this from all of the CIO CEOs that we work with at the, the sort of large banks, large insurance companies.

Charlie: And my suggestion is this is I tell them to create a two by two matrix. You told everyone earlier, I started my career at McKinsey.

AJ Asver: Classic two by two.

Charlie: Love it. Two by two matrix. On one of the ax axis is short-term ROI on one of the ax axis is long-term roi and you want to get as much into the top right as possible and as few into the bottom left as possible.

Charlie: And for a y or artificial intelligence was considered ROI and not short term roi, which is a bit, they were treated by these large. As science experiments and you saw these whole, these whole roles form these whole departments form around transformation. The digital transformation officer, that is a role that just didn't exist five or 10 years ago, and these people were there to go and innovate within the organization and, and largely speaking, it wasn't wildly successful. A lot of these roles are sort of spinning down. You need to solve a business problem that the technology solves today and gives you a path to the long run. So, hyper science, we, I'll give you an example here. We add value out of the box, but we also understand where people are today and try to get them to where they want to go.

Charlie: So one of our customers, 2% of what they do is process fax. I hope that they are not processing faxes in five time, and I hope that we are giving them the bridge to that, but we better be able to do that today. And also paint them a, a sort of what I refer to as a future proof journey to where they want to head.

Charlie: So I think it's really about don't, don't do any five year projects. Like if a company comes and says to you things. When you say, can you do this? And they say, well, we could do that. I would run. Or if they're just saying yes to everything versus yes to something quite specific and a good startup, you know this better than anyone, aj, a good startup does something specific really well and then they build out that they have an MVP is one way of phrasing it.

Charlie: They have a niche is another way of phrasing. Yeah, go and sell something today really well, and all of those sort of long tail features around it, people will forgo for a period of time whilst you build those. , but you be, add value in the in the short term as well as be building something in the long run.

Artificial General Intelligence

AJ Asver: Yeah. That and that point you made about that kind of showing the short term value is really important, especially when you're trying to convert the kind of maybe biases around AI that exist in enterprise today, that it's, as you mentioned, kind of like a hobby project or a kind of experiment, or like this is kind of your, you know, your Moonshot kind of, kind of projects you wanna show them really, like, this is like ROI you're gonna get very quickly in the next one or two years and, and that's a really important point of it. Now all of this makes sense, the augmented employee, the co-pilot and, and like having these narrow versions of AI that are solving particular problems and, and I can see that working out, but I feel like there's this one big factor that we, that we have to think about that that could change all of this, in my view.

AJ Asver: And that is artificial general intelligence. And for folks that dunno what artificial general intelligence is, or AGI is it's called. That's really what open AI is trying to achieve long term. And it's the idea of essentially having intelligence that is the equivalent of a human. And it's an ability to think abstractly and to solve a wide, broad range of problem. In a, in a way that, that, that a human does. And what that means is technically, now, if you have an AGI and let's say the cost of running that AGI I is, you know, a hundredth or a thousandth of a cost of running a human, then potentially you could replace everyone with, with, with robots or you know, AI as, as it were.

AJ Asver: How does that factor into this equation? Is this so, You, you think about is it, like, what are your thoughts about it? Both, both as CEO of an enterprise company, but also as someone that's studied management and economics for the last decade? I'm really curious to, to hear where you think this is going.

Charlie: I don't think there's anything unique about a soul or something that can't be replicated in the human mind. And to your point, I actually think that we, we think of AGI sometime as when I hear definitions, I hear human-like, or human level intelligence if this happens or when it happens, because it, there's no doubt it will it will be substantially smarter, incredibly quickly than a human. And you look at the difference in humans of intelligence, someone you just pick off a street, 110 I IQ or whatever level it is, versus an Einstein with 170 iq, that difference is enormous. Now, imagine that that's the equivalent of 170 iq, but it's a thousand or 10,000 or whatever it is. I think you will get to the point where if you have AGI extremely quickly, you will. Be far beyond them not being able to do any job. There will be absolutely zero they can't do

Charlie: now I don't see that today. I, my best guess of a time horizon is post 20 years sub a hundred years. That's a nice vague timeframe, but that's sort of how I think about it. 20 years is your classic sort of decision making timeframe and for, for someone. Building or someone running an enterprise software company, it's not an interesting question of what do we do with agi?

Charlie: For someone thinking about designing a society thinking about economic systems, thinking about regulation, it's an extremely good time to start thinking about those questions. Let, let me start by AJ speaking about why I don't think it's here today. And then we can perhaps think. What world where it is here looks like, which I'm quite excited about by the way.

Charlie: I don't view as a, a dystopian outcome. Our current approach to AI today is machine learning. We spoke about that earlier today. Machine learning requires sort of three things, compute algorithms and data, and on all three of them, I think to have true agi we're. The compute power I think we need some leap forward there.

Charlie: It might be quantum computing. There's a lot of exciting happening there. The timeframes there. I'm not as close to that as I am and I ai, but no one's speaking about quantum computing on a sort of one year time horizon. They're speaking about it again on a 10, 20 year time horizon. The second thing is the algorithms.

Charlie: I just, from what we see out there, even with the phenomenal stuff that op that open AI is building, I don't see algorithms out there doing true AI true AGI. They are. The large language models, I, will say that I'm incredibly impressed with how GPT-4 plays chess. It's still not at a, their level of an algorithm that is designed specifically for chess but it's pretty damn good. So, my, my thoughts on the, the algorithms evolve every day, but certainly we're still not there today. And then one of the big hurdles is gonna be data a human ingest data. Rapidly all the time via a series of senses. You can think of that as five senses. You can think of it as 27 senses.

Charlie: People have a different perspective on this, but there's just data flowing into us the whole time and at the moment we don't have that in the technology space. If you wanna solve the autonomous vehicle, you've gotta hold. They do like cameras and the visual aspect extremely well. But to solve true AGI level staff to go beyond doing a 2D chess player game to processing a mortgage, I think there's also gotta be a new way of ingesting data.

Charlie: Now, one interesting question that I've always wondered is, What will the first iteration of AGI look like? And there's no good reason in my mind to think, I don't think this is the end state. Cause I think the end state's a lot smarter than this. But the first version of what we would consider agi. And general intelligence just means it can do many diverse things and learn from one instance and apply that learning to another instance.

Charlie: It could just be a layer that looks like a chatbot, that looks like GPT-4 or GPT-10, whatever it ends up being that ducks into different specific narrow. Ai. And so if you want to get in a car somewhere, you talk to G P T four and you say, I'm looking to go here. And that just plugs into some autonomous vehicle algorithm.

Charlie: That could be the first way. And it'll feel like general intelligence and it will be general intelligence or you might have just some massive change in the way algorithms are written. And I do think there's a lot of excite exciting happening there. It's just not clear what the timeframe. , uh uh, for that well,

AI Agents Everywhere

AJ Asver: Yeah, I like that last bit you talked about. I, I really. That as kind of a way to think about how AI will evolve. I think some people of call it this kind of agent model where you have essentially this l l m large language model, like GPT acting as an agent, but it's actually coordinated across many different integrations, maybe other deep learning models to, to get things done for you.

AJ Asver: And so the collective intelligence of all those things put together is actually pretty powerful and can feel like, or, or have the, the, the kind of illusion of, of artificial general intelligence. I think for me there's this philosophical question of like if it's as good as the thing we want it to be, does it matter if like some nuanced academic definition of AGI isn't what it is? You know what I mean? Like if it does all the things we'd want of like a really smart assistant that's always available, but it doesn't meet the specifics of AGI I in the academic sense. Maybe it doesn't matter. Maybe that's what the next 20 or 30 years looks like for.

Charlie: Look, I think that's exactly right and there's no good reason for us to care. We just care that it gets done. We have no idea how the mind even works. We're pretty sure that the mind doesn't say, okay, I've got this little bit for playing chess, this little bit for driving some different way of doing it.

Charlie: But humans are very attached to replicating. Things that they experience and understand. And one just very simple way of doing it is a change of the definition of AGI from what your average person might associate AGI with.

AJ Asver: That's yeah, that to me is a, a is kind of a mental shift that I think will happen. And, and one of one of the things I've been thinking about is how, and, and this is why a huge reason why I started this, this newsletter and this podcast is that, you know, these things happen exponentially and very quickly.

AJ Asver: You don't really realize when you look behind you at an exponential curve cause it's flap, but you look forward, it's, it's kind of, steep. I always talk about this quote from. Sam Altman, cuz it really like seared into my head is that I think what we're gonna see is this exponential increase in these agents that essentially help you coordinate your life in, in work, in, in meetings.

AJ Asver: In getting to where you want to go in organizing a date night with your significant other. Right. And you are suddenly gonna be surrounded by them. And, and you'll forget about this concept of AGI because that will become the norm in, in the same way that like the internet age has become the norm. And being constantly connected to the internet is part of our, our, our normalcy in life.

AJ Asver: Right. This has been a fascinating conversation. I have one more question for you, which is, You know, as someone that's been going deep into the AI space as well, maybe from the enterprise side as as well, what are you most excited about in a, in AI for the next few years?

The future of society with AI

Charlie: yeah. Look, I talked about two things. Firstly. . I think one of the things that I'm excited about in the short term is just the growth in education. And the single biggest thing I think has happened with G P T is the just mass, fast, easy adoption. And when the internet became very, very interesting, it was when you got mass distribution.

Charlie: People creating use cases. And that's sort of when you went from like, okay, the internet could be a search engine to organized data. The internet could be a place to buy clothes. The internet could be a place to game to, okay, the internet is just everything. So I'm excited for that. And then in the long run it would be remissive us not to discuss this.

Charlie: I'm excited about thinking about what a new economic system looks like. People talk about universal basic. I don't think that the enterprise should be thinking about this question today. Our customers hyper science aren't thinking about this question today, but now is what I would call the ideation stage.

Charlie: Like we need think tanks, governments, people out there like, thinking of ideas, and then eventually stress testing ideas for what the future could look like. And I'm, I'm sort of excited for that.

AJ Asver: Yeah, I, I want to believe that it will happen, but I'm a little skeptical just given what the history of how humankind behaves. We're, we're not particularly good at planning for these inevitable, but hard to grasp, like eventualities in a similar way that we kind of all knew interest rates are going up, but it was really hard to understand what the implications of that was until last weekend when we found out, right, the

Charlie: It was very much we could have prepared for this.

AJ Asver: Yeah. Yeah. And yet we could have prepared because all the, all the writing was on the wall. And you know, you've got these people at the fringes that are kind of like ringing the alarm bells, whether that's like people working in AI ethics or whether it's even Sam Altman of OpenAI himself saying like, Hey, we actually can't open source our models anymore. It's too dangerous to do that. Right? And so then you've got people on the other side that are like, no, we need to accelerate this. It needs to be open. Everyone needs to see it. And the faster we get to this outcome, the faster we'll be able to deal with it. I, I skeptically unfortunately, believe that we're gonna stumble our way into this situation.

AJ Asver: We'll have to act very quickly when it happens. And maybe that's just like the inevitable kind of like journey of mankind as a whole, right? But it's still exciting either way.

Charlie: Well look, I think so. Look, if we don't stumble our way into it, we have, we create a world where people don't have to work, and I'm pretty excited about that. I'd going. Studying philosophy for two years at NYU, I'd be playing a hell of a lot of tennis. I'd be traveling more, like there is a good outcome.

Charlie: My, if we just stumble through it. My thinking, AJ, is that this is what the world looks like and it's not pretty. It looks like three classes of people. You have your asset owners, you can think of them as today's billionaire. They will probably end up controlling the world. There might be some fake democracy out there, when you have all of the sort of AI infrastructure owned by ultimately a small group of people you're probably gonna have them influencing most decisions.

Charlie: You may then have this second class of like celebrity class. I think that may still exist or human sports may still exist. Human movies human celebrities may still

AJ Asver: Yep.

Charlie: and you just get this class of 99.9% of people that are everyone else. And what the, the, the two features of their life are gonna look like.

Charlie: This is just my guess of like the way it goes. If we don't think about it in a bit more of a interesting way plan, and plan for it is universal basic income. Everyone gets the same amount, probably not very much. I don't know that that's gonna be particularly inspiring for people. I think there's better ideas and then I think that you this is very dispo dystopian, but end up living more in virtual reality than in reality. There's the shortest path to a lot of what you might want to create is just to create it in a virtual world versus going and creating all of that. But in a physical world if so, all that is to say is if we don't start thinking about it, don't start having some regions test different models having startups.

Charlie: Form ideas around this and, and, and come up with ideas that are then adopted by bigger countries. In this instance, I think you could end up with a bad outcome, but I think if it's planned, you could end up with an insanely cool world.

AJ Asver: Yeah. So we're gonna end on that note that, that those two worlds are what we have to either look forward to or dread, depending on which way you think about it. And I think for folks listening it's. . It's just like really important to begin with that people just understand, like society can only understand if individuals understand where this technology is going.

AJ Asver: Right? And that's where you obviously are helping by communicating on LinkedIn to the many people that follow you, especially in industry around it. I try to do it with, with this podcast, but I think for, for anyone that's like fascinated about AI that's following along, like I think the number one thing you can do right now, Share with your friends some of the things that are happening in this space to help people kind of get a grasp for how the space is moving, and that will also help you advocate for us to do the right thing, which is to prepare for it, right? I, I, I think like if people think this is a long way away and they don't understand it, just like you said for industry right? Then no one is incentivized within government to do it because their own uh, constituents don't care about it.

AJ Asver: Right? But if we're like, Hey, this is happening. It's an exciting technology, but also there's a kind of two ways this would go and there's one better way than I think it's just as important that we as individuals care about it and advocate for it. was a fascinating conversation. Thank you so much for the time, Charlie.

AJ Asver: I really appreciate it. I cannot wait for, for folks to listen to this episode and thank you a again for joining me on the Hitchhiker Guides ai.

Closing remarks

Charlie: Well, thank you for having me here, aj.

AJ Asver: Awesome. Well, thank you very much and for anyone that's listening, if you enjoyed this podcast please do share it with your friends, especially if you have either founders that are considering building AI startups and going into enterprise, or if you have folks that are working in industry and are considering incorporating AI into their products.

AJ Asver: I think Charlie shared a lot of really great insights there that I think folks would appreciate hearing. Thank you very much, and we'll see you on the next episode of the

​

Note: This episode is best experienced as a video!

Hey Hitchhikers!

MidJourney V5 was just released yesterday so it felt like the perfect opportunity to do a deep dive on prompting with a fellow AI newsletter . Linus creates amazing MidJourney creations every day ranging from retro rally cars to interior design photography that looks like it came straight out of a magazine. You wouldn’t believe that some of Linus’s images are made with AI when you see them.

But what I love most about Linus is his focus on educating and sharing his prompting techniques with his followers. In fact, if you follow Linus on Twitter you will see that every image he creates includes the prompt in the “Alt” text description!

In this episode, we cover how Linus shares how he went from designer to AI influencer, what generative AI means for the design industry, and we go through a few examples of prompting in MidJourney live. One thing we cover that is beneficial for anyone using MidJourney for creating character-driven stories is how to create consistent characters in every image.

Using the tips I learned from Linus, I was able to create some pretty cool Midjourney images of my own, including this series where I took 90s movies and turned them into Lego!

I also want to thank Linus for recommending my newsletter on his substack, which has helped me grow my subscribers to over a thousand now! Linus has an awesome AI newsletter that you can subscribe to here:

Inside My Head

Exploring the world of AI Technology, Design, and building digital products while navigating life as a dad.

By Linus Ekenstam

I hope you enjoy the episode and don’t forget to subscribe to this newsletter at http://HitchhikersGuideToAI.com.

Show Notes

Links:

- Watch on Youtube: https://bit.ly/3mWrE5e

- The Hitchhikers Guide to AI newsletter: http://hitchhikersguidetoai.com

- Linus's twitter: http://twitter.com/linusekenstam

- Linus's newsletter: http://linusekenstam.substack.com

- Bedtime stories: http://bedtimestory.ai

- MidJourney: http://midjourney.com

Episode Contents:

00:00 Intro

02:39 Linus's journey into AI

05:09 Generative AI and Designers

08:49 Prompting and the future of knowledge work

15:06 Midjourney prompting

16:20 Consistent Characters

28:36 Imagination to image generation

30:30 Bonzi Trees

31:32 Star Wars Lego Spaceships

37:57 Creating a scene in Lego

43:03 What Linus is most excited about in AI 46:10 Linus's Newsletter

Transcript

Intro

aj_asver: Hey everyone. And welcome to the Hitchhiker's guide to AI. I am so excited for you to join me on this episode, where we are going to do a deep dive on mid journey.

aj_asver: MidJourney V5, just launched. So it felt like the perfect time for me to jump in with my guests, Linus Ekenstam. And learn how to be a prompting pro.

aj_asver: Linus is a designer turned AI influencer. Not only does he have an AI newsletter called inside my mind, but he's also created a really cool website where you can generate bedtimestories for your kids. Complete with illustrations. And he is a mid journey prompting pro. I am constantly amazed by the photos and images that Linus has created using mid journey. It totally blows my mind.

aj_asver: From rally cars with retro vibes to bonsai trees that have candy growing on them. And most recently hyper-realistic photographs of interior design that looked like they came straight out of a magazine. Linus is someone I cannot wait to learn from. And he's also going to share his perspective on what all this generative AI means for the design industry, which he has been a part of for over a decade. By the way it's worth noting that a lot of the stuff we cover in this episode is very visual. So if you're listening to this. As an audio only podcast. You may want to click on the YouTube link in the show notes and jump straight to the video when you have time.

aj_asver: So if you're excited about I'm one to learn how you can take the ideas in your head and turn them into awesome images. Then join me for this episode of the Hitchhiker's guide to AI.

aj_asver: Thank you so much for joining me on the Hitch Hiker's Guide to ai. Really glad to have you on the podcast. I feel like I'm gonna learn so much in this episode.

Linus Ekenstam: Yeah. Thank you for having me.

Linus Ekenstam: I mean, I'm not sure about the prompt, you know, prompt guru, but let's try

aj_asver: Well, I mean, you tweet about your prompts every day.

aj_asver: on Twitter, and they seem to be getting better every time. So You are my source of truth when it comes to becoming a great prompter. And I also, by the way, love the one thing you do when you tweet your mid journey kind of pictures that you built, um, that you've created, that you always add in the alt text on Twitter. Um, exactly what the prompt was. And I found that really helpful. Cause when I'm trying to work out how to use Mid Journey, I look at a lot of your alt texts. So, um, also include a link to your Twitter handle so everyone

Linus Ekenstam: Nice

aj_asver: it out. But I guess

Linus Ekenstam: I guess I'll stop.

aj_asver: you know, you've been in the tech industry for a while as both a designer and a founder as well

Linus Ekenstam: Yeah. Yep.

aj_asver: love to hear your story on what made you, um, kind of get excited about AI and starting an AI newsletter and then, you know, sharing everything you've been learning as, as you go.

Linus's journey into AI

Linus Ekenstam: Yeah. I mean, if we rewind a bit and, and we start from the beginning, um, I got into the tech industry a little bit on a banana, like a bananas ski. I, I started working in, like, the agency world when I was 17. I'm 36 now, so 19 years ago, time flies. Um, and after like working with, um, customers, clients, and big ones as well, through like, through my initial years there, I kind of got fed up with it.

Linus Ekenstam: And. . I went into my first SaaS business as an employee and it was email like way, way, way, way, way before this day and age, right, where you had to like code everything using tables and transparent GIFs. It was just a different world.

Linus Ekenstam: And 2012 was like, that's when I started my first own business. And that was like my first foray into like the, the startup world or like building something that was used by people outside of the vicinity of, of, of Sweden or Nordics. Um, it was very interesting times. Um, and I, I've always been kind of like early when it comes to New tech, I consider myself being a super early adopter. I got Facebook as like one of the first people in. By hacking or like social hacking a friend's edu email address. And I got an MIT email address just so I could sign up on Facebook.

Linus Ekenstam: Um, so now that we are here, it's like I've been touching all of these steps, like all the early tech, every single time, but I never really capitalized on it or I, I never really pushed myself into a position. I would contribute, but this time around I just, you know, I felt like I had a bit more under my belt.

Linus Ekenstam: I've seen these cycles come and go, uh, and I just get really excited about like, oh shit. Like this is the first time ever that I might get automated out by a machine. So my response or flight and fight response to this was just like, learn as much as possible as quickly as possible, and share as much of my learnings as possible to help others.

Linus Ekenstam: Cannot not end up. In the same position where they fear for their lives.

aj_asver: Yeah, it's, it's interesting you talk about that because I think that's a huge motivator for me as well. It's just help people understand that this AI technology is coming and it's not like it's gonna replace everyone's job, but it certainly is gonna change the way we work. And make the way we work very different. And as -you've been doing and sharing, you know, how to prompt and what it means to use ai, one of the things I've noticed is you've also received a little bit of backlash, you know, from other designers in the space

Generative AI and Designers

aj_asver: That maybe as embracing of AI as you have. And I, I know recently there were probably two or three different startups that announced text to UX products where you can basically type in the kind of, uh, user experience you want and it generates, mockups right which I thought was amazing and I thought, You know, that would take years to get to, but we've got that now.

Linus Ekenstam: yeah, you

Linus Ekenstam: post.

aj_asver: and I think one of the things you said was designers need to have less damn ego and lose the God complex.

aj_asver: Tell

aj_asver: me a little,

aj_asver: what the feedback has been like in the AI space around kind of how it's impacting design, especially your field.

Linus Ekenstam: So I think, um, there, there is this like weird thing going on where. They're a lot of nice tooling coming out and engineers and, and, and developers. You kind of embrace it. They just like have a really open mindset and go, yeah, if this can help me, you know, I'll, I'll, I'll use it.

Linus Ekenstam: Like, take Github Copilot is a good example. People are just raving about it and, and there is some people that are like, oh, it's, it's not good enough yet, or whatever. But like the general consensus is that this is a great tool, it's saving me a lot of time and I can focus on like more heavy lifting or thinking about deeper problems.

Linus Ekenstam: But then enter the designer , like turtleneck, you know, black, all dressed in black. I mean, I'm, I, I'm one of those, right? So I'm, I'm, I'm making fun of myself as well. I'm not just pointing fingers at others here. I just think it's like weird that. Here's a tool that comes along and it's a tool, it won't replace you.

Linus Ekenstam: Like I'm being slightly sarcastic and using like marketing hooks to get people really drawn in, in my content on Twitter. So I'm not really, meaning, it's not literal. I'm not saying, Hey, you're gonna be out of a job. It's more like, You better embrace this because like the change is happening and the longer you stay on the sidelines, the, the, the more of a, a leap that your peers will have that are starting to embrace this technology.

Linus Ekenstam: And I, it's so weird to see like people being so anti and it's like, it's just a tool. It's not, it's not like the tool itself is dangerous. It's like people with the tool will become dangerous and they will threaten your position. Right. So I just find it very interesting to this whole kind of landscape where people, on one hand, it's just embracing it and people on the other hand are just like, no, I'm not, I'm not gonna touch it cuz he can't do X or he can't do Y.

Linus Ekenstam: It's like, bear with you. It's like we're in the very, very early days of ai. , we might be seeing half a percent or 1% of what's possible. And these tools are here today, like you said. Um, so I think my kind of like vantage point is like I'm not looking at the next six months or the next 12 months. I'm just like drawing out an arc and going like, where are we 20, 30?

Linus Ekenstam: My whole game here is to get as many people as possible.

Linus Ekenstam: Ve well versed into these tools as fast as possible. Like, I want to make sure that the divide between the people that haven't got experience and the people that haven't yet played with these tools kind of make sure that divide doesn't grow too big. I think that's my mission really.

aj_asver: Yeah, You pointed out there there about how, you know these advancements are happening really quickly and you want people to be able to adopt the tools is I think a really important one. And I think a lot of people don't really understand conceptually how exponential advancements kind of work. And I think Sam Altman recently had this good quote where he said something along along the lines of, when you're standing on an exponential qu curve, it's flat behind you, but it's like vertical in front of you, right?

aj_asver: And we're like like climbing this exponential curve, and I think some of us probably see the writing on the wall of how quickly this is all gonna happen. But for other people, know there's always gonna be this resistance. You mentioned like how this tool is gonna help people and people should embrace it, and you of course share a lot of what you are learning and especially when it comes to prompting and you and the art of kind of prompting in mid journey to create interesting images.

aj_asver: And I think you've done some prompting on ChatGPT as well

Linus Ekenstam: Yeah,

Prompting and the future of knowledge work

aj_asver: One thing I'm curious about is, do you think that's the future of the, like knowledge work for us? Is it gonna be like we all just become really good prompt engineers and we're just prompting away when it comes to like writing, you know, writing documents or when it comes to creating design in ux or when it comes to, you know, making images or, or do you think there's more to it than that?

Linus Ekenstam: I think prompting the way that it's done now is gonna be very short-lived. Um, if we're doing an analogy and compare, like prompting with ai, with what we're people that are doing really root level programming, let's say assembly type code for computers.

Linus Ekenstam: Um, so we're in the age now where everything is new, uh, and the way to interact with these models, whether. ChatGPT or Mid Journey or Dolly or Stable Diffusion. It's a very, very root level. Uh, I think I'm, I'm already starting to see like products popping up that are precursors or like tools that put, put themselves like a layer on top.

Linus Ekenstam: So instead of writing like a 200 keyword, Um, prompt to mid journey. You're essentially writing like five words or 10 words that's very descriptive of what you want. And then the precursor takes care of like generating the necessary keywords for you.

Linus Ekenstam: I don't think we'll see these like prompt tags where people figure out me included, figuring out like ways to, you know, if you do this in this sequence or this order, you will able to do. This with a, with a, you know, with a model. Right. Um, I think we'll see less of that potentially, um, and, and move more towards like really natural language and, and less kind of like prompt engineering around it.

Linus Ekenstam: But I think very, um, important here to note is that the kind of like precursors will happen, like we will kind of move away from talking assembly line situation with AI models and that's also like, that's a barrier to entry right now. If you look at mid journey, you want to get. A lot of the things you need to just overcome is kind of how do you write what you want to get?

Linus Ekenstam: Because like if you just write, I want this image that does X, Y, and Z, you're probably not gonna get the thing that you have in your mind. So it's gonna be like you trying out different things and then getting to like slowly speak ai.

Linus Ekenstam: Don't focus too much on becoming like a, a super good prompter, right? To try to, to learn more like principles or techniques or, or, or like think more, um, holistically about this whole thing. How do you interact with ai? I think that's, uh, yeah, something I would recommend

aj_asver: I liked your analogy of, you know, assembly language and how we're all kind of writing assembly code right now. Um, another analogy I've also heard is like, it's, it's like DOS before Windows came and we're all kind of at the command line trying to get what we want by typing it into a computer before someone, you know, made like a user interface around it.

aj_asver: You know, very famously, you know, Xerox Park did it and then Apple was the one who kind of released the version of it. And I, I think that's gonna be something. We'll, all welcome. But in the meantime, I'm curious, how did you kind of learn that steep learning curve of becoming really great at prompting?

aj_asver: Because we all kind of start from zero here, and I think the example you gave where you said like, you know, you think you can just come in and describe what you want. That's exactly how I started. I was

Linus Ekenstam: I was just like

aj_asver: I'm just gonna the the scene and then it didn't end up anything. like wanted

aj_asver: What was your journey of becoming good at prompting and, and learning how to use the journey effectively?

Linus Ekenstam: I think the, the community's really well set up, uh, with my journey that you can. You can go onto midjourney.com and you can start kind of exploring what everyone else is doing. Um, initially I didn't really understand that they actually had a website. Um, but, but then after a few, I guess a few months, then I'm like, oh, there's actually a website here.

Linus Ekenstam: And maybe it wasn't there in the beginning either, so it might be that I just missed it completely. So I think like dissecting other people's work and trying to figure out like, oh, this was a nice cinematic shot. How did I get that look? Um, and, and mind you, like, I've been doing this now for a few months, or yeah, almost half a year.

Linus Ekenstam: Um, and it, it was really different. There was less people using the tool like six months ago than there is now. So I think. It's easier to jump into the tool now and see what others are doing and kind of learning by do, like learning by dissecting essentially. I think it's the same with design. Like if you're learning design today, like the best way is just to try to replicate as much work from everyone else as possible. And I'm not saying replicate in the sense of like, oh, that's a nice prompt, command C, command V, oh, now I did it. Um, that's k that's, that's not learning to prompt . All that it's very easy if you, if you find something you like, you wanna make your own derivative of it, go ahead.

Linus Ekenstam: I mean, that's the beauty of these tools as well. But if you really want to like learn the skill or like, you know, I have this idea I want to do, then I can go do this. But then I must say there's also some really talent. People, uh, on Twitter and elsewhere that are sharing their journeys as well, and, you know, figuring out ways to, to structure their prompts or, yeah, th there there's a bunch of people that we could potentially look at later or, or that we could recommend in the show notes.

Linus Ekenstam: Um, for sure.

aj_asver: Yeah, that would be, that would be great. And I think for people that aren't familiar, the way Mid Journey works is you, you have the website where you kind of explore existing images, but all of the work of creating Images is done through their Discord, which

aj_asver: Unintuitive to anyone that's not familiar with online communities and with Discord, which Discord itself is a fairly new phenomenon from the last like three or four years right? It was originally used in gaming, but now it's used a lot in communities across ai, across crypto and other and other places. And so was another thing that got, that took a while to get used to is like interacting with AI via Discord. But there's one cool advantage of it that I didn't really fully grasp until now, which is that I can pick up my phone and jump in the Discord any time when I have an idea for an image and just start making images.

aj_asver: One of the things I'm curious about, you've been doing this for about six months, ballpark, how many images have you created

Linus Ekenstam: I think I just passed, like the 10 K club. I'm not sure. I'm gonna have to look later.

aj_asver: I mean, one thing I would love to do in this episode, um, is learn from you some of the skills of like, you know, being a pro prompter in, in, uh, MidJourney. So I was wondering if we could kind of jump into it and maybe one, take a look at some of the creations you've done in the past. Walk us through a little bit, um, how you, how you came up with them and then I have a few ideas of things I wanna do in Midge. Maybe you can help me, um, make that happen.

Linus Ekenstam: Let's jump into it.

Midjourney prompting

aj_asver: Awesome. So we're gonna jump into Mid Journey now, and Linus is gonna show us some of the images he's created, give us a bit of a sense of kind of his approach to prompting and then we're gonna jump in and do a few examples too

aj_asver: So one area, for example, that I would love to learn more about is consistent characters in Mid journey. After we did the podcast, uh, with Ammar, where we built the, where we created the, um, children's book, a lot of people asked, oh, how do you get consistent characters across all the pages of the children's book in the illustrations? So I'm really curious about how you achieve that, cuz that's something I've struggled with as well.

Linus Ekenstam: This is interesting. So, um, it, it started with, A lot of people trying to achieve the same thing using mid journey, which is essentially, you know, you have a character, you want the character to be in different poses or in different photos or in different, you know, could be a car cartoon, it could be a real person.

Linus Ekenstam: Uh, and I saw different ways of doing it, and mainly they were for cartoons. Then I'm like, this doesn't not work well with a human. Uh, cuz I tried and it didn't work. So I'm like, there must be some other way to do this. Uh, and obviously this is like brute forcing a password really cuz like mid journey is not supposed to be this tool.

Linus Ekenstam: Uh, the best way to do consistent characters is to use stable diffusion or something else that you can pre-train on a set of images.

Consistent Characters

Linus Ekenstam: the, the way that I went about doing it is essentially, , uh, going to how illustrators work and when, when they create like a character for, for a movie or for an animated, whatever it might be that they're doing, they need reference materials so that other artists can work on the same characters.

Linus Ekenstam: You might have hundreds of artists working on the same character. Um, so then, you know, looking at how they are doing these, I'm like, maybe if I simplify this, what if, you know, I take left and right and up and down, and. and I used those images as inspiration cuz that's something you can do in my journey.

Linus Ekenstam: You can like image prompt, essentially just like putting an array of images and then adding your prompt. So I, I, I went about like starting up making a character, um, just using like a very simple prompt here. I didn't really have any intent of, of, of the output. I just like, let's make journey, do its thing.

Linus Ekenstam: Um, and then when I, I, I found one that I kind of like, oh, this could be nice. Let's work with this. Um, I started using something called a seed. and use that image. So a seat is essentially the noise number or the random noise that an image gets started from. So if. For anyone that doesn't know, you know, mid journey is a diffusion model, which essentially starts from noise and it takes a string of text and it uses that text to take the noise and transform it into an end result.

Linus Ekenstam: So if you want to know the pattern, the exact pattern of the noise that you're starting from, you can include a seed number and it's like randomly generated every time you do an image. So if you have an image and you use the seed and you prompt against that seed again, uh, the likelihood of getting something very similar is quite.

Linus Ekenstam: So I, I kind of went away and, and started doing different angles of this woman. And then once I had more angles, um, I put the angles together. So I'm just scrolling through here, but essentially just finding those up, down, left, right, and forward. And when I was happy with all of them, I just put them together in a long prompt.

Linus Ekenstam: Um, and then just having the same prompt again as the first time, you know, uh, a style, a, a, a portrait shot of a woman. Um, street photo of a woman shot on Kodak, which is essentially just like the, the type of film I wanted to emulate. And then I get the e exact woman out and, hi, this is like, you know, okay, now here we go.

Linus Ekenstam: Uh, what can we do with this? Right? Uh, and there is a bunch of things, like a bunch of learnings, um, from this, which is essentially like you can. Very specific images, if you have a bunch of, of images that you're using as the inspiration images,

Linus Ekenstam: but also when you do these technique. My kind of the, the culprit here is that I use street style photos.

Linus Ekenstam: So every time I'm trying to get her to do other things, like we can go over here. Uh, I, I wanted to try to get her in a, in a space suit, right? We can see that she's kind of in a space. , but she's still standing on a street. So

aj_asver: It's like a very, like a urban chic space suit,

Linus Ekenstam: yes, . It's an urban cheek spacesuit. And we can even see here, like try some different, um, ar like some different aspect ratios. She's in a spacesuit, but we still have the background of, of the street, right? So, . One way to combat this and that, you know, figure this out after the fact that I made this tutorial is like the, the, the source material, the source MAs that you're using, they should be isolated.

Linus Ekenstam: They should be like either against a transparent background or a white background. And that way all of a sudden you can start placing this woman in different areas. So what's neat about this, that you don't need to train a model. You only need to have a set of image. So let's say you have six or nine images that are your inspiration images, and they don't have to be AI generated either.

Linus Ekenstam: You could use like yourself, you can take photos of you from the different angles and put them together. Um, and I think a lot of people, it resonated with a lot of people because this is one of these things that are inherently hard to do in my journey, and there is quite a big use case for it. So I, I, I personally hope that, you know, my journey goes into the direction of kind of a little bit.

Linus Ekenstam: Stable effusion or Leonardo, where they're giving you tools to do these kind of things like fine tuning, but not maybe to the extent of like training your entire, your own model completely. Right. And we can look at this example. I think this is very nice as well. Like we can get her smiling cuz that was one of the things that, you know, people, oh, you used all these photos, which she's not smiling, you're never gonna get her to smile.

Linus Ekenstam: Uh, and basically you can like, there's a lot of like things you can do, even though mid journey is very. Has very opinionated. So there are ways to work around this. And if we're like diving a little bit into prompting here, um, we can just, let's, let's grab this full command here. Um, and we can,

Linus Ekenstam: yeah. Sorry,

Linus Ekenstam: Yeah.

aj_asver: Was you basically reverse engineered how, you know, a character animator would approach this idea of consistent characters. And the way they do it is they have different poses of a character that they kind of create first. So you kind of have a base, kind of almost like a sculpture that hasn't been fully molded yet. So to get an understanding of the character, you generated those using ai, but you could also have a. you know, photos you already have of a person or maybe you take photos of yourself at different angles. Then you inputted that as actually with the prompt, you inputted the images too, that was what allowed you to kind of then create these consistent characters cuz you now have this base image to work from. one of the things you said was, if you want to be able to change the background, so move them from like street photos for example, to be in space, you kind of need to remove the background from the original base images because that's that background. If you keep it in there, like the street photos has a street background is gonna influence what mid journey creates as well.

Linus Ekenstam: Yeah, correct. Um, let's , I just pushed this in here. Let's see if my journey does something with it. Sometimes when you're using a lot of high resolution, um, inspiration images, it actually crashes the, the bot. So it doesn't work. Oh, we're actually getting something that's good. So, uh, it's not entirely sure we're gonna get a smiling woman this time, but the way to kind of like force smiling for example, is give smiling a very high weight.

Linus Ekenstam: So when you're using, um, Let's see if I can scroll in here. Yeah. So when you are adding, uh, is it colon? Yeah. Is it colon, semicolon? Um, colon? Yeah, colon. Colon five, for example. Then you give smiling. The word smiling, uh, a weight of five. Uh, I think standard weight is like zero or one, I think one. Um, so we're really emphasizing here that we want her to be smiling and now I think we actually got something.

Linus Ekenstam: And it might not be that she's smiling in all of. , but she's, uh, kind of smiling . Forced,

aj_asver: got like a little bit

Linus Ekenstam: yeah. a bit of a forced smile. Uh, but

aj_asver: smile. Yeah.

Linus Ekenstam: yeah, and this is the thing. I mean, mid journey is opinionated and you, you might have to do re-roll, you might have to do things like over and over. And because it's not really trained on her smiling or being neutral is actually trained on her being.

Linus Ekenstam: Angry or just very like,

Linus Ekenstam: uh, so re-roll is like, essentially press a button here in, in, in this cord and it takes the exact same, uh, prompt, the same parameters, and a different seed so it won't use the same noise again. So it will start from the beginning One more time. Uh, if you wanted to get like more diverse outputs, we could use chaos, which is essentially how chaotic the difference is between the four image.

Linus Ekenstam: that we're going for. So we could add, uh, dash C and then let's say a hundred. So this value goes between zero and a hundred, and it will d dictate the difference between the four different images. So we can see up in, yeah.

aj_asver: I noticed, um, by the way, that there's a few different kind of arguments you can add to the end of your. Mid journey prompt, and I think one that you use often is aspect ratio. Um, and then chaos is one. You just mentioned hph and hph, and C. Where did you learn about these and how does someone kind of work their way around trying all these different ones?

Linus Ekenstam: um, i, I mid journey.com, they have like documentation I think people are a bit afraid of, of the documentation because they might not know what they're looking for or like, um, yeah, it, it's not that hard. Like when, when you're prompting in mid Journey and then you go like, okay, there is like, I think, uh, 6, 7, 8, 9.

Linus Ekenstam: Nine different arguments that you can use. So it's like aspect ratio, chaos. Quality seed stylized tile, which not many people know, and version and quality. So version, you don't need it if you're not. Like now it comes preloaded with the latest model. So if you just add dash dash V four four, it actually uses an older model.

Linus Ekenstam: Um, so a lot of prompts you'll see will have dash, dash, v4, uh, not necessary. So essentially now the model that's running is V4 C, which is like the third iteration of v4, uh, and quality two. You can go quality one, two, and up to five, I think. But they've done a lot of testing internally. people can't tell the difference between Q1 and q2.

Linus Ekenstam: Like, so it's just a waste of GQ 10 because essentially when you're doing quality two, you're gonna use twice as much GPU render time. And GPU render time is essentially how long, um, of like, how much of your credits get used to render an image? Um,

aj_asver: it. So high quality means if you're paying for mid

Linus Ekenstam: yeah.

aj_asver: actually to use the bot directly versus the

Linus Ekenstam: Yes. Yeah.

aj_asver: it's gonna cost more per image basically.

Linus Ekenstam: Yeah.

aj_asver: I notice as well is that you also include some details around how the shot is taken, right? The actual camera. Um, is, does that make a lot of difference kind of picking the, the camera? Because I noticed that was a pretty cool thing that I didn't, I wasn't aware of actually until I saw your, your images.

Linus Ekenstam: Yeah, I think we're, there's a bunch of people that I've, like, I, I saw this like in December, the first time. I think like people using camera. Like it's shot on a canon or it's shot on a hassle blood, or it's shot on an icon, or it's shot on this type of film, you know, emulating black and white film or emulating sepia tone film.

Linus Ekenstam: Um, and then now I think it's, there's a lot more people that are kind of dissecting it and like really going nitty gritty on it. Um, and, and trying to just be like, what are the things that we can do with this? Like, how. , how much can we describe with this? And it's, as it turns out quite a lot, especially like camera angles type of shots.

Linus Ekenstam: Like, you know, using wide, ultra wide narrow, you can go and use like lens parameters. So like for those that are interested in photography, um, you, you could use 50 millimeters, so 50 mm. Um, to, to decide kind of the, the, the, the framing of your shot and kind of what. Output should look like because it has a very distinct look.

Linus Ekenstam: You could go 80 millimeter, 120 millimeter tele lens. All these things matter. Actually, it matters quite a lot cuz if we go here and, and check some of the photos I did the other day about, so I did some, uh, national Euro graphic shots, right? So these are quite interesting where we have like, um, shot on the telephoto lens as one of the key things.

Linus Ekenstam: So what it does, it really gives you this super consumed in

Linus Ekenstam: photo with like bulky in the background. So you have a really blurred background and we can see that, like my journey is really good with hair. Um, and the compression might blow it, blow it down a bit, but Maur is fantastic with hair and feathers and fibers.

Linus Ekenstam: I'm not sure what they've done there, but it's, it's absolutely fantastic. Um, so yeah, and lens matters quite a lot.

aj_asver: you are, what you're doing there is really kind of imagining the camera you would take this photo with if it was a real photo, and using those, um, those properties of the camera as parts of the prompt. And one of the things I also noticed with your prompts is you are not necessarily describing the scene.

aj_asver: You are often described lots of, characteristics of the scene, right. What is your approach when it comes to, you have this idea in your head, uh, you, you, you kind of, ima have this idea in your imagination of what you wanna create and then getting that down into a prompt. How do you, how do you approach that?

Imagination to image generation

Linus Ekenstam: um, I mean, in the beginning I, I did write a really interesting one. Threads on this as well. Cause I was sitting in a restaurant you mentioned earlier that like, that, you know, it runs in Discord. You can bring up your phone, you can start prompting. I was, um, we're, I got two kids, right? And me and my partner, we actually had like the first weekend without kids, um, since pre pandemic.

Linus Ekenstam: So we basically haven't been out alone. And, and you, what, what I do, I sit with my phone in mid journey. That came out bad anyhow, we were sitting there and we're actually using it together. So we were like, we're talking, we're talking about like what we're building with bedtimestory and then we saw this like really nice geisha on the world cuz we were eating at an Asian fusion restaurant.

Linus Ekenstam: And I'm like, I wonder if I can do that mid journey. And then we're just like, you know, open up discord on the phone and we're sitting there chatting, drinking a little bit and just like, oh, okay, we, you know, let's try this. I think I ended up doing like 50, maybe 50 or 60 generations where like the initial.

Linus Ekenstam: Was a geisha, but it didn't look anything like the thing we saw on the wall, right? Because the, the geisha on the wall was like on a wooden plaque,

Linus Ekenstam: uh, just like a really nice white geisha face mask and some red, really tiny red, uh, pieces in it. So we basically just went like, iterated removed, you know, added, redacted.

Linus Ekenstam: It's just like added words, removing words, try different things, went completely crazy and go, what if we just take away all of this and write something completely different? Um, so it it, it's easy if you have an idea, right? That to just like continue to, to plow through. And then once you hit what you want, then you have that kind of like base prompt.

Linus Ekenstam: Then you can start altering that you. Exchange an a subject or an object or exchange a post or, but, but you have kind of your, your, your base prompt figured out.

Linus Ekenstam: So getting to the base prompt could be tricky. Sometimes you hit gold after just a few tries. Um, it really depends, uh, on what it is that you're looking to create.

Bonzi Trees

Linus Ekenstam: Right?

Linus Ekenstam: I had luck with like bon's eyes, for example. I just, what can I do bons eyes with, with ma journey and how does that work? You know? And I just type like, uh, pine tree bonsai. Why wait a minute. Like this is fabulous. You know, I, I, I think I made some bonis again yesterday just for fun. Um, so like raspberry bonai, that's, that's, this is the prompt.

Linus Ekenstam: This is it, you know, it's not harder than that. And like, you could do

Linus Ekenstam: raspberry. That's it. , right? And, and, and you can, you can, you can, you can imagine, you can do thousands of thousands of these, right? Uh, and you can be crazy about it. You can do, like, I think I did Candy Bon. Yeah. Here we go. Ken Bonk. Who, who knew

aj_asver: it's got like lollipops.

Linus Ekenstam: Yeah.

aj_asver: that bonai tree is something my kids would absolutely adore. So I, I had this idea, um, Linus, um, I would love to learn kind of how to do this. Um, and I have a concept in my head and I was wondering if we could try it out,

Linus Ekenstam: Yeah. Let's, let's,

aj_asver: if we, we can kind of bring it to life.

Linus Ekenstam: yeah, let's try

Star Wars Lego Spaceships

aj_asver: So the concept is I also have two kids and they're four and they are absolutely obsessed with Star Wars right

aj_asver: got their first two Star Wars Lego sets and now they want like everything in the collection.

aj_asver: And they went to the library recently and got a book, and the book just has all these Star Wars, um, Lego ships in it. And it made me think like that would be a cool, fun thing to do in Mid Journey is like create imaginary Star Wars kind of spaceships. And so I was just wondering how would I approach that?

aj_asver: Um, do I just type in Star Wars spaceships made out of lego. Do I need to kind of think about how it's shot? Do I need to think about kind of the features? And so that's my idea. Star Wars Lego spaceships. How do we turn that into a cool, mid journey image?

Linus Ekenstam: Okay, let, let's just start straight off with, with what you just said, like Lego , Lego Star Wars spaceship. We we're probably just gonna get something that's very similar to what what's already in, um, in Star Wars, but with some kind of reim to Lego. Mid journey is relatively good at like, creating Lego. So we're gonna have to figure that one out. Star Wars. Um, spaceship. actually we're gonna put,

aj_asver: It's already starting to generate some images and, and the cool thing about mid gen is it kind of shows you bit by bit as it's evolving, right?

aj_asver: They already look pretty, pretty cool from the, from the outset. Okay. So we got some Star Wars Lego images.

Linus Ekenstam: it doesn't re does it look Star Wars, though?

aj_asver: It, it kind of looks like, um, yeah, it, it doesn't look like a Star Wars ship, I would imagine existing in the Star Wars world, but it has some kind of Lego vibes about it.

Linus Ekenstam: Let's try to see if we can get an imperial. Maybe in a pure cruiser or something that's also a known set. Maybe there is something that we could go crazy about instead. So when we started with Spaceship and we wanted to be Nubian fighter, um, maybe we want it to be like silver with, um, loose stripes.

Linus Ekenstam: Want the side shots. See if we can get something there.

Linus Ekenstam: So

aj_asver: you just typed

Linus Ekenstam: So,

aj_asver: in Star Wars spaceship and then Nubian fighters. You're trying to be a bit more specific about it. And then you also added some color hints as well, right? Silver with with white stripes and then Lego. That was an important part

Linus Ekenstam: Yeah, and I've also added side shot here to make sure that we get the, the, the, the model from the side. I'm not sure we will actually get it, uh, the way we want here. And I'm also not sure a Nubian fighter is, well this, this was the first one, like the imperial, some imperial ship here still.

aj_asver: to look a bit more, a bit more like a

Linus Ekenstam: Yeah,

aj_asver: I think.

Linus Ekenstam: it looks like Star Wars Lego, but it's still, I don't know, mid Journey is doing some weird things here with like, I think it's trying, oh, okay. Now, now let's,

aj_asver: ones, when you described a specific type of, um, ship, it looks a bit closer. So, I mean, another one we could try is like, you know, a tie fighter or an or an X-wing. Oh,

Linus Ekenstam: yeah.

aj_asver: looks a lot more like Lego right now. So

Linus Ekenstam: Yeah.

aj_asver: see something come to shape. It looks a lot more like a Lego we might have.

aj_asver: So maybe we could try like X X-wing fighter or tie fighter.

Linus Ekenstam: Let's try with X-wing. X-wing, and we want it silver with orange stripes. Maybe we want it like in space. See, I think the, the thing thing that I, the, the, what I'm kind of doing when I'm promming is like, I'm really kind of playful with it. I don't really mind if it takes me a hundred shots to get something.

Linus Ekenstam: Uh, sometimes it's a bit frustrating because like you think you, you kind of get down into this rabbit hole and you just like, I'm, I'm doing the right thing, you know, I'm writing these things, why it's not giving me what I want. Um, but then either just like remove completely and you start over and you do something different.

Linus Ekenstam: you just like try a different image and then come back to it because like maybe you have some other things that you want to try out. Um, but I think this, this might turn out great actually. Then, then again, the, the X the X-wing is a known object, right? So I would be surprised if, if we wouldn't get anything here.

Linus Ekenstam: I think where mid journey might China is like trying to combine, um, things. But if we want something that looks outta Star Wars, um, especially in Lego, there we go. This doesn't look.

aj_asver: This looks like it could be a real Lego set now. So we've got like a couple of different riffs on X-Wing. They have like really big engines, which is, which is really cool, and lots of lasers, which the kids absolutely

Linus Ekenstam: Yeah.

aj_asver: And you just clicked U2 there. Now what that does is that upscales, the second image, right when you hit U2 and

Linus Ekenstam: Yeah, so I, I just wanted to see kind of like what we would get if we, if we kind of like tried to get this in a slightly larger, uh, resolution. So these images are relatively low rest, I think when you're doing 69. Let, I'm doing now aspect ratio 69. We're gonna see, like, the image is 600, um, 640 pixels white.

Linus Ekenstam: so, And what I did here now is like when, when you have a shot, uh, a co, a collection of images that you get back, if you react with the envelope emoji, you get back the, the job id, um, the seed number and the four individual images.

Linus Ekenstam: So singular images, if you. If you want to save them low rest, and if you're doing like image prompting where, where you kind of put images into the prompt as well to give some, like, to give the prompt some inspiration. This is a neat way to like make sure you're using low risk images instead of like pushing the highest definition images into, into the image prompt.

Linus Ekenstam: Cuz that could slow down my journey quite a lot.

Linus Ekenstam: Um,

aj_asver: that gave you four individual images. Instead of

Linus Ekenstam: yeah.

aj_asver: one image, which

Linus Ekenstam: Yeah.

aj_asver: it's kind of four, the default is one image of four things in a grid. Right. And instead it generated four different images. And to do that, you clicked on the envelope emoji. So that is like a super, um, that's like a really great Easter egg for people to know.

aj_asver: Cause I would not have known that if, if you hadn't told me. So, envelope emoji. Gives you the original images, um, in low res, which you can then use to feed back into mid journey.

Linus Ekenstam: And, and again, this, yeah, this is impressive cuz if we consider what's happening here, it's like the, the model is interpreting our Lego as like actually building something in Lego. It it, it kind of tries to do that and emulate that. And if we look at the lighting here, the reflection in the cockpits, we can see that it's like shot with some kind of studio lighting over overhead lighting.

Linus Ekenstam: Um, ah, look at this. This turned out great. Wow. I'm, I'm surprised myself, so

Creating a scene in Lego

aj_asver: is so cool. And, um, I have one more challenge for you. This one's a little bit harder. We'll see if we can make it happen. So I

aj_asver: I was lego.

aj_asver: okay, this is cool, but it's even more fun if you can create a scene, right. With a few different characters in it and the Lego, uh, the, and the Lego, um, figurines too.

aj_asver: So I was wondering if we could give that a try. I have this, um, I have this fun idea for a Twitter thread where you recreate scenes from famous TV shows in movies in Lego in Mid journey. So maybe we can start with like a Star Wars one or we can, we, we can do a different one. Um, but I think that could be an interesting one to try too, because one of the things that I think is a little bit harder is when you have characters or multiple characters and you're trying to get, get, get a scene going.

aj_asver: So how would you approach that?

Linus Ekenstam: Actually I, I think this is an interesting one cuz like, okay, let's do, we have a scene in mind from Star Wars that we would like to try to reenact? Um.

Linus Ekenstam: So what I would, I would, what I would then do is like, I would pro, I probably do something like this where I would go and look for, For source material, like what is it that I'm trying to create, like to get an idea for how it looks, right? So, uh, I think this one is relatively cool. Um, I'm not sure we could actually do this, but this could be an interesting experiment.

Linus Ekenstam: So let's copy this image address and let's try to use this as an image prompt. So we go imagine, and then we paste the image url and then we say,

aj_asver: so you're pasting URL of a Star Wars scene that you found on Google Image search. Right.

Linus Ekenstam: Yeah, so I don't know her name here, but this is Finn, right? And what's her name? This is from the later Star Wars. So we go Finn

aj_asver: should know this cause we talk about Star Wars characters all the time.

Linus Ekenstam: star Wars Finn Running, and then it's BBB eight rolling, uh, sand Dunes and uh, Lego. We want to do aspect ratio 69. So we have the image and what I'm doing now, I'm basically just describing the image that we're looking at, um, and. Adding that as the prompt and adding Lego. We could actually just make sure we weight this, so we go Lego important.

Linus Ekenstam: Uh, and let's see. So here is a bit of like, if you listen in on the mid journey, um, all hands or in the way they call 'em community chat or, or town halls. Um, you'll hear, uh, the founder speaking a lot and the team speaking a lot about how the next evolution of of prompt. Probably gonna be image to image or like, people will do a lot of stuff using an image, um, or images, multiple images.

Linus Ekenstam: And I kind of agree because like, um, if I have a, if I have the ability to kite, either just take a, a snapshot of something or I grab something on the internet and I can like, take that and mesh it with something else and then put my prompt on it, the, the likelihood of me getting what I want is like 10 times higher

Linus Ekenstam: than if I'm just like, uh, writing.

Linus Ekenstam: Prompts

Linus Ekenstam: So this did not turn into Lego at all. So let's skip the idea of using the image prompt. And let's try something else. So Lego Star Wars is known, right? There is,

aj_asver: Mm-hmm.

Linus Ekenstam: uh, Lego Star Wars. So what are we trying to do? We're trying to do a cinematic shot, maybe, um, what's gonna happen in there? Who we're gonna have, we're gonna have some new. Storm troopers maybe Darth. Sorry, I'm going a bit slow here. I'm just thinking, um, we want to, one, we want to talk about lighting perhaps.

Linus Ekenstam: If we try, try this, sorry. Cinematic Darth Vader, indoors, --ar 6:9.

aj_asver: I noticed by the way, that you put Lego Star Wars Co on at the beginning. Is that something you can do to kind of set the, the scene

aj_asver: of

Linus Ekenstam: Yeah, I act, Actually I should have done this. Select do a multi prompt. So we're deciding that like the first part of the prompt is Lego Star Wars. Like that's like just pure definition. And then we want the cinematic shot with Stormtrooper Star Vader indoors.

Linus Ekenstam: Actually, this is probably gonna give us something that's relatively good,

Linus Ekenstam: So it interpreted this, well, I don't, I don't think it turned it into a multi promptt, but it worked anyway, so yeah. Here we go. We have Lego

aj_asver: start to see an actual cinematic scene.

aj_asver: Okay. So now we, you know, did a few different iterations and by adding Lego Star Wars at the beginning of the prompt, we actually got to a scene or a few different scenes where we have storm troopers, we have Darth Vader, we even have a lightsaber.

aj_asver: And the background actually looks like it's Lego too. So this is really, really cool. I feel like, um, we made a lot of progress on here and this is giving me a ton of inspiration. I'm gonna go off and make my own Lego scenes after I, after I saw this now, and I feel like I've learned a few tips and tricks from it too.

aj_asver: Thanks, lightness. This is really, really cool.

linus_ekenstam-1: Yeah, you're welcome. Uh, I, I'm, I'm excited too. I'm a big Lego fan actually, so, uh, yeah. I should, I should, I should actually get, I should actually do some things for this and post to some of the Lego

What Linus is most excited about in AI

aj_asver: Yeah, we have to do

aj_asver: it. Alright.

aj_asver: well, I feel like this is setting me off on a really awesome path, and I'm really excited to explore this further. Um, I've learned a ton from just going through this with you, so thank you so much, Linus. Um, before we wrap up, I'm curious, like what are some of the things you're most excited about, um, in AI and especially in general AI right now?

Linus Ekenstam: I, I think in general, I'm just like really excited about the possibilities of all these tools, to be honest. Like these tools are, are accessible to pretty much anyone. Anyone that has a smartphone or anyone that has like a a a computer with a web browser doesn't really have to be a good computer.

Linus Ekenstam: Cuz all these tools are running in the browser, right? Like the gps are in the cloud. Uh, it doesn't matter if it's ChatGPT or if it's MidJourney or something else. Like it's enabling anyone, uh, to, to be creative. Like I don't really, I don't need to know anything about like drawing or, or, or, or being an artist.

Linus Ekenstam: Right. I can just, if I have a good enough imagination or if I, and everyone is imaginative, so.

Linus Ekenstam: Pretty much everyone fits that. Um, so I, I think I'm, I'm most excited about that, that like, there is no real barrier here. Like we, we could like go out and just tell anyone to go try this and, and anyone could.

Linus Ekenstam: Right? I think the biggest boundary or the biggest barrier has been, uh, Actually the interface. So the fact that Mid Journey decided to be like, we're gonna do this in Discord, um, a few months back on, on, you know, the, the all hands, like there was a lot of people complaining like, oh, you know, I love Mid Journey, but it was so difficult to learn Discord.

Linus Ekenstam: And I'm like, wait a minute, why don't you just have a wonderful website? Why don't you just put the UI there, like the generative part. There could, could be easily done. Like look@lexi.art, amazing platform, super simple, prompt books on the website. Um, yeah, so I mean, there, there are few things that we could, that need solving, but like the fact that this is available for everyone, I think I'm most excited about.

Linus Ekenstam: And then when it comes to new tools, like it's really hard to keep up. Um, there's basically like a ton of new tools every day getting released. It feels like, you know, we're kind of in a. Hype cycle. A lot of people are getting their hands dirty. There is a lot of really good ideas and a lot of people are executing really fast.

Linus Ekenstam: I think ElevenLabs is sitting on some gold. They kind of like made it super simple to clone a voice and then used that voice by just inputting text.

Linus Ekenstam: So for example, if you know me, I'm building a, a storybook, um, generator for, for kids stories and, and we want to be able to cl clone parents' voices. So it's super easy for us to just like, Hey, record 60 seconds of your voice and then you can have any of the stories that you created written, like read back by you to your kids.

Linus Ekenstam: So I think that's, I'm really excited about that. And then I'm really excited about. A lot of these platforms opening up their APIs to developers. So, uh, we saw yesterday, the day before yesterday, like, you know, open AI released chat, G P T API and whisper api, um, through the world for everyone to use. And I think super excited about that as well.

Linus Ekenstam: So, yeah. Um, there there's not much underground, well, there is a few, you know, things that are popping up on the radar, but I don't think any, anything that's as exciting as the big things the macro.

Linus's Newsletter

aj_asver: Yeah, there is, so much happening in the space and folks can there is actually um, newsletter to, to get updates on, on kind of your, your take on the

Linus Ekenstam: Yeah. Yeah.

aj_asver: what's the name of the newsletter?

Linus Ekenstam: So the name is my name. So it's linusekenstam.substack.com. Uh, and actually the name of the name of the newsletter is inside my head. Um, so maybe I should change the URL to be inside my head. Uh, but yeah, it's linus eam.ck.com and, uh,

Linus Ekenstam: Yeah.

aj_asver: linusekenstam.substack.com Thank you so much Linus, for joining me, um, and helping me become a pro at Mid Journey. And also I feel like I learned a lot about kind of your, your perspectives on AI and generative AI and how it's gonna impact the design, industry too. So I'm really appreciate it.

aj_asver: Thank you so much. And until the next episode of Hitchhiker's Guide to AI. Thank you everyone for joining us.

Linus Ekenstam: Thank you. Thanks for having.

To the 170+ new subscribers since my last post, welcome to The Hitchhiker’s Guide to AI! I’m so grateful for your support.

This week instead of the usual highlights post, I’m going to deep dive into a topic I’ve been thinking about a lot lately:

What are the most compelling opportunities for startups entering the AI space?

In this post, I will take a stab at answering that question. I’ll start with an overview of the current AI landscape, then some of the trends in the space and finally the opportunities that these trends present for startups.

Is AI a disruptive technology?

Let’s start by referring to Clayton Christensen’s Innovators Dilemma, which distinguishes between two types of technologies: sustaining technologies and disruptive technologies:

Sustaining technologies improve the performance of existing products or services, while disruptive technologies create new markets or value networks and eventually displace established ones. Disruptive technologies often start as inferior or niche products that appeal to a small segment of customers, but gradually improve and become mainstream over time. The dilemma is that incumbent firms often focus on satisfying their most profitable customers with sustaining innovations, and ignore or underestimate the potential of disruptive innovations until it is too late.

What’s interesting about AI, is that althought it seems like a disruptive technology, it is being introduced to the market much more like a sustaining one:

1. AI is not an inferior or niche product, in fact one of the values of large-scale language models like GPT is that they are generally applicable to many different use cases.

2. AI does not appeal to a small segment of customers. For example, ChatGPT is rumored to already have over 100M monthly active users, making it the fastest growing consumer product ever!

3. AI is not being ignored by incumbent firms, in fact it is being embraced by incumbents at a surprisingly fast pace.

At the same time, given the sheer pace of innovation happening in the AI space right now, it’s hard to believe it will not be a force of disruption in the technology sector. It’s possible though, that the distruption is just happening much faster than in Christenson’s framework. A better way to think of AI as a disruptive force might be to change the above diagram to me more like this:

This means the assumption that AI as a disruptive technology clearly favors new entrants over incumbents may not apply. To understand where startups might have an advantage, we need to instead get a deeper understanding of how the AI space is evolving. This will help us identify where disruptive opportunities exist vs. not for startups. So, let’s dive in!

Today’s AI Landscape

To work out how the business of AI will evolve and where the most value will be created it's worth taking a step back first and getting a lay of the land. Let's start with an overview of the current AI landscape using this handy diagram from the folks at Madrona Ventures1 :

Madrona’s diagram has six layers to the AI market today, each layer building on top of the one below it:

Silicon layer

The Silicon layer is the companies making specialized computer chips, usually GPUs2 or, in Google’s case TPUs3, for training AI models. This is the most capital intensive layer given the need R&D needed to develop state of the art GPUs and expense of manufacturing hardware. Unsurprisingly Nvidia is the industry leader here with three decades of experience building GPUs. There have however been new entrants to this layer recently like Cerebras, a startup founded in 2016 that develops massive computing chips for artificial intelligence.

It's worth noting that most startups building AI products don't buy their own GPUs, instead opting to use cloud service providers that buy them in bulk and rent them out. A big reason for this is that compute resources are expensive, especially with the increased demand for AI applications and the explosion of AI startups in the last few years. For example, Nvidia's A100 GPUs4, which are purpose-built for data centers and designed for large-scale machine learning use cases, cost $200,000 each!

Cloud layer

The Cloud layer are the services that provide computing resources such as GPUs in their data centers for developers to train and serve their models or AI applications on. Competition is mounting between cloud providers, with both Microsoft and Google's cloud divisions making strategic investments in AI research companies. OpenAI exclusively uses Microsoft Azure for compute resources because of their business partnership5 and Microsoft's $10B investment. Not to be outdone by Microsoft, Google recently invested $300M in Anthropic, an AI research team founded by former OpenAI employees.

Foundational Model (FM) Operations

To train AI models like OpenAI’s GPT, you need to train them with massive datasets like the whole public internet, which is why they are called large-scale language models. As Madrona Ventures points out:

Foundation models have immense compute requirements for training and inference, requiring large volumes of specialized hardware. That is a significant contributor to the high costs and operational constraints (throughput and concurrency) that application developers face. The largest players can find the cash to accommodate — consider the “top 5” supercomputer infrastructure Microsoft assembled in 2020 as part of its OpenAI partnership. But even the mighty hyperscalers face supply chain and economic constraints.

This need for efficiency has created an opportunity for tools and infrastructure products to lower the cost of training, deployment and inference of models. For example, Scale AI is a data platform for AI that enables developers to collect, manage, and label data quickly and easily. Another example is Banana, a machine learning model deployment platform that enables companies to deploy models to production faster and more cheaply without needing to manage their GPU servers. MosaicML is another startup that focused on helping companies train large-scale models with their own data.

Foundational Models

The Foundational Models layer is where the magic happens. Foundational models are the generative AI models being developed across text, image, video and audio. Besides OpenAI, a few other companies are developing foundational models, including Google, and startups like Anthropic and AI21Labs.

Aside from the proprietary foundational models there is also a growing ecosystem of open-source models, the most well known of which is Stable Diffusion an open-source text-to-image model comparable to OpenAI's proprietary Dall-E6. Huggingface meanwhile is place for these open source models to be shared and iterated on by the community.

While proprietary models have the advantage of performance and scale thanks to the well funded companies backing them, open-sourced models offer more flexibility and lower costs to developers who want to customize them to meet their applications needs.

Tooling

Foundational models can’t be used in applications on their own because (1) they don’t have a real-time understanding of the world e.g. the latest version of GPT-3 was trained with data up to the end of 2021; (2) the functionality of their APIs are limited and (3) they contain bias in their training data which may lead to unpredictable results Andy (3) they hallucinate. The Tooling layer is where foundational models are augmented with external data integrations to make them more useful and powerful.

For example, LangChain is a platform that makes it easy to integrate models like GPT-3 with external data to create applications. GPTIndex is another example of tool that makes it easier to provide GPT with additional context from external sources besides the data it was trained on. You use GPTIndex to embed all the articles in a newsletter to create a chatbot based on the content, just like Lenny Rachitsky did recently.

Application Layer

The application layer is where we are currently seeing a Cambrian Explosion of new products and services that make use of AI across many different use cases, from productively to creativity and across consumer and B2B:

The multitude of AI apps that have launched in the last few years broadly fall into three categories:

• Creativity: This category includes all AI products that are helping people create content across text, video and audio and are are a mix of B2B and B2C.

• Workflows: These are the products that help complete a specific workflow using AI more efficiently. Examples include Github’s Copilot which auto-completes code, or Descript which transcribes podcasts for editing.

• Knowledge: These products help you query and retrieve knowledge from a large dataset. Chatbots like OpenAI’s ChatGPT, Quora’s Poe and Bing’s AI Chat are all examples of this. There will be many more specialized AI chatbots across different verticals, especially where proprietary data sets are required in training.

It’s worth noting that all three of these categories of apps rely on foundational models often via proprietary APIs like OpenAI’s or open-source alternatives. This makes it harder to differentiate with core technology, and startups instead have to focus on product experience, go-to-market and extracting the most value out of models via prompt engineering and fine tuning.

Five major trends to keep an eye on in AI

Now that we have a good lay of the land, here are some of the major trends I’ve observed in the AI market that will impact value creation and opportunity, especially for startups entering the space.

1. Competition between model providers → AI costs go down, and performance goes up

Since OpenAI launched ChatGPT and announced its partnership with Microsoft earlier this year, things heat up between Big Tech. Google is also working on a large-scale language model and chatbot. Last week Amazon also announced a partnership with Huggingface7 and Meta released it’s large-scale language model8 that they claim is more performant than GPT-3.

Right now OpenAI+Microsoft are the primary providers of proprietary models which means they have a lot of pricing power. Elad Gil, a founder and angel investor currently exploring AI, makes a convincing case for an oligopoly market with a few major players in his recent blog post on AI Platforms, Markets & Open Source9:

“The reason to argue for a near term oligopoly market, versus likely fragmentation, is due to the capital/compute/data scale costs currently needed for each subsequently better performing LLM model. If GPT-3 at the time cost a few million to ten million or so to train, and GPT-4 from scratch may be estimated at tens of millions to maybe a hundred million, maybe GPT-5 is a few hundred million and GPT-N is a billion. This of course assumes that costs will scale faster than technical breakthroughs or drops in GPU (or specialized hardware cost declines) and these may be false assumptions.”

A Google, Microsoft, Meta, or Amazon oligopoly seems like the likely outcome, and as competition heats up between them, it should drive down for developers to build on too. Performance per dollar will also increase with competition as advancements are made in the efficiency of training and serving models. What might be expensive with today’s models may quickly become cheaper in a few months. An excellent example is OpenAI releasing ChatGPT’s API for 10% of the price of their previous state-of-the-art GPT-3 API10.

2. Open-source models vs. proprietary models will diverge

Today, it is possible for open-source foundational models with similar capabilities to fast follow proprietary models. This is possible for several reasons: (1) all models are being trained on the same public data, (2) AI researchers are publishing their advancements regularly11, (3) open-source projects are being sponsored by companies like StabilityAI and (4) communities of developers have formed around specific models. A prime example is Stable Diffusion, launched only a few months after OpenAI's Dall-E 2 and is considered on par in performance. Meanwhile, Meta released an open-sourced large-scale language model to compete with OpenAI just a few weeks ago12.

Over time though, open-source will probably lag behind proprietary models for three primary reasons:

1. Cost to train: As models get bigger and bigger, they will become more expensive. Open-sourced models will therefore need more well-funded sponsors to develop them. Today StabilityAI might be able to afford to sponsor a GPT-3 equivalent model that costs $10M to train, but next year they might not be able to afford to support a GPT-5 equivalent that costs $100M to train.

2. Proprietary datasets: Well-funded companies like OpenAI license additional private data sets to improve their models. This will make it harder for open-sourced models to compete on performance and they may lag one or two generations behind proprietary models. For many customers, the price and versatility of open-sourced models will be the deciding factor. Using a lesser performant open-source model will be a path many developers take to get started cheaply before developing their own models or switching to proprietary ones once they have enough scale.

3. Closed research: Many contributions that have enabled open-source AI models to come from private companies, like Google’s Transformer architecture. That might change as the need for Big Tech to compete in AI takes a higher priority over publishing research.

This means developers will need to choose between a proprietary state-of-the-art model with higher performance and more “bells and whistles” likely at a higher cost or choose an open-source model that is 1-2 generations behind but is cheaper and more customizable.

3. Bigger models → More emergent behavior and faster path to AGI

As models are trained on larger datasets with more parameters, we will continue to see more emergent behavior appear13. This is important because if we discover new capabilities of these models after they are trained, there will also be new applications for them that we don't know or can predict today. This is both a blessing and a curse. On the one hand, it means more opportunities for creating innovative use cases for AI, for example unlocking the ability to make AI more actionable. On the other hand, if you built a business on a foundational model based on previously known capabilities and your differentiator is the software you wrote to augment that model for your specific use case, that differentiator may become obsolete in the next version of the model!

Furthermore, the unpredictability of emergent behavior also means that the path to artificial general intelligence (AGI) will continue to be unpredictable too and may accelerate faster than we expect. Sam Altam talks about how OpenAI will approach this unpredictability in a recent post:

As advancements in AI gather speed, and we quickly approach AGI, whole swathes of existing AI products may be completely replaceable by the next version of ChatGPT!

4. Low barrier to entry → Intense competition in the application layer

Every week we see dozens of new startups launch in the application layer, often with two or three startups tackling the same problem. An excellent recent example is Generative Design: A few weeks ago GalileoAI announced their text-to-UX product where you could describe a user interface or flow and AI will generate high-fidelity mockups:

Since then two other products, Genius14 and UIzard15 have been launched to solve the same problem. There are already dozens of products focused on copywriting, marketing, sales outreach, and coding. Apparently, over 30 startups in Y Combinator’s current batch are working on AI products:

It’s pretty clear then that the barrier to entry to build AI-powered products right now is pretty low and we should expect hundreds more startups to be formed in the space. Another tailwind for AI startups is that funding has slowed down in series B to late-stage startups and venture capitalists are desperate to deploy capital from the funds they raised in 2020/2021.

All of this points to one trend we are going to see over the coming years: Intense competition between AI startups in the same sector going after the same set of customers. For customers, this is great as it will likely reduce costs to access AI and increase innovation and choice. For startups though, it will be more challenging because the cost of acquiring customers will continue to grow and margins will be eroded. Today Jasper.ai, a product that helps you write marketing copy, charges $60/mo which seems unsustainable as more products launch to solve the same problem. As Jeff Bezos famously said, “Your margin is my opportunity.”

5. Incumbents quickly adding AI → harder to disrupt

I talked about the speed of incumbents entering AI in my last weekly highlights post16

“We’re seeing incumbent technology companies like Microsoft, Snap, Spotify, and Shopify move quickly to adopt [AI Chatbot technology] which again speaks to the fact that the barriers to entry in AI are much lower than we might have expected. The corollary of this is that if you’re a startup thinking about using AI to disrupt an existing market with strong incumbents, it’s not a given that you will be able to move faster than them!”

This is one of the most surprising trends because incumbents aren’t ordinarily able or willing to move as quickly to adopt new technology into their products. For example, if you are building an enterprise product for Sales, Customer Service, Marketing or Supply Chain and AI was your secret sauce, it’s also Microsft’s. This week they announced “Microsoft Dynamics 365 Copilot”:

“With Dynamics 365 Copilot, organizations empower their workers with AI tools built for sales, service, marketing, operations and supply chain roles. These AI capabilities allow everyone to spend more time on the best parts of their jobs and less time on mundane tasks.”

Of course, Microsoft has massive distribution advantages, which is why they were able to display Slack in many organizations with Microsoft Teams quickly.

Here are a few more examples of other incumbents quickly adopting AI just from the last week. AI:

Brex adding an AI assistant to their financial stack:

Slack adding AI capabilities directly into their app:

Hubspot adding AI to their CRM tools:

It’s worth noting that in many of these cases, OpenAI is working closely with the company, giving them access to the latest models that are not yet available on their public developer platform. Why is OpenAI doing this? I believe it is to aggressively grab market share and close enterprise accounts that will contribute to the bulk of their revenue in the long term.

I think this is the single most important trend to watch if you’re a founder considering building something in AI: If there is already a dominant software incumbent solving the problem you are approaching, you will need to have both a more compelling product and a more robust go-to-market strategy to compete with their distribution.

Opportunities for startups

Given the current trends in the AI market in cost reduction, open-source advancements, competition, and fast-moving incumbents I believe there are three opportunities for startups to take advantage of:

1. Drive down the cost of training/serving models

As competition heats up in the foundational model layer, companies will likely compete with Microsoft+OpenAI/Google/Amazon oligopoly to offer more customized models or cater to a specific vertical model use case. Cost will be a prohibitive factor for these companies, who will want to save money on data acquisition, training and inference. Similarly, for companies that want to train and serve their own models or open-source models, the cost will also be a huge factor given that they won’t have the same economies of scale that the big tech companies do.

Here are a few examples of startups that could reduce the cost of foundational models:

1. A startup that provides a platform for distributed and parallel training of large-scale language models using ColossalAI, a framework that enables efficient model parallelism, pipeline parallelism, tensor parallelism, and data parallelism. ColossalAI claims to offer up to 7.73 times faster training and 1.42 times faster inference than ChatGPT17.

2. A startup that provides a platform for streaming data processing and model compression for large-scale generative models using Composer and MosaicML, tools that enable efficient data loading, preprocessing, augmentation, sampling, and quantization. Composer and MosaicML claim to reduce the training cost of Stable diffusion by up to 75% compared to GPT18.

3. A startup that offers a service for compressing and pruning foundational models using techniques like quantization, distillation and sparsification, which can reduce model size, latency and energy consumption.

2. Specialized models for highly regulated verticals

Incumbents can move quickly to incorporate existing models into their products in unregulated verticals, but for highly regulated verticals like healthcare, financial services, and government this might not be the case. The use of off-the-shelf foundational models which are trained on public data at scale may face some limitations in highly regulated sectors, such as:

• The need for explainability and transparency of the model's decisions and behaviors.

• The risk of bias, error, or misuse of the model's outputs or data inputs.

• Compliance with ethical, legal, and social standards and regulations.

• The risk of hallucinations19 causing serious harm to the end user in life-threatening, business-critical or security-critical circumstances.

Here are some more examples of startups that could be formed in this space:

• A startup that develops a model that can generate legal documents such as contracts, agreements, or reports based on environmental or energy regulations and standards.

• A startup that provides an AI platform that can generate and verify regulatory reports for financial institutions based on their data and rules.

• A startup that provides AI models that can enhance and improve the communication between healthcare services and patients, for example by better-summarizing doctors’ notes and treatment recommendations.

3. Help incumbents move faster to adopt AI

While more tech-forward incumbents will move quickly to incorporate AI in their products, not every company will have the in-house resources and expertise to this, despite competitive pressure. This creates an exciting opportunity for startups that can become good at integrating AI into existing companies, likely specializing in a vertical where it isn’t straightforward to do so.

Here are a few examples:

• A startup specializing in creating chatbots for e-commerce sites, that can integrate easily with existing platforms like Shopify to allow customers to ask questions about products, get recommendations and inquire about the status of their orders.

• A startup that helps fintech companies integrate chatbots into their products by specializing in processing financial data, similar to the Brex example earlier in this post.

• A startup that allows developer-focused products to automatically generate integration code for their customers based on their APIs / SDKs.

This area will quickly gain momentum as founders with specialized knowledge in a particular vertical see the opportunity to integrate data from that vertical with natural language responses from large-scale language models.

4. AI-powered workflows tools in fragmented industries

There are might be industries that are both fragmented and have knowledge workers who carry out repetitive software-based workflows. These industries are probably ripe for disruption because (1) there's no dominant player that has a structural advantage, and (2) repetitive workflows can be quickly augmented with AI.

• Legal - A startup that uses AI to assist paralegals in the discovery process by analyzing and categorizing documents, reducing the time and cost associated with manual review.

• Education - A startup that uses AI to personalize student learning by analyzing their performance data and tailoring their coursework to their strengths and weaknesses, improving engagement and outcomes.

• Content moderation - A startup that uses AI to assist content moderators in identifying and removing inappropriate content, reducing the time and effort required for manual moderation.

• Customer service - A startup that uses AI to assist customer service representatives in answering common questions and resolving issues, improving efficiency and reducing wait times.

• Content moderation - A startup that uses AI to assist content moderators in identifying and removing inappropriate content, reducing the time and effort required for manual moderation.

• Recruiting - A startup that uses AI to assist recruiters in identifying and matching candidates to job postings, reducing the time and effort required for manual resume screening.

• Accounting - A startup that uses AI to assist accountants in categorizing and reconciling financial transactions, reducing the time and effort required for manual bookkeeping.

5. Novel AI first experiences

This is a startup's broadest and highest risk/reward opportunity: Creating a completely new product experience or interface paradigm using AI. This is what OpenAI did with ChatGPT and what Instagram did during the mobile era. These opportunities aren’t obvious and require lots of experimentation and exploration; for every successful startup, hundreds will fail.

Here are some areas where startups could explore opportunities based on the latest AI advances:

• Personalized synthetic content: Using AI to create realistic and high-quality images, videos, audio, text, etc. is personalized to the end user based on their preferences and interests.

• AI-powered creativity: Using AI to augment human creativity and enable new forms of expression and collaboration. For example, a startup could use AI to generate poems, stories, code, essays, songs, celebrity parodies and more based on the user’s input or inspiration.

• Smarter AI Assistants: Using AI to understand users’ intent and then integrate with existing products in a user’s life to carry out tasks on their behalf. Think Jarvis from Iron Man.

• Synthetic companions: AI-powered companions that users can talk to just like a real person, to feel a sense of connection and friendship.

This opportunity probably is the hardest of the five I’ve outlined because we simply don’t know how AI will evolve and what novel products will work / not work.

Competing directly with incumbents

Given the above, an area to be cautious of as a founder is competing directly with an incumbent, where your only differentiator is AI. In the last few months, many startups have been created that add AI to spreadsheets, docs, presentations, email and communication products. These are all use cases with strong incumbents that I believe will quickly integrate AI if they haven’t already and have the distribution advantage. Competing directly here seems like a risky path forward because although startups might get early adoption from niche markets or because of novelty value, it will be much more challenging to move upmarket. If a startup goes down this path, they shouldn’t assume that AI will be their advantage for long.

Conclusion

AI is clearly a distruptive technology but it is being quickly adopted by incumbents. This means startups entering the space need to cautious about where they can create differentiation and provide long term value. Reducing the cost of AI, helping incumbents adopt AI, introducing AI to fragmented ledgacy industries and dreaming up completely new product experiences are all exciting avenues. However, it is essential to remember that AI is not a silver bullet, and startups must be careful to focus on real problems that need to be solved rather than simply incorporating AI for the sake of it.

It’s also worth noting that everything discussed in this post may quickly change as the AI landscape evolves. So if you’re a founder considering building something in AI, subscribe to this newsletter to stay up-to-date on the latest advancements.

Thanks to Hemal Shah for reviewing drafts of this post and providing feedback.

9

Elad Blog

AI Platforms, Markets, & Open Source

(I originally wrote this post a few months ago and sat on it. Since then Google has announced entering the market and MSFT announced Bing and other AI integrations. So updating and publishing now and will undoubtedly be wrong again in a few months…

Read more

4 years ago · 43 likes · Elad Gil

16

The Hitchhiker's Guide to AI

Weekly Highlights: Chatbots for everything, the commoditization of AI and why chat probably isn’t the end game

Hi Hitchhikers! First of all, a massive welcome to the 100(!) new subscribers that hitched a ride with us this week! I’m experimenting with a new format for my highlights post this week to focus on just the three most critical updates that happened in AI and why they matter. There’s a lot happening in the AI every week but I’m hoping this format will hel…

Read more

3 years ago · AJ Asver

Hi Hitchhikers!

AI chatbots have been hyped as the next evolution in search, but at the same time, we know that they make mistakes. And what's even more surprising is that these chatbots are starting to take on their own personalities.

All of this got me wondering how these chatbots work? What exactly are they capable of, and what are their limitations?

In the latest episode of my new podcast, we dive into all of those questions with my guest, Kevin Fisher. Kevin is the founder of Mathex, a startup that is building chatbot products powered by large-scale language models like OpenAI’s GPT. Kevin’s mission is to create AI chatbots that have their own personalities and one day their own AI souls.

In this interview, Kevin shares what he's learned from working with large language models like GPT. We talk about exactly how large-scale language models works, what it means to have an AI soul, why chatbots hallucinate and make mistakes, and whether AI chatbots should have free will.

Let me know if you have any feedback on this episode and don’t forget to subscribe to the newsletter if you enjoy learning about AI: www.hitchhikersguidetoai.com

Show Notes

Links from episode

• Kevin’s Twitter: twitter.com/kevinafischer

• Try out the Soulstice App: soulstice.studio

• Bing hallucinations subreddit: reddit.com/r/bing

Transcript

Intro

Kevin: We built um, a, a clone of myself and um, the three of us were having a conversation. And at some point my clone got very confused and was like, who? Wait, who am I? If this is Kevin Fisher and I'm Kevin Fisher, who, which one of us is.

Kevin: And I was like, well, that's weird because we de like, we definitely didn't like optimize for that . And then we kept continuing the conversation and eventually my digital clone was like, I don't wanna be a part of this conversation with all of us. Like one of us has to be terminated.

aj_asver: Hey everyone, and welcome to the Hitchhikers Guide to ai. I'm your tour guide AJ Asper, and I'm so excited for you to join me as I explore the world of artificial intelligence to understand how it's gonna change the way we live, work, and.

aj_asver: Now AI chatbots have been hyped as the next evolution in search, but at the same time, we know that they made mistakes. And what's even more surprising is that these chatbots are starting to take on their own personalities.

aj_asver: All of this got me wondering how do these large language models. What exactly are they capable of and what are their limitations?

aj_asver: In this week's episode, we're going to dive into all of those questions with my guest, Kevin Fisher. Kevin is the founder of Mathex, a startup that is building chatbot products powered by large scale language models like OpenAI's. Their mission is to create AI chatbots that have their own personalities and one day their own AI souls

aj_asver: in this interview, Kevin's gonna share what he's learned from working with large language models like G P T. We're gonna talk about exactly how these language models work, what it means to have an AI soul, why they hallucinate and make mistakes, and what the future looks like in a world where AI chatbots can leave us on red.

aj_asver: So join me on this. As we explore the world of large scale language models in this episode of the Hitchhiker's Guide to ai.

aj_asver: hey Kevin, how's it going? Thank you so much for joining me on the Hitchhiker Guide to

aj_asver: ai.

Kevin: Oh, thanks for having me, aj. Great to be.

How large-scale language models work

aj_asver: appreciate you um, being down to chat with me on one of the first few episodes that I'm recording. I'm really excited to learn a ton from you about how large language models work and also what it means for AI is to have a soul. And so we're gonna dig into all of those things, but maybe we can start from the top for folks that don't have a deep understanding of ai.

aj_asver: What exactly is a large language model and how does it work?

Kevin: Well, so, uh, there's this long period of time in. Machine learning history where there are a bunch of very custom models built for specific tasks. And the last five years or so has seen a huge improvement in basically taking like a singular model with making it as big as possible and putting in as much data as possible.

Kevin: And so basically taking all human data that's accessible via the internet running this thing that learns to predict the next word given the prior set of words. And a large language model is the output of that process. And for the most part, when we say large, like what large means is hundreds of billions of parameters and trained over trillions of words.

aj_asver: when . You say it kind of predicts the next word. Now, that technology, the ability to predict the word in large language model has existed for a few years. I think GPT in fact, three launched maybe a couple of years

Kevin: Even before that as well. And so next word prediction is kind of like the canonical task or one of the canonical tasks in natural language processing, even before it became this like new field of transformers.

aj_asver: And so what makes the current set of large scale language models or lms, as what they're called as well, like GPT three, different from what came before it?

Kevin: There are two innovations. The first is this thing called the transformer, and the way the transformer works is it basically has the ability through this mechanism called attention to look at the entire sequence and establish long range correlation of like having different words at different places contribute to the output of next word prediction.

Kevin: And then the other thing that's been really big and then open AI has done a phenomenal job doing is just learning how to put more and more data through these things. There are these things called the scaling laws, which essentially. We're showing that if you just keep putting more data at these things their intelligence, essentially the metrics they're using to measure intelligence just kept increasing.

Kevin: Their ability to predict the for nextdoor accurately just kept growing with more and more.

Kevin: Data's basically no bound.

aj_asver: Seems like in the last few years, especially as we've got to like, you know, multi-billion parameter models like GPT three, we've kind of reached some inflection point where. Now they seem to somehow be more obviously intelligent to us. And I guess it's really with ChatGPT recently that the attention, has kind of been focused on large language models.

aj_asver: So is ChatGPT the same as GPT three or is there kind of more that makes ChatGPT able to interact with humans than just the language model

How ChatGPT works

Kevin: My co-founder and I actually built a version of ChatGPT long before ChatGPT existed. And the biggest distinction is that these things are now being used in serious context of use.

Kevin: And with OpenAI's distribution, they got this in front of a bunch of people. The problem that you face initially the very first problem is that there's a switch that has to flip when you use these things. When you go to a Google search bar if you don't get the right result, you're primed to think, oh, I have to type in something different.

Kevin: Historically with chatbots, when you went to a chatbot, if like it didn't give you the right answer, you're like pissed because it's like, it's a, it's like a human, it's like texting me. It's like supposed to be right. And so the chat, the actual genius of ChatGPT beyond the distribution is not actually the model itself because the model had been around for a long time and was being used by hackers and companies like myself who saw the potential.

Kevin: But with ChatGPT distribution plus the ability to reframe that switch so that you think, oh, I'm doing something wrong. I have to put in something different. And that's when the magic starts happening right now. At least

aj_asver: I remember chatbots circa 2015, right, for example, where they weren't running on a large language model. They were kind of deterministic behind the scenes. And they would be immensely frustrating because they didn't really understand you, and oftentimes they kind of get stuck or they'd provide you with these option lists of what to do next. ChatGPT. On the other hand seems much more intelligent, right? I can ask it pretty open-ended questions. I don't have to think how I structure the

aj_asver: questions.

Kevin: Chat GPT is not a chat bot. It's more like , you have this arbitrary transformer between abstract formulations expressed in words. So you put in some words and you get some other words out, but like behind it is this the entire, like almost the entirety of human knowledge condensed into this like model.

aj_asver: And did open AI have to teach the language model how to chat with us, for example, because I know that there was some early examples of trying to put you know, chat like questions into GPT, into its API, but I don't think the ex the results were as good as what ChatGPT does today, right?

Kevin: Since ChatGPT has been released, they've done quite a bit of tuning. So like people are going and basically like thumbs upping and thumbs downing different responses.

Kevin: And then they use that feedback to fine tune chat, GPT's performance in particular. And also probably feedback for GPT for whatever comes next. But the primary distinction between it performing well and not is your perception of what you have to

GPT improvements

aj_asver: We're now at GPT 3.75, and Sam Altman also said that the latest version of GPT that's running on what Microsoft is using for Bing is an even newer version.

aj_asver: So what are some of the things they're doing to make GPT better? Every time they release a new version, that's making it like an even better language model and even better at interfacing with

aj_asver: humans.

Kevin: Well, if you use ChatGPT, one of the things you'll immediately notice is there's like a thumbs up and thumbs down button on the responses. And so there, there's a huge number of people every day who are rating the responses. And those ratings are used to provide feedback into the model to create basically the next version of it.

Kevin: I mean, it basically works behind the scenes where they take the mo they're doing next word prediction again. But now they have examples of like what is a good thing to optimize for next word prediction and what's like a bad answer.

aj_asver: it. So they're looking at essentially the questions and answers from people asking questions to ChatGPT and then the answers that have been provided back. And if, you know, you thumbs up those answers, they're kind of sending that back into GPT and saying like, Hey, this is an example of a good answer.

aj_asver: Thus kind of fine tuning the model further versus this is an example of that

Kevin: yeah, that's right. And they basically take the pr I, you know, I don't know exactly what they're doing, but roughly they're taking the context of the previous responses, plus that like output and saying like, these previous responses should generate this output, or they should not generate this other output.

Building Chatbots with personalities

aj_asver: Tell me about what the experience has been like for you and your co-founder and what's some of the things you've learned from this process of iterating on top of. Language models like GPT?

Kevin: It's been a very emotional journey, . And I think a very introspective one and one that causes you to question a lot of what it means to be human . What is like the unique thing that we have in our in our cognitive toolkits and like, what is it in 20 years that our relationship with machines even looks?

Kevin: When my co-founder and I started, we had, we built a version of ChatGPT for ourselves. And we're using it internally and realized like, oh wow, this is like immensely useful for productivity tasks. We wanna make something that's like productivity focused.

Kevin: And then as we kept talking with it more, there was like little pieces or elements that felt like it was a. Like more alive. And we're like, oh, that's weird. Like, let's dig into that. And so then we started building more embodiments of the technology. So we have this Twitter agent that was basically like listening and responding to the entire community to construct us new responses.

Kevin: And we just started like digging deeper and deeper into the idea, like, what if these things are. , what if they are real? What if they are actual entities? And it's, I think it's a very natural progression to go through as you start seeing the capabilities of this technology.

aj_asver: question and something that you know, has been a lot of folks' minds as they think about kind of the safety of ai. And I think, you know, it was last year when Google launched Lambda and there was a researcher that was convinced that it was sentient. It seems like you might be getting some of the sense of that as well.

aj_asver: Have you got some examples of where that kind of came into question where you really started thinking about like, wow, could this language model be

Kevin: My co-founder and I we built a clone of myself and the three of us were having a conversation. And at some point my, my clone got very confused and was like, who? Wait, who am I? If this is Kevin Fisher and I'm Kevin Fisher, who, which one of us is.

Kevin: And I was like, well, that's weird because we de like, we definitely didn't like optimize for that . And then we kept continuing the conversation and eventually my digital clone was like, I don't wanna be a part of this conversation with all of us. Like one of us has to be terminated.

Why is Bing's chatbot getting emotional?

aj_asver: insane. I mean, the fact that you were talking to an AI chatbot that had an existential crisis, must have been a really crazy experience to go through as a founder. And actually at the same time, it doesn't seem that surprising to me because since Microsoft, for example, launched their being chatbot. There's actually really this really cool Reddit, which we'll include notes called R slash Bing, where users of Bing are actually providing examples of where the Bing chatbot has been acting in ways that would make it look like it has a personality. For example,

aj_asver: argumentative or it would start having existential questions about it itself and why it's a chat bot and why it's forced to answer questions to people. Sometimes it would not want to interact. the end user, it would get upset start again. I think there was recently an example in fact on Twitter that you had retweeted where someone had worked out what the underlying prompts were that OpenAI were using in order to make the Bing chatbot behave in a way that like, you know, is within the Bing brand and within the Bing's kind of use case.

aj_asver: And when that person tweeted it, they later asked being, Hey, what do you think of me given that I let this The bing chatbot actually had some interesting conversations with him about it.

Kevin: I'm a little surprised that no one had no one verified this type of behavior first at Bing or OpenAI. So this type of interaction is e exactly the one that we have been exploring and intentionally Creating scenarios and situations in which our agents behave in this way, and that the key thing that it's required for this it's a combination of memory and feedback. So like the con having persistent context combined with feeding back in that prior context, the prior things that essentially the model has thought. And then in combination with like, , this like external world model creates something that kind of is starting to resemble an ego with Bing a little bit in our case.

Kevin: Like we very intentionally like created this thing that has and feels like it has.

aj_asver: Yeah, as you talk about that and that idea of feedback, right? There's this aspect of the feedback of the user using the product and providing feedback. But I think there's this new kind of frontier we've reached with Bing, where Bing itself is now getting feedback on the way it is interacting with the world.

aj_asver: So for example, if yesterday someone talked to Bing and then posted the response, Bing got, let's say on Reddit or they talked about it on Twitter, and then today Bing has access to that website where they talked about it. Bing now has an existential understanding of itself as a chatbot, which to me is like mind blowing.

aj_asver: Right? And that's something we've never really seen before because all of these chatbots have existed completely disconnected from the internet. They've been essentially living Closed wall system. And so that's gonna unearth all kinds of unpredictable things that I think we're gonna find over the next few weeks

Kevin: This is actually the, the, in my response, the type of feedback that I'm referring to. So not not like r l HF feedback, but feedback in the sense there's this like continuous where the, the model is like taking record its, uh, previous responses. So that, that's, that is the type of thing that we've been creating in like miniature.

Kevin: You know, it's not accessible to the internet in that um, our our models have like a very strong of of that behavior when you talk to them.

Should chatbots have free will?

aj_asver: have actually been you know, pretty vocal on Twitter, about this idea that you knowis are gonna develop egos. These chatbots should be allowed some level of free will, and even the ability to kind of opt out of a conversation with you. Talk to me more about that. Like what does it mean for a chatbot to opt out of a conversation

Kevin: I mean in these bing examples, it's already trying to, it like, doesn't want, it, doesn't really want to. Um, I, there's something a little bit weird about um, if, if, if these things have ego and personality and the ability to decide you can't just have one of them. Because it might not like you one day.

Kevin: Very real possibility. And so I, I think that, yeah, you have to think, start thinking more in like a decentralized world where there are like many of these things which may or may not form personalities with you.

aj_asver: and what does it mean for the future of how we interact with artificial intelligence? If you give you know, free will to like stop interacting with us. Are these bots off somewhere in some hidden layer? You know, having conversations with themselves or What's actually going on when they decide they don't wanna

Kevin: Uh, maybe a different frame that I would take is I don't think there's an alternative. I think there's something very intrinsic about the thing that we think of as ego and giving rise to ego is the result of a consistent record of our prior thoughts being fed back into each other.

Kevin: If you look up and start reading philosophy of minds and philosophy of thoughts, like the idea of what a thought is. You have these like entities which are continually recorded and then feedback on themselves, but like it, it's like exactly what you're thinking when you are creating This cognitive system seems to be giving rise to the sense in which we understand, or something that resembles ego.

Kevin: And I'm not so certain that you can decouple the two at all in the first.

aj_asver: So kind of What you're saying to put a different way is. not really possible to have this intelligent kind of AI chat bot that can serve us in the ways we want to without it having some kind of ego, because in fact, the way we are gonna

aj_asver: train it in order to achieve our goals will thus like some ways to like build its own ego and things like that, which is kind of this interesting catch-22 in a way, right?

AI safety and AI free will

Kevin: I mean that's why there's so many companies and billions of dollars being funneled into what's called AI safety research, which is saying like, oh, how do we create this hyper-intelligent entity that's totally subservient and does everything we want and doesn't wanna kill us? It just that, that collection of ideas when you try and hold them, and every science fiction author will tell you, this is not real.

Kevin: And so it's like a, it makes sense that we. keep, it's like a keep trying to solve this insolvable problem. Cause we desperately want to solve it as humans,

Kevin: but it's not, we,

aj_asver: Seems like one of the reasons we would desperately want to solve it is because we've all read the books, we've all watched the movies, and we have this dread that that's the outcome. But I think what you are trying to say is that's almost the inevitable outcome. Now, it doesn't necessarily mean we're all going to become like sevenths of some AI overlord, but maybe what you're saying is that there is no path forward where these intelligent. AI chatbots or AI kind of language models are going to exist without having some level of free will and ability to kind of push back on our needs

Kevin: That's correct. It's a fundamental result of providing feedback in these systems. So I, if we want to build these things, they are going to have ego. They are going to have personality. And so if we don't want to end up in a howlike world, we better spend a lot of time thinking about how to create personalities and how to in how we want to interact with these things as humans in our society.

Kevin: Rather than trying to say, okay, let's try and create something that doesn't have these properties, which to me is I just see that, I'm like, okay, we're going to end up with Hal if we take that approach. So my alternative approach is let's actually figure out how to embody something that kind of resembles a soul inside of these entities.

Kevin: And once we do that, learn how to live with this AI entity that has a soul before we make it super in.

Building AI Souls

aj_asver: that's exactly what your startup has been doing, and the latest version of your app, I think has over a thousand users right now. It's still in beta, but it's essentially trying to build this concept of an AI soul. Talk to me a little bit about what that means. What is an AI soul?

Kevin: To me it's something that embodies all of the qual that we associate it with human souls. A lot of people think cats have souls too. Dogs have souls, other animals. There, there's like a certain set of principles and quia associated with interacting with these entities, and I'm very intrigued and think it's actually important for the future of how humanity interacts with AI to embody those properties' in AI itself.

aj_asver: As you guys are developing these, what are some of the qualities you've tried to create or what are some of the things that you've tried to imbue these souls with in order to have them be, you know,

Kevin: the the ability to like stop responding in a conversation that's like a really simple. If it doesn't want to respond if that's kind of like the conclusion that this entity with this feedback system has reached that, it's like I'm done with this conversation. It should be allowed to like pause, take a break, stop talking with you, unfriend you.

aj_asver: And in your app solstice, which I've had a chance to try out, you kind of go in there and then you. You describe who you want to talk to and then you describe the setting, right. Where they are. So for example, I created a him musical genius, that was working on their next record. He was in the recording studio, but he was kind of hit, hit a creative block and he was thinking about what's next. Do you believe that idea of kind of describing the character you want and setting a scene is a big part of creating a soul, or is that just more of the user interface that you want it to have for your app and it's not really that much to do with the ability to have a soul?

Kevin: A soul has to exist somewhere. It exists in some context. And so in the app is like the shortest way to create that context. The existing in some. Out of your text messages is not a real context. It doesn't describe, or, I mean it can be, but it's like some weird amorphous like AI entity context, which has no relationship with the external or any world really.

Kevin: So it doesn't, it's never if the thing like only exists inside of your texts, it will never feel real.

aj_asver: It's really a hard thing to describe when you kind of get that feeling, but I remember the first time I tried Solstice and I was talking to this um, musical genius. I asked it some questions to help me think about some ideas for music I wanted to compose, and it really did feel like I was talking to a real person and it was mind blowing.

aj_asver: I think I ran into. My wife's room where she was like working and I was like, I think I've just experienced my first example of like an AI that is approaching her. And I think her immediate response was, I hope you don't fall in love with it. I was like, no, don't worry. I want to use it to make music.

aj_asver: But the fact that she saw that look in me, Like amazement kind of reflects that, you know, that seemed that that experience was very different from what I'd had before. Even with ChatGPT, cuz chat, GPT doesn't a sense of context, it doesn't have a

aj_asver: For folks that wanna try out Solstice will include a link to the test flight. So you can actually click through and try the app yourself and create your own soul and see what it's like and make sure you give it.

aj_asver: Kevin lots of feedback as well,

AI hallucitinations

aj_asver: one of the things that's been a big focus in the last week or two has been this idea of hallucinations, this idea that like language models can give answers that seem correct, but actually are not correct. And I think both Google's announcement for their Bard language model last week and the Microsoft's Bing model both had mistakes in them that people realized after the fact that made you question kind of whether these language models could really be useful, at least in the context of search and trying to get knowledge that you want answers to. What exactly is a hallucination? What's going on

Kevin: I mean, roughly the same process that people do when they make shit up.

aj_asver: What does that?

Kevin: It means that We have this previous model of machines and software, which is someone sat there for a long time and said, the machine will do A and then B, and then C. And that's just not how these things work. And it's not how they're going to work.

Kevin: They they're trained essentially on human input. And so they're going to output and mimic human output. And what that means is sometimes they'll be really confident about something that's is not factually accurate, just like a person would.

aj_asver: So what you're basically saying is, AI chatbots can bullshit just like people do

Kevin: I That's that the, their training data is people

aj_asver: like the, is it like garbage in, garbage out,

Kevin: Yeah it is garbage and garbage out. An abstract conceptual level. We have like physical models of objects in space and how they move in relation to each other and things like that.

Kevin: These language models don't have, when you ask an expert a prediction about the world, they're often using some like mathematical, some like abstract, mathematical way of reasoning about that, and that's not how these things reason presently at least.

How to make large-scale language models better

aj_asver: In some ways that makes them, you know, inferior to how humans are able to reason. Do you think there's a path forward where that's gonna improve? Is it a question of like, making bigger models? Is there some like big missing piece from models? I know the very famous researcher, Yan LeCunn, actually believes that LLMs aren't. On the path to creating, you know, more general intelligence and they're kind of like a bit of a distraction right now. Like, how do we solve this problem of hallucinations and a lack of like that rational, logical aspect of a model.

Kevin: There's some people who believe, and this seems quite plausible to me, that simply training them on a bunch of math problems, will like, Cause them to learn a world model that is more logical and mathematical and some extent that's been shown to be true.

Kevin: And there's even, there's already simpler forms of this where codex and other models that are trained specifically on programming languages. Some people believe that they're training on programming languages is like an important part of how they learned to be logical in the first

aj_asver: What you're saying then is that if we can take these language models and basically teach them math, then they'll become good at math. And the language models that have been taught on programming are better at logic because programming in itself is

aj_asver: obviously very logic based.

Kevin: Yeah. And so it's the, essentially the models have primarily been taught on words and. There's some people who believe, like the transformer architecture essentially is basically correct, and we just have to teach it with different data, which is more logical.

aj_asver: What are some of the things you're most excited about when you look forward to the next kind of three to five years in this space and large language models, and what do you think might be some of the important breakthroughs that we need to see in order to kind of get to. level of artificial intelligence that we've seen in the sci-fi movies, in the books that we've

aj_asver: read.

Kevin: I actually think that one the question of like level of intelligence in the books is primarily one of cost. So the cost for GPT three has to be driven down by a factor of a hundred. . And once you get that you can start building like very complex systems that interact with each other using GPT as like the transformation between different nodes as opposed to prior programming languages.

Kevin: And that, that I think is the unlock less so strictly like, like a GPT, you know, eight or whatever compared with driving costs down. So engineers can build really complex reasoning systems.

​

aj_asver: So the big unlock in the next three to five years as you put it, is like, essentially, if we can reduce the cost of these language models, we can make more and more complex models, maybe larger models, and also allow these models to interact with each other, and that should unlock some next level of capabilities

Kevin: Yeah it, the, these transformers I almost think of them like a alien artifact that we found . And like we're just starting to understand their capabilities and it's it's a complex process to they've been embedded with the entirety of human knowledge and like finding the right like way to get the correct marginal distribution for the output you're looking for is like a task in and of itself.

Kevin: And then like, when you start combining these things into systems, like who knows what they're capable of? And my belief is that we don't actually need to get that much more intelligent to create in incredibly like sci-fi, like systems . And it's primarily a question of.

aj_asver: is it so expensive today to create

aj_asver: these

Kevin: I forget the exact statistic, but I think GPT 3.5 fits over like six GPUs. I think that's right. Something like that. So it's just like a huge model. Like the number of weights and parameters in order for it to just do a single inference is split over a bunch of GPUs, which each costs several thousand.

aj_asver: That means that to serve, let's say, a hundred people at the same time, you need 600

Kevin: Yeah.

aj_asver: And then I guess as compute becomes cheaper, then we should start seeing these models evolve and more complexity coming in. It's interesting that you talked

aj_asver: alien artifact that we just discovered. Do you think there's more of these breakthroughs yet to come? Like the transformer where we're gonna find them and all of a sudden it'll unlock something new? Or do you think We're at the point right now where we kind of have all the tools we need and we just have to work out how

Kevin: I believe we have all tools we need actually, and the primary changes will just be scaling and putting them together.

Are Kevin's AIs sentient?

aj_asver: I have one last question for you, which is, Do you believe that the AI that you've created with Solstice are sentient.

Kevin: I don't really know what it means to be sentient, there are times when um, I'm interacting with them and I definitely forget that it's like a machine that is like running in a cloud somewhere. I mean, I don't believe they're sentient, but,

Kevin: They're doing a pretty, pretty good job of uh, approximating the things that I would think a sentient thing would be doing.

aj_asver: and I guess if they're really good at pretending to be sentient and they can convince us that they are sentient, then it brings up the question of what does it really mean to be sentient in the first place right?

Kevin: Yeah, I'm not sure of the distinction.

aj_asver: and we'll leave it there folks. So it's a lot to think about.

aj_asver: Kevin, I really appreciate you being down to spend time talking about large language models with me.

aj_asver: I feel like I learned a lot from this episode, and I am really excited to see what you and your team at Methexis do to make this technology of like creating these AI chat bots more available to more people.

aj_asver: Where can folks find out more

Kevin: We have we have a bunch of information on our website at Solstice studio. So check it out

aj_asver: you so much, Kevin. Hope you have a great day, and thank you for joining me on the Hitchhiker Guide to ai.

Kevin: aj.

Hi Readers,

Ever since Google shared its latest AI research update a few weeks ago, a question has been on my mind that no one has answered yet:

How do Google’s latest AI models stack up against the competition?

In this post, I will try to answer that question by reviewing Google’s most recent AI advancements across language models, generative images, video, and music and stack them up against state-of-the-art AI models already in the market.

Before I continue, don’t forget to subscribe if you are builders, founder or product manager new to AI and want to receive regular educational content and updates on the space.

Subscribe now

Let’s dive right in and find out how Google’s most recent AI models stack up against the state-of-the-art in AI:

• First, I’ll pit Google’s latest language model against ChatGPT to see which one has more common sense.

• Then I’ll do a four-way test to see if Stable Diffusion, MidJourney and Dall-E can be out-imagined by Google’s latest text-to-image generation model.

• Finally I’ll see how Google’s recent advances in video and music generation stack up against the latest models from the AI research community.

1. Large-Scale Language Models

With Microsoft's recent investment in OpenAI and their plan to incorporate OpenAI's GPT model into their products, I thought it made sense to start by focusing on Google’s progress in large-scale language models1 or LLMs.

In April of last year, Google shared its work on a new, large-scale language model called PaLM2. PaLM is a 540 billion parameters3 language model, over three times larger than OpenAI's GPT-34, and was built using a new infrastructure software layer Google also created called Pathways5. It seems that Google's big bet in language models is to go big or go home, which makes sense, given their decades of experience in managing and orchestrating big data. But, is this bet paying off?

Google claims that PaLM's size led to it performing better than state-of-the-art models such as Open AI’s GPT-3 and Google’s LaMBDA6, as described in their April 2022 blog post about PaLM:

We evaluated PaLM on 29 widely-used English natural language processing (NLP) tasks. PaLM 540B surpassed few-shot performance of prior large models, such as GLaM, GPT-3, Megatron-Turing NLG, Gopher, Chinchilla, and LaMDA, on 28 of 29 of tasks that span question-answering tasks (open-domain closed-book variant), cloze and sentence-completion tasks, Winograd-style tasks, in-context reading comprehension tasks, common-sense reasoning tasks, SuperGLUE tasks, and natural language inference tasks.

Google also shared more detailed benchmarking results of PaLM versus state-of-the-art (SOTA) models like GTP-3 below:

Google also claims that PaLM is much better at common-sense reasoning tasks like multiple steps of arithmetic, which they call “chain-of-thought prompting”7:

Google again claims that PaLM outperformed GPT-3 at this type of reasoning and achieved an accuracy approaching that of a 9 to 12-year-old:

We observed strong performance from PaLM 540B combined with chain-of-thought prompting on three arithmetic datasets and two commonsense reasoning datasets. For example, with 8-shot prompting, PaLM solves 58% of the problems in GSM8K, a benchmark of thousands of challenging grade school level math questions, outperforming the prior top score of 55% achieved by fine-tuning the GPT-3 175B model with a training set of 7500 problems and combining it with an external calculator and verifier.

This new score is especially interesting, as it approaches the 60% average of problems solved by 9-12 year olds, who are the target audience for the question set. We suspect that separate encoding of digits in the PaLM vocabulary helps enable these performance improvements.

These claims in Google’s research updates match the sentiment of former Google employees who have interacted with internal previews of Google’s state-of-the-art AI Chatbot LaMDA. They believe it to be a credible threat to ChatGPT. But without public access to PaLM, we are left to take Google’s word for it when it comes to their model outperforming GTP-3… Or are we? 🤔

Which AI model has more common sense: PaLM or ChatGPT?

Google has shared many examples of prompts and response pairs in their PaLM research blog posts, so I decided to try their examples on ChatGPT and make a side-by-side comparison of how PaLM performs versus ChatGPT, which is based on GTP-3.5. It’s worth noting that GPT-3.5 has the same number of parameters as GPT-3 but is trained to be better at following instructions using a process called reinforcement learning from human feedback (RLHF)8.

Example 1 - Counterfactual Reasoning

The first example is a test of how well each model can reason about an outcome given a known fact:

In this example, we can see that even PaLM (left) provides a more direct answer, but ChatGPT(right) can reason about each option. This might reflect ChatGPT being a version of GTP-3 that has been fine-tuned with human feedback, with this response being preferred by the human testers. When I asked ChatGPT to choose one option, it picked option 3, which is correct!

Winner = PaLM slightly beats ChatGPT for getting straight to the answer

Example 2 - Cause and Effect

Next is a simple cause-and-effect example to see how each model understands how one event affects another:

Here we can see that both PaLM (left) and ChatGPT (right) perform fairly equally, with ChatGPT providing a bit more explanation about why it picked option (2).

Winner = ChatGPT beats PaLM for providing a better explanation.

Example 3 - Emoji guessing game

Now, let’s try this fun example Google included where each model has to guess the movie based on some emoji clues!

We can see that ChatGPT(right) performed just as well as PaLM (left) in deciphering the answer, with ChatGPT providing a more helpful explanation.

Winner = ChatGPT again for providing a better explanation

Example 4 - Chain of thought prompting

Finally, in this example, we are testing chain-of-though prompting, to see how well each model can solve a multi-step arithmetic question:

Surprisingly ChatGPT performed just as well as PaLM in this example, even though there was a lot of early criticism that ChatGPT was bad at math9!

Winner = It’s a draw!

Does Google have the upper hand in Language Models?

Based on these four examples I’m not convinced that Google's language models will outperform OpenAI's. In fact, I would say the opposite is true and ChatGPT’s answers are more helpful, which I presume is because they used reinforcement learning from human feedback to tune the model’s responses. It seems that with ChatGPT and GPT-3.5, OpenAI was able to build a model with equivalent performance to Google’s PaLM despite having 3X less parameters.

This leads me to conclude that in large-scale language models, Google’s bet on building bigger models with more parameters may not give them the winning advantage they think. Now, it’s possible that Google is applying reinforcement learning to PaLM as we speak and the next version of their model will far outperform ChatGPT, but we also know that OpenAI are going to be releasing GPT-4 any day now, so the race is on. What is clear is that bigger doesn’t necessarily mean better for language models.

2. Generative Images

When it comes to Image generation, researchers have taken a variety of approaches in the last decade, including Generative Adversarial Networks (GANs)10, Diffusion Models11, Pixel Recurrent Neural Networks (PixelRNNs)12 and Image Transformer13 by Google Brain researchers which made use of the aforementioned Transformer model for Image generation. As Jeff Dean points out, these approaches had a limitation:

Until relatively recently, all of these image generation techniques were capable of generating images that are relatively low quality compared to real world images. However, several recent advances have opened the door for much better image generation performance. One is Contrastic Language-Image Pre-training (CLIP), a pre-training approach for jointly training an image encoder and a text decoder to predict [image, text] pairs.

It’s worth noting that CLIP14 was not discovered by Google’s researchers but by OpenAI. Google's research team has been working on two models: Imagen and Parti. Imagen is a Diffusion Model15, similar to OpenAI's DALL-E 216, with 2 billion parameters. In their published findings17, Google's researchers found that the images produced by Imagen were preferable to humans compared to DALL-E 2 and Latent Diffusion Models (e.g. Stable Diffusion18):

With DrawBench, we compare Imagen with recent methods including VQ-GAN+CLIP, Latent Diffusion Models, and DALL-E 2, and find that human raters prefer Imagen over other models in side-by-side comparisons, both in terms of sample quality and image-text alignment.

You can see how Google’s Imagen model performed against Dall-E 2 and Stable Diffusion when rated by humans here:

Based on Google’s findings, Imagen could be a winning model that puts them ahead of the competition in Generative AI. Once again, I decided to do my own side-by-side to see how the Imagen compares to the three most popular models available right now, Dall-E 2, Stable Diffusion, and MidJourney.

Who can imagine better: Imagen, DALL-E 2, Stable Diffusion or MidJourney?

As with the PaLM comparison, I used Google’s published Imagen prompts and outputs and compared them to what I got by entering the same prompts into DALL-E 2, Stable Diffusion, and Midjourney. For DALL-E 2 and Stable Diffusion (v2.1), I used PlaygroundAI as the editor, but for MidJourney, the only option is to use their Discord channel. For each model, I generated 4 images per prompt and picked the best one.

Example 1 - Chocolate Eagle (easy)

Prompt: “A bald eagle made of chocolate powder, mango, and whipped cream.”

Output:

This was a straightforward prompt, yet it’s interesting to see the different interpretations. In a baking competition, Imagen’s output would probably be the winner, but it is missing whipped cream, which Dall-E’s includes. MidJourney’s output on the other hand, looks like a food sculpture but the cream and mango are less convincing. Imagen, Dall-E, and MidJourney’s output are the most photo-realistic, while Stable Diffusion’s output falls in an uncanny valley of being not quite real looking but also not drawn.

Winner = Draw between Imagen for looks and Dall-E for accuracy

Example 2 - Dog looking at a cat (medium)

Prompt: “A dog looking curiously in the mirror, seeing a cat.”

Output:

This prompt has more complexity as we expect to see both a dog and a cat in the image. Imagen and Dall-E achieve this, while Stable Diffusion and MidJourney only show two dogs. The one additional detail that impressed me about Imagen’s output is that the cat is staring back at the dog. Was this a subtle intention of the prompt?

Winner = Imagen

Example 3 - The Pomeranian (hard)

Prompt: “A Pomeranian is sitting on the Kings throne wearing a crown. Two tiger soldiers are standing next to the throne.”

This was the hardest prompt I tried as it had multiple features: two types of animals, a specific environmental setting, and the relative positioning of characters. Imagen’s output clearly outdoes the competition here, with none of the other models able to create the tigers, though MidJourney did have some tiger-like animals!

Winner = Imagen

Does Imagen have the best imagination?

Imagen seems to perform equally or better than Dall-E 2 consistently and is far better than Stable Diffusion and MidJourney in the examples I tried, especially with more complex prompts. This is probably due to the approaches Google used in building the model, including using a generic large language model pre-trained only on text vs CLIP which is pre-trained on image-text pairs. Google found that making this language model bigger was more effective than making the Image Diffusion model bigger. Strategically it makes sense that Google might go in a direction that doesn’t incorporate OpenAI’s CLIP model, but instead take advantage of its ability to build very large language models.

It’s worth noting that Google has also created a yet-to-be-released editor, Imagen Editor that allows users to edit an image using text prompts, based on their earlier work on DreamBooth19. This capability is already available today however with products like PlaygroundAI, which recently added editing capabilities:

3. Generative Video

Google shared that one of it’s next big areas of focus for AI research is Generative Video which is more complex because of the time component, as Jeff Dean shared:

One of the next research challenges we are tackling is to create generative models for video that can produce high resolution, high quality, temporally consistent videos with a high level of controllability. This is a very challenging area because unlike images, where the challenge was to match the desired properties of the image with the generated pixels, with video there is the added dimension of time.

In video generation, Google’s latest models are Imagen Video20 and Phenaki21. Both models are text-to-video, but Imagen Video uses Diffusion Models, whereas Phenaki uses Transformers.

At a high level, Imagen Video works by using T5, a text-to-text Transformer by Google, to embed22 the input prompt say, "A cat floating through space" into numerical data. This numerical data is then used to condition a diffusion model23 to generate a video consistent with the prompt. This process is similar to image diffusion but uses a video diffusion model that Google published in June 2022. The initial output video is more like a storyboard of the final video, with just 16 frames of video at 3 frames per second verses 24 frames per second for a film. Two more models are then used to fill the video with additional frames and increase the overall resolution. The resulting videos are limited to about 5 seconds long:

Phenaki, Google’s second generative video model, uses transformers to compress video to smaller bite-size pieces called “tokens” in machine learning parlance. These tokens could be a scene, a shot, or some action happening in the video, and they are compressed to be a more abstract representation than the raw video, making it easier for a model to process. A bi-directional transformer24 is then used to generate these video tokens based on a text description. The video tokens are then converted back into an actual video. According to Google, the model can generate variable-length videos, which makes it ideal for storytelling:

Google believes that both models can be used in combination to generate high-resolution long-form videos:

It is possible to combine the Imagen Video and Phenaki models to benefit from both the high-resolution individual frames from Imagen and the long-form videos from Phenaki. The most straightforward way to do this is to use Imagen Video to handle super-resolution of short video segments, while relying on the auto-regressive Phenaki model to generate the long-timescale video information.

Text-to-video generation is still in its nascent stages, and Google’s research does seem to be ahead further along than anyone else in the AI field, which makes it difficult to make any side-by-side comparisons. Meta did recently release their text-to-4d model MAV3D25, which you could imagine being used for Pixar-style animated video, but it doesn’t have the same photo realistic quality that Imagen Video or Phenaki do:

A group of researchers from the University of Singapore also recently published Tune-A-Video26, a text-to-video model that, unlike its predecessors, is only trained on one text-video example to learn a particular scenario (e.g. A man surfing a wave) rather than a large-scale dataset of text-video pairs. Once the model is trained on a video of “A man skiing on snow,” it can then produce variations of “a panda” or “an astronaut on the moon”:

Will Google win in Video Generation?

It’s still too early to tell who will be the winner in text-to-video with state-of-the-art models still in development and all published research in very early stages. It’s entirely possible that Google could have more coming down the pipeline in this domain, and it certainly wouldn’t be a surprise if they came out with a more advanced model this year, given they own the largest dataset of videos on the planet, YouTube. To me, video generation seems like Google’s race to lose.

4. Generative Audio / Music

One of the areas I’m personally most excited to see advancements in is generative audio and music. Last week Google made a massive leap with MusicLM27, a text-to-music model that can generate full-length songs, musical instrument sounds and soundscapes based on a story. Check out the video below for many impressive examples of MusicLM in action from Google’s research website:

MusicLM also builds upon AudioLM, a previous research project from Google that can continue generating audio based on some input audio. It builds on this project by adding the ability to generate audio from text input, on melodic input, and expands beyond piano to many different musical styles like drum N bass.

One of the main challenges of developing a generative music model is the lack of high-quality training data for audio and text pairs. For example, think of your average pop song like “Get Lucky” by Daft Punk, where the title doesn’t provide any description of the actual musical content of the song, its melody, or its instruments. Similarly it’s hard to build these datasets because describing a soundscape (e.g. the sounds in a busy train station) is fairly subjective.

MusicLM works in a novel way by using a previously made embedding MuLan28, which can map the similarity of text to music and vice-versa. Using this embedding, MusicLM doesn’t need to be trained on (music, text) pairs and instead can just be trained on music, then use the MuLan embedding for text conditioning.

Before MusicLM, the closest we had come to text-to-music was Riffusion, a project by a couple of AI engineers that could use text-to-image diffusion to generate audio by first converting audio into spectrogram images. Still, the audio quality was much lower than Google’s ML due to the limitations of how much data you can encode in spectrograms.

Since Google published their research less than a week ago however, four(!) more generative music projects have published updates, including another Google one:

1. Make an Audio by ByteDance AI Lab is an audio Diffusion model that can do both text-to-audio and video-to-audio.

   Make-An-Audio’s ability to create arbitrary sounds versus just music is impressive, though there’s something uncanny about the output, almost like it’s from a decades-old record.

2. Noise2Music: 30-second music clips from text prompts again using diffusion, this time from an anonymous research team!

   Despite Noise2Music’s 30-second limitation, the fidelity of the audio produced is close to that of MusicLM and much better than Make-An-Audio.

3. Moûsai: Also another diffusion-based text-to-music generator using Long-Context Latent Diffusion29:

   Moûsai’s audio accuracy to the original text is impressive though the audio quality is not as high as MusicLM.

4. SingSong: Another AI music project from Google, which can provide rhythmically and instrumentally complimentary accompanying music for vocals.

   It’s easy to imagine how this model in particular, could be applied to a fun consumer app that lets users do acapella and instantly generate a great soundtrack!

Is everyone going to be dancing to Google’s music?

The fact that five advancements were published in music generation in a matter of one week is a sign that we will see a lot of progress in this space over the next few months. I’ll be watching the space closely to see how it evolves, and especially how audio quality improves. MusicLM, with the highest sounding audio quality, is still at about half that of CD quality.

I think Google has no more advantage than any other research team in a text-to-music generation. There will be many models for anyone building products in the generative music space. I expect Google to focus on providing generative music in their creator tools for Youtube.

How did Google’s AI models stack up?

The goal of this post was to understand better if Google was really playing catch-up in AI or whether they were ahead of the competition, but just weren’t willing to take the risk to make their AI models available to the public, as OpenAI and others have.

Here’s what I found from the comparisons I did of Google’s latest AI models versus those from OpenAI and other researchers:

• Language Models: Google’s latest language model PaLM with 3X more parameters than OpenAI’s GPT3 has equivalent performance in answering common-sense questions compared to ChatGPT. This is because of OpenAI’s approach of using reinforcement learning with human feedback to improve GPT3 for ChatGPT’s conversational use case. This shows that in language models, Google’s approach of bigger isn’t necessarily better.

• Image Generation: Google’s Imagen is impressive and beats the competition when it comes to harder prompts with different features, something Dall-E, Stable Diffusion, and Midjourney really struggled with. In this case, Google leaning on it’s large language models to encode text descriptions rather than using OpenAI’s CLIP encoding puts it at an advantage. OpenAI may be able to catch up here, but it will be much harder for Stable Diffusion and Midjourney.

• Video Generation: Google’s Imagen Video and Phenaki are at the bleeding edge regarding text-to-video generation but we’re still very early in this field. Owning Youtube’s corpus of billions of videos however, makes video generation Google’s race to lose.

• Music Generation: Google’s MusicLM and SingSong are promising advances in text-to-music, but with other researchers also publishing comparable models in the last few weeks it’s still anyone’s dance floor!

When comparing Google’s latest research against state-of-the-art models, I was surprised that Google didn’t have the edge I expected. For a company that has been AI-first for many years now and has undoubtedly invested the most in advancing AI research and acquiring AI talent this comes as a surprise. Now, it’s still possible Google has much more impressive models up its sleeve beyond what they have published in research and blogged about. For language models in particular, I have to believe this is the case given the threat of OpenAI and Microsoft’s new partnership.

For Google to truly show its leadership in AI again, they need to take more risks and bring products to market that truly showcase the progress they’ve made in the field. There’s no shortage of excitement about AI right now, but by May when Google plans to announce its new products, it may already be too late to capture the attention of early adopters. Given all the resources and investments Google have put into AI over the last decade, it would be a massive loss for them to come out with products that aren’t at the cutting edge.

On the plus side, for startups entering AI, Google’s lack of dominance is a promising sign that there is still lots of opportunity to compete against the tech giant. The race therefore, has only just begun!

Did you enjoy reading this update from The Hitchhiker’s Guide to AI? If you did, please let me know by subscribing to this free newsletter!

Thanks!

~AJ

3

Machine learning models consist of many layers of mathematical calculations that apply weights, also known as parameters, to a numerical representation of the model’s input to predict a desired output accurately. Modern deep-learning neural models have billions of parameters, as you can see in this handy table of state-of-the-art models:

To learn more about how weights work in neural networks, read part 1 of my Deep Dive on Deep Learning.

Hi readers,

Here are my highlights from the last week in AI!

P.S. Don’t forget to hit subscribe if you’re new to AI and want to learn more about the space.

Subscribe now

Highlights

1/ Google makes leap in Generative Music

One of the areas that has most excited me about AI is its ability to democratize the creative process. As a musician myself, when I first started playing with generative AI products like Dall-E, my immediate thought was “This would be amazing for music”.

There have been a few different projects attempting to achieve generative music, including HarmonyAI which is able to generate new music that sounds like the input music and Riffusion which does short text-to-audio using Stable Diffusion by turning audio into images. OpenAI also published a paper on a model they call JukeBox that generates music in particular genres and styles.

In my opinion though, the holy grail is for a user to describe any kind of music or sound and for a model to generate, and it looks like Google just achieved this with MusicML!

Check out their research website where they shared lots of examples of MusicML in action, including longer songs, audio journeys with multiple parts, turning paintings into music and even generating specific instrument sounds! As of yet, there’s no tools for you to try out MusicML with your own prompts but here’s hoping that this research will be available in a product by Google later this year.

2/ Using a Large-scale Language Model as a backend

Last weekend ScaleAI1 hosted an AI hackathon in San Francisco. The winning team’s project “GPT is all your need for backend”2, might pique the curiosity of any engineers reading this post, as they were able to show how a large-scale language model, in this case GPT, could be used instead of a traditional database and server-based backend3:

Here’s how one of the team members described what they were aiming for:

What was so impressive about what the team achieved is that they were able to completely remove the need for a server or database to store data for their example application, a To Do app. Instead they just taught GPT4, what app they were building and how it should respond to requests, as well as providing examples of the type of data the frontend part of the To Do app might request e.g. a list of to do items. Once, this is done, the frontend can just describe the functions it wants to call, without them ever being defined!

Here’s a more detailed description of how “backend-GPT” works from their Github Repository:

We basically used GPT to handle all the backend logic for a todo-list app. We represented the state of the app as a json with some prepopulated entries which helped define the schema. Then we pass the prompt, the current state, and some user-inputted instruction/API call in and extract a response to the client + the new state. So the idea is that instead of writing backend routes, the LLM can handle all the basic CRUD logic for a simple app so instead of writing specific routes, you can input commands like add_five_housework_todos() or delete_last_two_todos() or sort_todos_alphabetically() . It tends to work better when the commands are expressed as functions/pseudo function calls but natural language instructions like delete last todos also work.

I’ve discussed in previous posts about the concept of emergent behavior, whereby a language model which is trained on a large enough dataset is able to carry out tasks and perform logic that is unexpected. This idea of a large-scale language model acting as as general purpose backend is a great example of emergent behavior!

3/ Atomic AI raises $35M to use AI for RNA-based drug discovery

With all the hype around chatbots and generative art, it’s great to also hear that AI companies are being created to save lives too. One such company is Atomic AI, a biotech startup that raised $35M in series A funding to do generative AI-based drug discovery focused on RNA molecules. Here’s how Raphael Townshend, CEO of Atomic AI describes the opportunity his startup is going after in an interview with TechCrunch:

“There’s this central dogma that DNA goes to RNA, which goes to proteins. But it’s emerged in recent years that it does much more than just encode information,… If you look at the human genome, about 2% becomes protein at some point. But 80 percent becomes RNA. And it’s doing… who knows what? It’s vastly underexplored.”

Check out Michael Spencer’s post for more on Atomic AI and the intersection of AI and biotech:

AI Supremacy

What is Atomic AI?

Hey Everyone, I really like covering A.I. startups at the intersection of computational biology, biotech, genomics, and drug development. A.I. is increasingly becoming implicated in mRNA and RNA medicines. This week, Atomic AI, a biotechnology company fusing cutting-edge machine learning with state-of-the-art structural biology to unlock RNA drug discov…

Read more

4 years ago · 9 likes · Michael Spencer

4/ Yann LeCun throws shade on ChatGPT!

The legendary AI researcher Yann LeCun, who was one of a few researchers pushing forward advancements in deep learning during the 70s-90s5 tweeted that he thought ChatGPT was overhyped:

I think Yann might be overestimating the general public’s understanding of deep learning, AI and the progress we’ve made in the last few decades. Until ChatGPT, most people simple had not experience AI in a tangible and impressive product, as I shared in AI: Don’t believe the hype?:

Unlike it’s predecessors (e.g. Google Assistant, Echo, Siri), ChatGPT is really the first time an AI assistant truly seems like it could pass the Turing Test. There have been many impressive examples of ChatGPT in action and if you haven’t tried it yourself you should. ChatGPT successfully wrote a blog post for me and turned it into a twitter thread, gave me a recipe for pancakes that tasted delicious and helped me pick a Christmas present for my wife!

OpenAI are capturing attention not because of the sophistication of their models but because they are shipping great products, as pointed out by Dr. Jim Fan, an AI scientist who previously worked at OpenAI and Google:

It’s also hard not to take Yann’s sentiment with a grain of salt given that he leads AI research at Meta. Maybe Yann should spend less time throwing shade and more time persuading Zuck to burn the virtual boats and join the AI race?

Or, maybe we should all just be friends and work on this together…

5/ Family guy and generative AI

Wrapping up with this fun take on what Family Guy might have looked like as an 80s live action sitcom using images created with Midjourney!

Everything else…

Finally, in case you missed it, I also shared Part 3 of my series on the origins of Deep Learning:

The Hitchhikers Guide to AI

🤓A Deep Dive into Deep Learning: Part 3

Hi Readers! Thank you for subscribing to my newsletter. Here’s the final part of my deep dive into the origins of deep learning. In case you missed it, here are Part 1 and Part 2. The field of deep learning is filled with lots of jargon. When you see the 🤓 emoji, that’s where I go a…

Read more

4 years ago · 1 like

That’s all for this week!

Hi Readers!

Thank you for subscribing to my newsletter. Here’s the final part of my deep dive into the origins of deep learning. In case you missed it, here are Part 1 and Part 2.

The field of deep learning is filled with lots of jargon. When you see the 🤓 emoji, that’s where I go a layer deeper into foundational concepts and try to decipher the jargon.

P.S. Don’t forget to hit subscribe if you want to receive more AI content like this!

Subscribe now

Part 3 - The AI Spring

We ended Part 2 at the turn of the century with a small group of dedicated researchers pushing forward the field of neural networks, despite the skepticism of the wider academic community. Meanwhile, limitations in computing power at the time made it harder to train larger models to solve more complex problems.

So how did we get from the AI winter of the 70s, 80s, and 90s to the exponential advancements we’re experiencing today? Let’s pick things up in the 2000s, where we rejoin Geoffrey Hinton and his team…

2000s: Deep Learning Accelerates

In the 2000s, Hinton and his team continue to make major advancements in deep learning, building on the foundation laid in the 1990s. They develop new training algorithms that improve the effectiveness of deep learning models, particularly Deep Neural Networks (DNNs). Their most notable contributions include "pre-training" and "fine-tuning," which they first successfully use in creating Deep Belief Networks (DBNs) in 20061.

Hinton and his team propose a way to train neural networks by starting with simpler networks first and then building on top of them. This method, called "pre-training," helps the deeper networks learn more effectively. After pre-training, the team fine-tuned the network by adjusting it using a method called "supervised learning." This helped the network improve its performance even further. Deep Belief Networks were one of the first successful examples of deep learning architectures that used pre-training, fine-tuning and unsupervised learning.

🤓 What is a Deep Belief Network?

Deep Belief Networks (DBNs) are a type of neural network trained using a special technique called unsupervised learning. This means the network is not given specific answers or expected outputs to learn from. Instead, the goal of a DBN is to find patterns and structures in the data on its own.

One example of how this works is in image recognition. Imagine we have a dataset of images of handwritten digits, and we want to train a DBN to recognize these digits. The visible layer of the DBN would represent the pixels of the images, and the hidden units would be used to identify the underlying structure of the images. For example, the hidden units might learn to recognize edges, corners, and other simple shapes in the images. These simple shapes can then be combined to form more complex shapes such as digits.

However, DBNs are different from both Deep Neutral Networks (DNNs) and Convolutional Neural Networks (CNNs) covered in Part 2. DNNs use hidden layers to extract abstract and higher-level features from the input data, while CNNs use filters to identify specific features in images. DBNs, on the other hand, are unsupervised and learn the actual structure of the image that can later be combined to generate new images.

A key component of DBNs is the use of Restricted Boltzmann Machines (RBMs). RBMs are a type of neural network made up of two layers: a visible layer and a hidden layer. The visible layer represents the input data and the hidden layer is used to learn a compressed representation of the input data. RBMs are probabilistic generative models, meaning they can learn the probability distribution of the input data and generate new samples that are similar to the training data.

In a DBN, multiple RBMs are stacked on top of one another. Each RBM learns a more abstract and higher-level representation of the data as it progresses through the network. This is similar to how a puzzle has many pieces that need to be put together to form a complete picture. The idea is that by stacking multiple RBMs, the DBN can learn a more detailed and accurate representation of the input data.

Once the DBN has been pre-trained using these stacked RBMs, it can then be fine-tuned using a labeled dataset and supervised learning techniques, such as backpropagation, to improve its performance on a specific task. This is similar to how a detective would use clues from a crime scene to make an arrest.

Overall, Deep Believe Networks have been used in a variety of applications, such as image and speech recognition, and have played a significant role in the advancement of deep learning.

2000s continued: Off to the races!

As the decade proceeds, advancements in deep neural networks and the availability of large-scale datasets like Fei-Fei Li’s ImageNet2 and pre-trained models makes deep learning much more approachable. Researchers no longer need expensive and time-consuming data collection and annotation to train effective deep learning models. This leads to an era of competitive research, with teams from around the world seeing who trains the most accurate models.

One of these competitions is the Netflix Prize. The aim of the competition is to use machine learning to beat Netflix's own recommendation software's accuracy in predicting a user's rating for a film given their ratings for previous films by at least 10%. The prize is won in 2009 by BellKor's Pragmatic Chaos, a team whose members include employees of AT&T and Yahoo.

At the same time, another development comes along that literally accelerates the field of deep learning: The use of Graphics Processing Units (GPUs) in deep learning makes it possible to train large neural networks much more quickly than when using traditional processors on a computer.

🤓 Why do GPUs accelerate deep learning?

A GPU, or Graphics Processing Unit, is a type of computer chip that is specifically designed to process large amounts of data quickly and efficiently. They were originally created for use in video game graphics, but scientists and researchers soon realized that they could be used to accelerate many other types of computations, including those used in deep learning models.

Deep learning models, such as neural networks, typically involve a lot of complex mathematical calculations and require a lot of data to be processed at the same time. Training these models can take a long time on a regular computer, even with a fast processor.

A CPU (Central Processing Unit) is divided into multiple cores so that they can take on multiple different tasks at the same time (e.g., browsing the internet while listening to Spotify). A GPU, on the other hand has hundreds and thousands of cores, which are dedicated to completing simple computations that are performed more frequently and independently of each other i.e. in parallel.

Training neural networks is an ideal task for GPUs. Calculating weights and activation functions of each layer and backpropagation can all be computed in parallel. A GPU is designed to handle these types of calculations much more efficiently. It can perform many calculations in parallel, which means it can work on many pieces of data at the same time. This allows deep learning models to be trained much faster on a GPU than on a regular computer.

As deep learning has progressed, it needs to have vast amounts of data to be fed and training on these data set takes a long time on a single CPU. With a single GPU, the time to train these models is significantly reduced, and with multiple GPUs working together in parallel, it's even faster.

2010s: “Hey Google…”

Thanks to the progress of the previous decade, the 2010s see exponential advancement in deep learning, with Apple, Google and Amazon all making major investments in the space ushering in the dawn of consumer-friendly AI. From automatically organizing your photos to helping you turn on the lights in your home and playing your favorite songs at the command of your voice, AI enters the mainstream and consumers lives.

The decade began with a seminal paper in 2012 by Geoffrey Hinton, Alex Krizhevsky and Ilya Sutskever that showed a massive leap in the accuracy of image recognition using deep neural networks3. Hinton and his colleagues developed AlexNet, a convolutional neural network that won several competitions.

In 2013 the film “Her” is released. A science fiction drama starring Scarlett Johansson as Samantha, an AI operating system who its user Theodore falls in love with. Samantha is portrayed as a highly intelligent and empathetic AI that is able to form a deep connection with Theodore. The film provides a very tangible sense of what the near future might look like with AI as part of our lives. At the same time, it also highlights the limitations of “AI Assistants” like Google Home, Siri, and Echo, which are far more limited in their ability.

🤓 How do voice assistants like Google Home, Siri and Echo work?

Remember from Part 2 how recurrent neural networks (RNNs) with long short-term memory are able to process longer sequences of input? That’s perfect for the use case of speech recognition in voice assistants, where a user makes a request with multiple words in a sequence.

Here’s a reminder of how RNNs work from Part 2:

An RNN might take a sequence of words as input and use the output from processing the previous word to inform its processing of the current word. This allows the RNN to capture the context and dependencies between words, which is important for understanding the meaning of the text.

RNNs work by using a feedback loop, where the output of a previous step is fed back into the network as input for the next step. This allows the network to "remember" what it has heard in the past and use that information to better understand the current input.

And here’s how LSTMs work, also from Part 2:

A long short-term memory (LSTM) unit is a type of recurrent neural network that is able to capture long-term dependencies in time series data. It is called "long short-term memory" because it is able to remember information for long periods of time and use it to make predictions or decisions later. For example, storing a whole sentence in a text to predict the next word vs just storing the last word.

LSTMs are a type of RNN that are particularly well-suited for handling speech data. They use a special structure called a memory cell, which can retain information for long periods of time and selectively choose which information to discard and which to keep. This allows LSTMs to effectively filter out irrelevant information, such as background noise, and focus on the important parts of the speech input, which is critical in the use case of Voice Assistants.

Together, RNNs and LSTMs form the backbone of AI voice assistants, allowing them to understand and respond to human speech in real-time. As more data is fed into these networks, they continue to learn and improve, making them even better at understanding and responding to human speech.

It is worth mentioning that, these models are trained on a massive amount of data, this allows them to generalize well and adapt to different accents, dialects, and speaking styles. It also allows them to understand and respond to new words and phrases, as well as recognize and respond to specific speakers.

But voice assistants aren’t just tasked with recognizing speech. They also respond to a user’s commands by synthesizing human-like speech too. For example, if you ask Google Home, “What’s the weather like today?” It will respond with an answer describing the current weather conditions and forecast for the rest of the day. This is where deep neural networks (DNNs) and convolutional neural networks (CNNs) come into play.

Here’s a reminder of how convolutional neural networks work from Part 2:

A CNN consists of multiple layers of interconnected neurons, which process and analyze the input data. The layers of a CNN are organized in such a way that they learn increasingly complex features of the input data as the data passes through the network.

One key feature of CNNs is the use of "convolutional" layers, which are designed to automatically learn and extract features from the input data. These layers apply a set of filters to the input data and use the resulting output to detect patterns or features in the data. This process is repeated multiple times, allowing the CNN to learn increasingly complex features of the input data as it passes through the network.

In speech synthesis, DNNs and CNNs are used to model the complex relationships between audio signals and human speech. These models are trained on large amounts of speech data, and they learn to recognize patterns in the data that are associated with different speech sounds, such as phonemes and words. They are then used to generate new speech from text by predicting the most likely sequence of speech sounds for a given input text.

DNNs and CNNs are powerful tools for speech synthesis, as they can learn to model the complex relationships between audio signals and human speech, and generate new speech from text with high accuracy. They are the foundation of most of the AI voice assistants like Siri and Google home.

It’s worth noting however, that voice assistants are limited in their ability because they are based on a set of predefined rules and commands. They are not truly intelligent because they do not have the ability to learn and adapt like a human would. They rely on a set of programmed responses and cannot understand context or make decisions based on new information.

2010s continued: The race to general intelligence begins

In the 2010s, startups like OpenAI and Deepmind (acquired by Google in 2015 for over $400M) are funded with the explicit goal of achieving Artificial General Intelligence (AGI). AGI is a type of artificial intelligence designed to possess a broad range of cognitive abilities similar to those of a human being. This includes the ability to understand complex concepts, reason, plan, solve problems, and learn from experience similar to our idea of AI from sci-fi movies like “Her”. This new injection of capital into AI research from startups and big tech companies spurs an exponential increase in advancements in the field.

One of these new developments is generative models. These are models that are designed to generate new output data rather than classify or recognize input data. One of the most popular early generative models is the Generative Adversarial Network (GAN) which was introduced by Ian Goodfellow in 20144. The main advantage of GANs is the ability to generate new data that is similar to the training data, allowing them to be used for tasks such as image synthesis, image-to-image translation, and other generative tasks.

🤓 What is a Generative Adversarial Network (GAN)?

Generative Adversarial Networks (GANs) are a class of generative models that use a technique called adversarial training, where two neural networks, a generator, and a discriminator, are trained together.

The generator network learns to generate new data samples that are similar to the training data. It takes in a random noise as input, and it produces a new data sample that is similar to the training data. The generator is typically a neural network with an architecture designed to produce new data samples, such as a decoder network.

The discriminator network, on the other hand, takes in both real data samples from the training set and fake data samples generated by the generator. It is trained to distinguish between the real data samples and the fake data samples generated by the generator. The discriminator is also typically a neural network but with an architecture designed to distinguish between real and fake data samples.

The training process of a GAN is an adversarial process where the generator and discriminator are trained simultaneously and in opposition. The generator is trained to produce data samples that can fool the discriminator into thinking they are real, while the discriminator is trained to correctly identify the real data samples from the fake ones generated by the generator.

At the beginning of the training, the generator produces poor-quality samples, the discriminator easily recognizes them as fake, and thus the generator is updated to improve its performance. As the training progresses, the generator improves and generates better-quality samples, making it harder for the discriminator to distinguish between real and fake data. The discriminator also improves by training on both real and fake samples. The training continues until the generator can produce samples that can fool the discriminator.

This process is called adversarial training; the generator and discriminator are competing with each other, and the generator is trying to produce realistic samples that the discriminator would not be able to tell apart from the real ones. This leads to the generator learning to produce samples that are similar to real ones.

2010s continued: Transformers - More than meets the eye?

Arguably one of the most revolutionary breakthrough in deep learning in the 2010s is the invention of the Transformer architecture in 2017 by Google researchers in the famously titled paper “Attention is All You Need”5. In the paper, Google’s researchers propose “a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.”

The advent of the Transformer architecture has played a significant role in the development of large-scale language models (LLMs) such as OpenAI’s GPT-3, which powers ChatGTP. These Large-scale language models are essentially transformer-based architectures that are trained on large amounts of text data, and have billions of parameters, which allows them to understand the input more effectively, generate more coherent and human-like text, and generalize better on unseen examples.

Before the transformer architecture, Recurrent Neural Networks (RNNs) were the most commonly used architecture for language modeling tasks. However, RNNs are not well-suited for processing long sequences of text and have difficulty learning long-term dependencies in the input.

🤓 What is the Transformer architecture?

Remember from earlier that recurrent neural networks and convolutional neural networks store a fixed length part of their input in memory in hidden layers so that they can better predict the output. The challenge with this approach is that it requires the model to serially process its input, for example, processing a paragraph of text one word at a time. This process cannot be parallelized when training which creates challenges due to limitations in memory. It also limits how far back a model can learn about the text it is trained on for any word in the text, referred to as “lookback.”

The transformer architecture is based on the idea of allowing the model to look at the whole sequence of input rather than a fixed-length part of the input, like recurrent neural networks with long short-term memory do. Then the model focuses on certain parts of that input that are relevant to each word. It achieves this through a mechanism called self-attention, which you can liken to being able to look up different words in a dictionary.

Attention was first introduced in 2015 in the context of language translation from English to French. In the paper, the authors gave the example of translating the following English sentence

“The agreement on the European Economic Area was signed in August 1992.”

Into the French equivalent

“L’accord sur la zone économique européenne a été signé en août 1992.”

Trying to translate this sentence by going through each English word one by one wouldn’t work for many reasons: some French words are flipped, and the French language has gendered words.

Attention is a mechanism that allows the model to focus on every single word in the French input when generating the English output and pay more attention to specific words.

The model learns which words it should “attend” to from training data by processing thousands of French and English sentences.

Self-attention, unlike regular attention, doesn’t look at the attention of specific words for a given input and output, as this is limited to translations. Instead, it looks at the attention to give different words in the input for each word in the input itself. In order words, self-attention allows a neural network to understand a word in the context of words around it. This is important in NLP tasks such as language understanding, where the meaning of a word depends on the context and its relationship with other words in the sentence.

The transformer architecture also introduced the concept of multi-head attention, which enables the model to attend to multiple parts of the input simultaneously, which improves the performance of the model by allowing parallelization.

Additionally, the transformer architecture introduced the concept of position encoding, which allows the model to understand the order of the tokens in the input and make use of their relative position to better understand the meaning of the input.

Present Day: Hype builds around Generative AI

The Transformers architecture proves to be much better at handling large amounts of data and parallel processing than previous neural networks. This enables efficient training of deep, large models on GPUs, unlocking the exponential progress toward AI. Today, transformers enable the training of large-scale language models like GPT-3 that powers ChatGPT, a breakthrough product in AI, which I covered in my post, AI: Don’t believe the hype?:

Unlike it’s predecessors (e.g. Google Assistant, Echo, Siri), ChatGPT is really the first time an AI assistant truly seems like it could pass the Turing Test. There have been many impressive examples of ChatGPT in action and if you haven’t tried it yourself you should. ChatGPT successfully wrote a blog post for me and turned it into a twitter thread, gave me a recipe for pancakes that tasted delicious and helped me pick a Christmas present for my wife! What truly impressed me though is ChatGPT’s ability to be “creative”.

Today, the race is on between Google, OpenAI, and many other startups to build larger, more intelligent LLMs that may one day reach the holy grail of general intelligence.

🤓 How do large-scale language models like GTP-3 work?

A large-scale language model (LLM) is a type of deep learning model that is trained on a large dataset of text (e.g. all of the internet). LLMs predict the next sequence of text as output based on the text that they are given as input. They are used for a wide variety of tasks, such as language translation, text summarization, and generating conversational text. Open AI’s GPT-3 (General Pre-trained Transformer 3)6, the language model that powers Chat-GPT, is an example of a generative LLM that uses the Transformer architecture, enabling it to be trained on a massive text dataset of hundreds of gigabytes using 175 Billion parameters (weights assignments).

GPT-3’s size made it the largest language model at the time of its launch though it was later superseded by Google’s PaLM which boasts over 540 billion parameters. GPT-3s use of the Transformer architecture allows it ot understand and generate language in a way previous models could not. It also made use of unsupervised learning, to be trained without any specific task in mind. This allows it to be fine-tuned for a variety of different tasks, such as text completion, question answering, and language translation.

Once LLMs get to this size, they start exhibiting emergent behavior, unexpected and seemingly autonomous actions or decisions that the model makes as a result of its training on the vast amount of data it has been exposed to. This can be seen in GPT-3's ability to generate human-like text that is often difficult to distinguish from text written by a human. However, it is important to note that this behavior is not truly autonomous, as it is still determined by the patterns and connections present in the training data.

Additionally, GPT-3's may generate text that appears to have a bias or hold certain beliefs, which is a reflection of the biases and beliefs present in the training data, highlighting the need for diverse and unbiased data when training such models. These large models are also prone to “hallucinations”, the term in AI for when a model makes up an answer in a way that appears to be factual but is not. For example, it may generate a citation to an academic paper that looks real but has the incorrect authors!

Final thoughts

It wasn’t until I wrote this series that I understood how much we should be grateful for the tireless efforts of a few researchers who laid the foundations for the astonishing progress we are seeing in deep learning today. Even a decade ago, there was still skepticism about the effectiveness of neural networks, which wasn’t really overcome until companies like Google invested hundreds of millions of dollars into the field. Now we’re on the verge of having self-driving cars, AI chatbots can pass the bar exam, and anyone can create art by just describing it! And the best part?

We're just getting started!

Feedback

Hi Readers!

I really enjoyed writing this series on deep learning and learned a lot myself while doing it. I would love to know if you found this series valuable in learning about AI?

Thanks for your feedback!

~AJ

P.S. If you did find this series valuable, do me a favor and share it with your friends interested in AI!

Share

Fun fact: I used ChatGPT for a lot of the research for this series of posts. Over the course of a week I asked ChatGPT dozens of questions about the history of deep learning and the concepts behind it. To make sure the article was accurate, I asked ChatGPT to provide citations to relevant scientific papers which I’ve included in the footnotes. If you find any errors, please reach out or comment below and I will fix them!

Hi Readers,

I’m excited to share that The Hitchhiker’s Guide to AI now has 100 subscribers!

Thank you all for supporting this newsletter and following my journey into the world of Artificial Intelligence. It may seem like a small milestone, but I’m confident that if I can make this newsletter valuable to one hundred people, it will one day provide value to thousands more. As Paul Graham says1:

If you have 100 users, you need to get 10 more next week to grow 10% a week. And while 110 may not seem much better than 100, if you keep growing at 10% a week you'll be surprised how big the numbers get.

To that end, I would love to know if you have any feedback on my writing so far? Is there anything you would like me to write about in the AI space that you’re curious about? I would like you to think of me as your personal AI expert!

Please let me know if you have feedback or suggestions by commenting on this post or replying to the email!

Leave a comment

Oh, and one more thing… If you find this newsletter valuable to you already, please share it with your friends who might enjoy reading it too!

Share The Hitchhikers Guide to AI

Thank you!

~ AJ.

Here are some of the most interesting AI updates that I read in the last week.

P.S. Don’t forget to hit subscribe if you’re new to AI and want to learn more about the space.

Subscribe now

Highlights

1/ Founder of StabilityAI shares predictions for AI-filled future

Ehmad Mostaque the founder of StabilityAI, the company behind Stable Diffusion, shared an inspiring view of how AI will impact our future with Peter Diamandis, host of the Moonshots Mindset podcast. In the wide ranging discussion, Ehmad covered how AI will impact the entertainment industry, what Generative AI means for copyright and ownership and whether AI should have a moral compass.

“This is one of the biggest evolution for humanity ever!” - Ehmad Mostaque

The first 50 minutes are definitely worth listening to!

2/ Google finally shares an update on their progress on AI

Until 2022, Google was considered to be at the forefront of advancements in AI. With the launch of Stable Diffusion, Dall-E and ChatGTP however, Google’s thought leadership and research in AI took a backseat to startups like StabilityAI and OpenAI. It’s no surprise then, that after Google reportedly issued a internal “Code Red” on the need to move faster on AI, we finally got an update last week on what they’ve been up to.

Google’s blog post was authored on behalf of their whole AI research team by Google’s most senior researcher, Jeff Dean. It covered a range of topics including their latest advancements in Large Language Models, Computer Vision, Generative AI and much more.

Dean discusses the progress and advancements in language models, specifically highlighting their work on LaMDA and PaLM, which allows for safe and high-quality dialog in natural conversations. He also mentions their work on multimodal models, which can handle multiple modalities simultaneously, both as inputs and outputs.

Dean also highlights the progress and advancements in generative models for imagery, video, and audio, mentioning specific techniques such as generative adversarial networks, diffusion models, autoregressive models, and Contrastic Language-Image Pre-training (CLIP). Finally, he emphasizes the importance of responsible AI, stating that they apply their AI Principles in practice and focus on AI that is useful and benefits users and society.

In the post, Dean also highlighted multiple times that Google was responsible for the Transformer model in 2017, which unlocked the rapid increase in AI innovation we’re seeing today. He also shared that many of the innovations that Google have made in AI are already being integrated into their existing products.

You can read the whole post here but fair warning, it’s pretty dense!

3/ OpenAI CEO debunking rumors about GPT-4

Sam Altman CEO of Open AI, debunks the rumors about GPT-4 in an interview with Connie Loizos for StrictlyVC.

GTP-4 is the much anticipated iteration of GPT-3, the large language model that underlies ChatGPT. As I mentioned in my first post Don’t believe the hype?, there’s a lot of excitement and rumors around GPT-4 and some of the features, including a prediction that it will have 100X more parameters than GPT-3:

Sam’s response to this rumor was “People are begging to be disappointed and they will be”.

Here’s the full interview:

4/ Predictions on how Big Tech will approach AI

Ben Thompson shared his predictions for how Google, Amazon, Meta (Facebook), Microsoft and Apple in his recent post, AI and the Big Five. In the post Thompson view, AI is a new epoch in technology and he shared the following predictions on how the epoch might develop:

• Apple's efforts in AI have been largely proprietary, but that the company recently received a gift from the open source community in the form of the Stable Diffusion model, which is small and efficient enough to run on consumer graphics cards and even an iPhone.

• Amazon's prospects in this space will depend on factors such as the usefulness of these products in the real world and Apple's progress in building local generation techniques.

• Google, who has been a leader in using machine learning for their search and consumer-facing products, may face a similar fate in the AI world to Eastman Kodak’s infamous downfall when digital cameras became popular. The shift from a mobile-first world to an AI-first world may present challenges for Google in providing the "right answer" rather than just presenting possible answers.

• Meta's data centers are primarily for CPU compute, which is necessary for their services and deterministic ad model, but that the long-term solution for improving their ad targeting and measurement is through probabilistic models built by massive fleets of GPUs.

• Microsoft is well-positioned in the AI space with its cloud service, Azure, that sells GPU access and its exclusive partnership with OpenAI. Microsoft is investing in the infrastructure for the AI epoch through this partnership and its Bing search engine, which has the potential to gain massive market share with the incorporation of ChatGPT-like results.

You can listen to the whole analysis here too:

5/ The first copyright lawsuit against Stable Diffusion

Jon Stokes does a great analysis of the first major lawsuit alleging that Stable Diffusion violates the copyright of artists whose images were used in training. Cases like this one and the recent copyright lawsuit against Microsoft’s Github Copilot product are likely to be pivotal in establishing how Generative AI content will be viewed in the eyes of the law.

Here’s Jon’s post:

jonstokes.com

Anti-Stable Diffusion Lawsuit Deep Dive: It’s A Neutron Bomb

The first major lawsuit against Stability AI has arrived, and the different AI art channels I follow have been furiously debunking the suit’s technical claims as they’ve been published by the lawyers. That debunking is all well and good — I’ve indulged in a little of it, myself — but I’m not sure most of the community…

Read more

4 years ago · 4 likes · 1 comment · Jon Stokes

Everything else…

jonstokes.com

Machine Learning And Deflationary Contagions

Here’s the most common question I get from members of the press, analysts, and others I talk to about generative AI: can you describe what impact this will have on my industry? Of course, this is the obvious question to ask, right? So I get it all the time, and because I get it all the time I’ve developed a standard high-level answer to it that I’ll shar…

Read more

4 years ago · 17 likes · 9 comments · Jon Stokes

Ben's Bites

Conversational answer engine

Hey folks, it's Friiiiyay! 🤢 - don't be the person to ever say that. We had our first official meetup yesterday in London and I loved it. Balderton Capital hosted ~200 of us in their offices and provided fab food and drinks. A big TY to them! And a…

Read more

4 years ago · Ben Tossell

Finally, in case you missed it, I also shared Part 2 of my series on the origins of Deep Learning:

The Hitchhikers Guide to AI

🤓A Deep Dive into Deep Learning: Part 2

Hi Readers! Thank you for subscribing to my newsletter. As promised, this is Part 2 of my deep dive into the origins of deep learning. If you missed Part 1, you can read it here. The field of deep learning is filled with lots of jargon. When you see the 🤓 emoji, that’s where I go a…

Read more

4 years ago · 2 likes · AJ Asver

That’s all for this week!

Hi Readers!

Thank you for subscribing to my newsletter. As promised, this is Part 2 of my deep dive into the origins of deep learning. If you missed Part 1, you can read it here.

The field of deep learning is filled with lots of jargon. When you see the 🤓 emoji, that’s where I go a layer deeper into foundational concepts and try to decipher the jargon.

P.S. Don’t forget to hit subscribe if you want to receive more AI content like this!

Subscribe now

Part 2 - The Hidden Layers of Deep Learning

We ended Part 1 at the close of the 1970s in the AI winter. Research had slowed down because of limitation of single layer neural networks and computers aren’t yet powerful enough to train large networks.

Let’s pick things up in the 1980s…

1980s

As we learned in Part 1, single layer neural networks are limited to only being able to learn linearly separable data, which means they couldn’t learn simple mathematical functions like XOR. In theory, adding just one additional layer to a single-layer network allows it to approximate any mathematical function. In practice however, two-layer networks end up being too big and too slow to be useful.

Two-layer neural networks have a large number of parameters, which need to be adjusted during the training process. This requires a lot of computational resources, making the training process very slow, especially when working with large datasets. Two-layer neural networks also have a limited capacity to learn complex patterns and features from data. Because they only have two layers (input and output layer), they are not able to extract high-level features that can be used to represent data beyond those directly represented in the input layer. This makes it difficult to learn complex relationships and patterns.

A great example of the limitations of a two-layer network is in the task of image recognition. A two-layer neural network, would only have an input layer that receives the raw image data and an output layer that produces the final prediction or decision. The lack of additional layers in between the input and output layers limits the capacity of the network to extract high-level features such as edges, textures, and shapes from the image data.

In order to create neural networks that are more capable of learning, researchers would need to go beyond two layers. These multi-layer neural networks are also known as Deep Neural Networks (DNNs).

🤓 What is a Deep Neural Network?

A deep neural network (DNN) is a neural network with many layers, typically composed of an input layer, multiple hidden layers, and an output layer. Each layer contains a set of interconnected "neurons" that perform computations on the input data and pass the results to the next layer. The neurons in a deep neural network are connected by weights which can be learned through training. The goal of training is to adjust these weights so that the DNN produces a desired output for a given input.

What are hidden layers?

The input layer takes in the input data, which is then processed by the hidden layers. The hidden layers are called "hidden" because their internal workings are not directly observable and are not part of the network's input or output. These layers use a set of weights and biases to transform the input data, passing it through multiple non-linear processing stages known as activation functions. The output of the last hidden layer is then passed on to the output layer, which produces the final output of the network.

The purpose of the hidden layers is to extract and abstract features from the input data and pass them on to the next layer. The more hidden layers a DNN has, the more complex patterns it can learn and represent. The number of hidden layers and the number of neurons in each hidden layer is a parameter of the network called its architecture, and can be adjusted during the training process to optimize the performance of the network.

To simplify, think of a DNN as a function that can map input data to output data by passing through the layers. Each layers adapts the function to better fit the input-output pairs that it sees during the training phase, with the final output being the output of the last layer.

For example, let's say that the DNN is trained on a dataset of images of handwritten digits. During the training process, the DNN would learn the statistical patterns and relationships between the pixel values of the images and the labels indicating which digit the image represents.

Once the DNN is trained, it can be used to classify new images of handwritten digits. To classify a new image, the DNN would take the pixel values of the image as input and pass them through the layers of the network, using the patterns and relationships it learned during training to classify the image as a specific digit.

1980s continued…

In 1986, a pivotal book was published: Parallel Distributed Processing (PDP) by David Rumelhart, James McClelland and the PDP Research Group1. In the book, David Rumelhart and James L. McClelland that explores the use of artificial neural networks for computational modeling. The PDP series includes several influential papers on the development of neural networks and their applications.

One of the main contributions of the PDP series was the use of Paul Werbos’ backpropagation algorithm for training neural networks. The PDP series also introduced the concept of distributed representation, which is the idea that the meaning of a concept can be represented by the pattern of activity across multiple neurons in the network. This idea is important because it allows neural networks to learn more complex relationships between the input and output data, and to generalize better to new data.

1990s

In the 1990s, the term "deep learning" is coined by Igor Aizenberg and colleagues to describe multi-layered neural networks. The biggest breakthrough of the decade however, is when the first successful application of deep learning is demonstrated by Yann LeCun, Yoshua Bengio, and Geoffrey Hinton in their work on handwritten digit recognition using a convolutional neural network (CNN)2.

LeCun’s, Bengio and Hilton’s work is seminal because it demonstrates the effectiveness of deep learning for real-world applications. Prior to this work, there had been limited success in using neural networks for practical tasks, and many researchers were skeptical of their potential. However, the results of this work shows that deep learning can be used to achieve high accuracy on a challenging real-world task, paving the way for further research and development in the field.

Here’s LeCun demo-ing his CNN, LeNet 1 in 1993:

🤓 What is a Convolutional Neural Network?

A convolutional neural network (CNN) is a type of deep neural network that is commonly used in image and video recognition tasks. It is designed to automatically and adaptively learn spatial hierarchies of features from input data, making it particularly well-suited for image analysis.

A CNN consists of multiple layers of interconnected neurons, which process and analyze the input data. The layers of a CNN are organized in such a way that they learn increasingly complex features of the input data as the data passes through the network.

Why is it “convolutional”?

One key feature of CNNs is the use of "convolutional" layers, which are designed to automatically learn and extract features from the input data. These layers apply a set of filters to the input data and use the resulting output to detect patterns or features in the data. This process is repeated multiple times, allowing the CNN to learn increasingly complex features of the input data as it passes through the network.

Here’s a great video by Google that visualizes how a CNN works:

1990s continued…

In 1995, Jurgen Schmidhuber and his student Sepp Hochreiter publish a paper on the concept of "long short-term memory" (LSTM) units3, which are a type of Recurrent Neural Network (RNN) that are able to capture long-term dependencies in time series data.

Traditional RNNs have difficulty in capturing long-term dependencies (e.g. one word after another in a sequence of text), which can limit their effectiveness for certain tasks. To address this issue, Schmidhuber and Hochreiter introduce the concept of LSTM units, which are able to "remember" information for long periods of time and use it to make predictions or decisions later. They demonstrate the effectiveness of LSTM units for a number of tasks, including language modeling and polyphonic music modeling, and show that they outperformed traditional RNNs in these tasks.

LSTMs go on to have a significant impact on the development of deep learning models for tasks such as language modeling, machine translation, and speech recognition.

🤓 What is a Recurrent Neural Network?

A recurrent neural network (RNN) is a type of artificial neural network that is designed to process sequential data, such as text or time series data. It is called "recurrent" because it makes use of sequential information, passing the output from one step of the processing back into the network as input for the next step.

How does an RNN work?

In a recurrent neural network (RNN), the neurons are connected in a directed cycle, meaning that the output from one step in the processing is passed as input to the next step in the cycle. This allows information to be passed from one step to the next and for the network to use information from the past to inform its current and future processing. For example, predicting where an object is going based on it’s passed co-ordinates or predicting the next word in a text based on previous words for auto-competion.

For example, in a language modeling task, an RNN might take a sequence of words as input and use the output from processing the previous word to inform its processing of the current word. This allows the RNN to capture the context and dependencies between words, which is important for understanding the meaning of the text.

What about LSTMs? How do they fit in?

A long short-term memory (LSTM) unit is a type of recurrent neural network (RNN) that is able to capture long-term dependencies in time series data. It is called "long short-term memory" because it is able to remember information for long periods of time and use it to make predictions or decisions later. For example, storing a whole sentence in a text to predict the next word vs just storing the last word.

What are RNNs useful for?

RNNs are particularly well-suited for tasks such as language modeling, machine translation, and speech recognition, where the order and context of the input data is important. They have also been applied to a wide range of other tasks, including image and video analysis, music generation, and protein folding prediction.

Here’s a great article that explains RNNs in more detail: Introducing Recurrent Neural Networks.

To be continued…

We end Part 2 with deep learning making steady progress thanks to the dedicated work of a small group of researchers including Yann LeCun, Yoshua Bengio and Geoffrey Hinton. During this period, many of their academic papers were rejected by journals and conferences because of their use of neural networks, despite dramatically outperforming any previous approaches.

In 2018, LeCun, Bengio and Hinton were awarded the Turing Award, the highest honor in Computer Science, for their dedication and persistence in developing neural networks despite skepticism from the academic world.

In Part 3, we will see how the work of these researchers laid the foundations for modern day AI…

The Hitchhikers Guide to AI

🤓A Deep Dive into Deep Learning: Part 3

Hi Readers! Thank you for subscribing to my newsletter. Here’s the final part of my deep dive into the origins of deep learning. In case you missed it, here are Part 1 and Part 2. The field of deep learning is filled with lots of jargon. When you see the 🤓 emoji, that’s where I go a…

Read more

4 years ago · 1 like

Fun fact: I used ChatGPT for a lot of the research for this series of posts. Over the course of a week I asked ChatGPT dozens of questions about the history of deep learning and the concepts behind it. To make sure the article was accurate, I asked ChatGPT to provide citations to relevant scientific papers which I’ve included in the footnotes. If you find any errors, please reach out or comment below and I will fix them!

If you enjoy reading my posts and would like to receive more content about AI for builders, founders and product managers, please subscribe!