Self driving cars are one of the first “real” robots people will see. And by “real” robot, I mean something acting out in the real world, doing something that most people thought of as a human-specific skill.

Tesla is planning an invite-only launch event for its robotaxi service Sept 3, 2026. They seem to be preparing a great number of vehicles for the launch in Austin, TX. In addition, there have been sightings all over the United States, in cities like New York and Bellevue, Washington.

It’s an interesting moment, and hopefully marks the beginning of an explosion of genuine self-driving-car competition and expansion across many different cities. Time for a short blog post ahead of the Robotaxi launch.

Subscribe now

Expansion Plans

Tesla’s robotaxi service right now is only in a few cities, but it seems like at the very least they’re thinking about rapidly scaling. Here are 25 robotaxis in Tampa, Florida; here are 17 in Bellevue, Washington. Anecdotally, I saw one in Pittsburgh this weekend, joining Pittsburgh’s occasional exploratory Waymo Ojai trials.

Seeing vehicles all over the country is promising. Tesla is clearly betting that their end-to-end-learning focused strategy will allow them to roll out much more quickly and to more cities compared to Waymo.

There’s some evidence this will work. Tesla Autopilot already deploys in a lot of countries; Wayve, which has a similar learning-focused strategy, once drove through 90 cities in 90 days. The question has largely been one of reliability, but Tesla has been focused on collecting driver-in-the-loop data with its own supervised robotaxis for a while now.

The new vehicle has Tesla’s brand-new HW4 self-driving compute, with 20 CPU cores at ~2.35 GHz, and upgraded cameras. It also seems like it will have a Starlink antenna, to allow data upload and remote interventions.

The Competition with Waymo

Unsurprisingly, Waymo has gone on the offensive, talking about how multiple sensors are required for safety. This makes sense; Waymo’s vehicle is almost certainly much more expensive than the Cybercab, and Tesla is aggressively vertically integrated. Waymo will be in a tough position if Tesla actually proves its ability to out-scale the Google spinoff.

Waymo runs its successful robotaxi service in 14 cities, having recently added San Diego, Denver, and Tampa as of the time of writing. The two still operate on wildly different scales, with Tesla starting its pilot service in only five cities in Texas and Florida:

The question is rate of progress. Tesla, again, is aggressively vertically integrated; they can iterate on vehicle production, design, and software much faster than Waymo. Their stack is simpler, and they have a lot less specialized sensors.

Will Tesla run out of steam, or will their much more minimalist, economical stack allow them to drive success rates up while scaling much more aggressively than Waymo? Or will Waymo continue their steady expansion, offering a high-quality and relatively intervention-free ride experience to more and more people?

It remains to be seen. We will find out more on Sept 3rd.

Subscribe now

Share

Leave a comment

The best way to interact with your computer, clearly, is with human language. You tell it clearly what to do, and a modern coding agent can accomplish a great number of coding tasks. What’s the correct way to interact with a robot?

The problem is that language generally under-specifies any particular task. Language can tell you “put away the groceries,” but not the precise way you like all the labels arranged, or exactly where you put the milk, or the fact that you like all the jars pushed up against one side of the fridge in a particular way. Language is symbolic, but real-world, multi-modal data is generally lacking in meaningful symbols: we need to pick and choose and create them in order to transfer meaning via text across the internet. This, in large part, is the skill of a talented writer.

This leaves us a huge problem when trying to teach robots what to do. Robotics, as with most AI, works best when a problem statement is very clear, when there’s no room for interpretation, no ambiguity left for the model to guess at and potentially mess up.

We might imagine the perfect way of specifying a task, then, being a single demonstration of that task, on the same hardware, in a new context: it explains exactly how the robot should move and what it should do at each instant in an incredibly precise way (compared to text).

It’s also a way that is very intuitive for a human. We learn by watching and trying; we don’t learn manipulation by following sheets of written instructions. If you want to learn how to fix a dryer, you go to YouTube, not to Wikipedia. In short, if text and next-token-prediction was the foundation of GPT, it seems clear that video understanding is the equivalent for robotics and manipulation. Instead of prompting a robot with text, you must be able to prompt it with video.

And there have been a number of incredibly promising results this year which show how, finally, we can do just this: provide a video of a robot, and without re-training, execute a new task. Let’s take a look.

Subscribe now

Research Proves the Way

Above: video from Behavior Prompting Policy, by Austin Patel et al.

Giving a single, intuitive demonstration in order to teach a robot is a powerful idea. One of our earliest episodes of RoboPapers covered Instant Policy, work by Vitalis Vosylius and Edward Johns which won the best paper award at the ICLR Robot Learning Workshop last year. The authors trained a specifically conditioned “foundation model” in simulation to make it possible to specify many different pick and place tasks at test time.

Such work was, however, a bit limited in what types of tasks could be performed, and this year we have seen more work which shows more scalable and general in-context learning. Prof. Edward Johns and his team had more great work featured on the RoboPapers podcast:

RoboPapers

Ep#94: Learning a Thousand Tasks in a Day

For general-purpose robots to be useful, we must be able to teach them new tasks quickly and repurpose them to new roles. But most techniques for teaching robots — even from human demonstrations — take a great deal of new data. But it turns out that it is possible, in part through decomposing tasks into component subtasks, that it’s possible to learn a …

Listen now

a month ago · 6 likes · Chris Paxton

In their most recent work published in Science Robotics, the team taught a thousand tasks in a single day, using only a single demonstration per task. Another great read on this topic is the paper “Behavior Prompting Policy” (paper video above), which uses individual UMI trajectories as context to specify robot policies at test time. (UMI is Universal Manipulation Interface; an isomorphic data collection tool, or 1-to-1 copy of the robot gripper that a human can hold in their hands for fast and scalable data collection.)

These offer better specification for tasks and, importantly, allow robots to be “taught” given only a single demo trajectory. What this means, in the future, is that we can imagine the robot as a tool - one that someone who isn’t a robotics PhD or industry veteran can then use to solve their own problems. And that, to me, is the essence of a GPT moment. But for this capability to really work, it needs to scale.

Scaling Comes to Robot In-Context Learning

The most exciting robotics results of the year have come from scaling in-context learning for robotics, and these have not come from established players but from three well-funded and trendy Silicon Valley robotics startups, as well as from the phenomenal open research of Ant Group’s RobbyAnt robotics team. Generalist released GEN 1.5, a one-shot learner with results showing sim-to-real, robot-to-robot, and human-to-robot transfer. Rhoda showed human-to-video in context learning earlier in the year, based on their base model pretrained on years of egocentric and exocentric video data. And most recently, Skild went into great detail explaining their foundation model with incredibly exciting scaling results for in-context learning.

Taken together, these four results have demonstrated conclusively that it is possible to scale in-context robot learning for general manipulation tasks, and have provided some insights into what data is necessary and how to build such models.

The training recipe will look familiar in some ways; it’s a prompt, adding conditioning tokens to a transformer. But unlike in large language models, this is not a behavior that seems to arise naturally. Instead, you often need to specifically train robotics foundation models to include conditioning on previous video.

The Skild S1 version is interesting. It shares one commonality with the Generalist 1.5 model, which is that in neither case do they do data collection with purely human hands, instead using the end-effector cameras on the wrists. See the image above, a still from their release video.

But the big unique part that they push is completion of previously-unseen long horizon tasks. They show tasks like pancake flipping and plant potting which can take up to 10 minutes to execute. Particularly important is that it can do all this while moving beyond compositionality: meaning that it makes novel motions like pressing a coffee filter into a tunnel, or digging into soil in a particular way (GEN-1.5 and ACT-2 demonstrate compositionality).

Also, remember that part of why we care about in-context learning is that video is a better way of specifying tasks than language. With the Skild S1 blog post, we have some decent evidence showing this holds true at scale.

Another key result from their blog post: it took hundreds of post-training examples to reach the same level of performance. Post-training, in this case, means recording demonstrations of that particular task and fine-tuning the model — adjusting its weights — to improve success rate just on this task.

But post-training in this way is not without cost: it usually hurts generalization performance. And generalization performance, largely, is model performance: a robot needs to be able to handle the “long tail” of corner cases in order to be useful, meaning that any meaningful fine-tuning is likely going to hurt performance eventually unless it’s really, extremely comprehensive and the deployment domain is very well understood (or it’s just done in a closed loop with your final deployment, taking many extra months and thousands of data-hours).

So here we see a route out of this expensive, risky fine-tuning trap. Instead, just train bigger models on more data and prompt your way out! This is a much better solution in my mind, and will make robotics deployments far more scalable.

Training a Robot Foundation Model with ICL

As mentioned above, for in-context learning on robots to work, we want to train models which take in long-context video and potentially proprioception information, and which specifically pair this long video input (from mixed embodiments) with robot execution trajectories. This makes it easy to, say, teach your home robot how to fold a particular new shirt, as shown in the Sunday Robotics ACT-2 blog post.

Mixed observation and proprioception information go into the robot model and form the “prompt,” which specifies how the robot will act. There’s a clear analogy here to how large language models work, but based on video and proprioception information instead of text.

One interesting note will be where the data comes from. Some of the coolest, most futuristic results — such as stacking from GEN 1.5 and bin sorting or drawing from Rhoda — show tasks being executed by a human, with no special tools or gear to make the human more robot-like (as per Skild S1). This doesn’t seem to work very reliably yet, but when it does it shows a real vision of a robot that, say, your parents could use: one where you unpack it and show it something, and it “just works.”

The Lingbot VA-2 paper shows how this can be implemented. They took real robot trajectories and synthetically generated paired human video out of these trajectories, which they can then use for in-context learning on a variety of pick-and-place tasks.

On GPT Moments

The term “a GPT moment for X” is somewhat overused. To me, personally, the mark of general intelligence is when we can build tools — robots — that users will be able to deploy, which will do things that surprise the tools’ creators with their generality and capability when put in the hands of other people. In the world of LLMs, this is most evidenced by the recent trend of AI models solving open problems in frontier mathematics.

The reason that, to me, these long-context video results feel like a hint at a true GPT moment, then, is that robotics can now become a true general-purpose tool. With in-context learning on video or a single demo, I can finally imagine unboxing a robot, taking it into a new place of business, and showing it some task to get it to run within a single day. Are we there yet? No. But the road seems clear.

Final Thoughts

There are still open questions as to how best to implement in-context learning for robots. Generalist GEN 1.5, for example, does test-time training — they can improve results by taking a few gradient steps. This might be a good tool in your toolbox, if you want to improve performance. More research and development will help figure out which tools are best across a very wide range of tasks.

Eventually, this becomes a human interface problem: what tools will make robot users most successful? Which will help humans best envision the tradeoffs between policy specification choices and enable large-scale robot deployment?

One final caveat: in many of these results, the in-context learning success rates are still nowhere near production level, and many of the results are constrained to task components that have been seen before, even if zero-shot tasks are possible. And we do see signs of long-horizon unseen tasks from Skild S1. Still, the robot isn’t yet going to learn some wholly new skill, at high success rates, without more data or on-robot reinforcement learning.

All in all, I think this has been an incredibly exciting year for robotics, and it feels more obvious than ever that this technology will have a massive impact in the coming years.

If you like this, please subscribe and share the post, and leave a comment with your thoughts below.

Leave a comment

Share

Subscribe now

The World Humanoid Games are underway in Beijing, and the world is watching. Robots are sprinting, long jumping, boxing, and playing soccer. The results are, at times, comical. Sights of one robot after another slamming into a wall at the end of a race, or soccer-playing Booster robots tripping over one another into a gigantic heap, make this all seem like a bit of a joke sometimes.

But feeling this way would be a mistake. Instead, I would argue that these games show a robust, mature ecosystem which can both build things and, crucially, isn’t afraid to break them. And that’s key to real progress.

It’s staggering how much different things are from last year. The robots are faster, more intelligent, and more capable. They’re also a lot more flammable, with multiple instances of fast-moving sprinters exploding or bursting into flames due to the tremendous heat being generated.

A very rapid pace of robotic hardware and software innovation is enabling a robust ecosystem, in which many different companies and laboratories can build high-quality products, and even develop one-off robots like the Honor Lightning, a humanoid robot designed purely for sprinting and which recently beat (human) Usain Bolt’s 100 meter record.

Subscribe now

Production Scale Explodes

Unitree sold over 5,500 humanoid robots in 2025, mostly its G1 humanoids. And after an explosive IPO, it seems unlikely that this will change any time soon. There’s tremendous demand right now for capable, usable, and deployable robots — and Unitree’s humanoids are something of a gold standard for the industry.

Their secret, in larger part, has been shipping and iterating. Unitree is ruthless about punishing and testing their robots. In 2024, they were hauling the first G1 around to every academic conference, watching the robot slowly accumulate scratches and nicks, even though it could at the time barely walk. At the same time, they would bring out their quadruped robots and kick them down the stairs, showing exactly how durable and reliable they were when the robot could be piloted right back up.

They’ve been iterating quickly on their humanoids, going from what used to be a relatively useless G1 to an incredibly popular and well-oiled product, in part powered by rapid hardware cycles, aggressive vertical integration, and intense optimization.

And yet by most accounts they’re not even the biggest producer of humanoid robots. That would be Agibot:

Agibot is far less seen outside China, though robots like the Agibot X2 (their answer to the Unitree G1). US companies like Faraday Future have been selling some suspiciously similar robots to those Agibot offers. Agibot also sells a number of industrial semi-humanoids like the A2-W:

These robots are designed for industrial use cases as they’re somewhat easier to deploy in different industrial use cases. Robots like the Agibot G2 support autonomous charging and have been shown in real industrial use cases already.

Companies like Galbot on the industrial side, with its heavy-duty S1 robots, and Booster with its developer-focused K1 and T2, also provide great robots. The Galaxea R1 has established itself as a very standard manipulation research robot, starring in the outstanding Stanford Behavior-1k benchmark. In short, there’s a massive number of companies, producing a truly amazing number of robots.

And lest we think the companies mass producing mobile robots are ALL Chinese, American humanoid company Figure, clearly at this point the manufacturing leader in the United States, also built its 1000th robot in July of this year.

And multiple other humanoid companies are gearing up for increased production as well. But no one seems close to reaching the scale of hardware depth and quality as the ecosystem apparent in China right now.

Innovation Follows

Humanoid robot designs are both showing diversity, but also converging on a couple form factors around certain roles. We’ve seen a number of robots pick up a wheeled elevator base pattern seen on Sunday and Weave robots, a number of full-sized humanoids like Figure, 1X, Tesla, and the Unitree H2, and a range of smaller developer-focused robots. The biggest robot producers (like Unitree and Agibot) all have a range of different platforms now, filling different roles — though the biggest sellers are still development-focused, child-sized robots like the Unitree G1.

Hardware and software innovation is everywhere. LimX, for example, has been developing its modular TRON series robots, and Fourier has built its more care-centric GR-3 humanoids, with soft and friendly bodies.

This empowers learning research as well. Massive open datasets like the Galaxea Open World Dataset and Agibot World are invaluable for researchers, and couldn’t exist without abundant hardware. It also enables research and development of robot foundation models like Lingbot-VA-2.

It’s a robust ecosystem that will pay dividends in the future, because one of the biggest barriers in robotics is just having robots, and having robots that can do useful things! No longer is this an issue. We have a lot of robots, for every possible niche, and the real barrier now is just making them smart enough. And as we saw with LLMs, good data is key to solving that problem.

Past Competitions Drove Progress

Think back to events like the DARPA Grand Challenges of the 2000s in the United States. DARPA asked universities and researchers to build a self-driving car, as American troops were dying to roadside bombs in Afghanistan and Iraq.

The researchers gathered in the Mojave desert to drive along a 240-km route. Not a single vehicle completed the route. Next year, five vehicles out of 23 completed a new, 212-km route. In 2007, at the Urban Grand Challenge, only six out of 11 teams completed a much shorter 89-km route through George Airforce Base.

Despite the atrocious failure rate, and the poor performance by human standards even of the winners, the finalists would go on to form the core of the self-driving car boom, founding companies like Waymo and Zoox which have autonomous vehicles on the roads today.

And this is where we come back to the World Humanoid Games. Competitions like this might be filled with failures — watch this amazing compilation of falls from the DARPA robotics challenge — but they spread information, ideas, and help enable real progress, just the same.

Final Thoughts

We’re at a strange point in time for robotics. Robotic hardware — good hardware — is available in a way that has never been possible before. It’s more robust — massively so — and robots are increasingly both affordable and capable. In this environment, it’s thrilling to see stuff like the Humanoid Robot Games, which will I think inspire and drive forward a new generation of engineers and researchers who will build robots that are actually useful.

One thing that strikes me, is that the most important thing when building robots seems to be just building robots. That’s been a huge part of Unitree’s success: they build a lot of robots. If a robot breaks, they figure out why, and they make it stop breaking in that way. They build actuators because those are the bottleneck in the price of a robot. They sell robots and make money doing it — 278.21 million yuan of profit in 2025 alone — and their explosive IPO saw their company value rise to $50 billion. They’re the company building and selling general-purpose robots at scale, and the recipe works for them. But, importantly, it did not start that way: it started with a line of robot dogs and a Unitree G1 that could barely put one foot in front of the other.

So, even if right now a lot of these robots fall down and burst into flames, a massive wave of real, working robots seems like it’s just on the horizon.

Subscribe now

Share

Leave a comment

Further Reading

SemiAnalysis

Unitree's Impossible Trajectory Is Still Overlooked

We are witnessing the birth of another Chinese hardware giant. Three years ago, Unitree was a quadruped company. By last year, they parlayed quadruped dominance into creating and leading the humanoid market. This year, their G1 humanoids are finally entering into viable deployments, and three new designs are on the way, including their most-direct…

Read more

3 months ago · 312 likes · 2 comments · Reyk Knuhtsen, Niko Ciminelli, Jacob Rintamaki, Robert Ghilduta, Joe Ryu, Jeremie Eliahou Ontiveros, and Dylan Patel

There’s an increasing number of companies trying to build robots for homes — Figure, 1X Technologies, Weave, Sunday, and many more. One perennial debate that we see, again and again, is whether robots should have legs or wheels.

Personally, I am of the opinion that we’ll see some of both; that neither wheeled nor humanoid robots will dominate absolutely everything. But there are many passionate arguments on either side, from “wheels have no future” to “legs are pointless and dangerous.” Let’s go into the arguments in favor of each.

Subscribe now

First, Defining “Robot”

Your home is probably already full of robots, in a trivial way. A dishwasher or washing machine has an actuator, a motor, and sensors; it’s just stationary and not general-purpose. Your car may drive itself. You possibly have a Roomba-style robot vacuum.

None of these are what I am referring to. For the purposes of this article, a robot has these characteristics:

• It is mobile. Else why would we care about legs vs wheels at all?

• It has arms. It can pick something up.

• It is multi-purpose — it can do more than one task, like fold clothes and look out a window.

There are many varieties of robots, but we’re particularly interested in the vision of a multi-purpose home helper robot.

Legs Are More Mobile

Legs have some clear advantages. If you are on robotics and AI twitter you have likely seen many posts like this:

Basically, humans have legs, and so our environments often make strong assumptions about having legs. For example, legs allow you to go up and down stairs. They allow you to step over ground-level clutter. They let you go over uneven ground more easily while adjusting for balance.

Beyond this, legs have a few other advantages. A robot with legs can dynamically adjust its support polygon. It can take a wider stance when it needs to, bring its feet together to move through a narrow passageway, catch itself with a leg when it’s hit near the top. This means that, subject to good, reliable hardware and high-quality whole-body control model training, a robot with legs should be able to operate in a much wider range of environments and perform a wider range of tasks.

Legs have another advantage, less often discussed: redundancy. See this video by startup Light Origins: when it loses access to legs, it can hop or crawl to get around. More generally, as motors degrade, a legged robot can handle it more gracefully and not be fully disabled. This applies, too, to when the robot falls: it can stand back up, or fall intelligently, subject to robust knowledge of its environment.

… But Most Robots Have Wheels

Wheels, on the other hand, are statically stable (they will not fall over if they lose power), and they are much, much cheaper. They also are able to carry much larger payloads.

This means we tend to see three kinds of wheeled robots: extremely cheap development and “home robot” platforms, mobile manipulation platforms with relatively lightweight arms but good quality arms, and extremely high-payload “heavy industry” stuff-carrying robots with large, very heavy bases.

The lightest wheeled bases are much less capable of carrying objects, and have a harder time reacting to disturbances (see previous section). But generally you’re assuming that these robots will not be making unexpected contact with the world while manipulating stuff. Incredibly heavy robots like the Galbot S1 (right, above) are basically impossible to knock over, but also not very portable.

These robots have far fewer actuators; for a given capability range, they can be much cheaper, as actuators are a huge part of the robot’s cost. They’re more reliable, more or less for the same reason but also because they’re less able to make unstructured contact with the world and that’s where robots tend to damage themselves or others (though this is because they’re less capable).

One understated advantage is that wheels are much, much easier to work with for developers. They can be plugged in more easily. They do not need a gantry. If you want to just push it over to a table and practice picking up cups all day, you can do that pretty easily. All of these steps have a lot more friction with legged robots.

This, to me, means that it’s likely that for a given task, if a wheeled robot can do it, it will likely be cheaper and smarter than the corresponding legged robot (at least in the near term). And the capability envelope for wheels, while lower than legs, isn’t so low it can’t provide value.

So, Legs or Wheels?

I am bullish on legs long-term. I think they’re clearly more capable, and will be safer, once the AI models to do perceptive whole-body control mature. I actually think that, eventually, legs may even be safer for the same capability level — a heavy-duty wheeled robot that can pick up a jug of milk would not be a good thing to have in your home, compared to a much more nimble humanoid,

I think the first robot in most peoples’ homes, though, will have wheels. I expect there will be an increasing number of decent, low-cost hardware options. A wheeled robot at $5,000 to $9,999 USD is something I can see many households purchasing — all it needs to do is, say, be smart enough to pick up toys off a floor, or fold some (not even all!) of the laundry.

That isn’t to say we won’t see legged robots in homes — we absolutely will — just that there will be room for both varieties going forward.

Leave a comment

Share

Subscribe now

There’s a recent trend towards emphasizing the importance of real-world deployments in robotics. It’s common wisdom. On one level, this makes sense: real data is scarce, and real deployments are, in the end, what drive revenue in robotics.

I want to make a counter-argument: deployments in robotics are important, but most robotics deployments are fundamentally not scalable, requiring large, specialized support teams and large amounts of resources to maintain, and the data that these deployments produce is almost certainly less valuable than you think it is.

Only a few classes of robotics deployments can truly be scaled nearly without end — problems like self-driving. That’s because the “single task” is so valuable that collecting endless in-domain data actually does work very well.

And even in this ideal case for robot deployments, data scaling has not thus far been enough! Tesla has millions of miles of driving data, and that data on its own did not result in a working, totally self driving robotaxi — instead, they are deploying robotaxis with drivers in the loop in cities like Austin, collecting what amounts to intervention data of how to improve model performance.

The fundamental reason for this is because to deploy robots, they need to reach high success rates on a particular task; the easiest way to reach high success rates is to constrain the problem; once the problem is highly constrained, it no longer helps you learn anything useful for the next problem because robotics is a wildly high dimensional problem space.

So, let’s talk about robot tasks, deployments, and what’s missing.

Subscribe now

The Distribution of Robot Tasks

We tend to see a few tasks showing up again and again in modern robotics:

• Rigid or semi-rigid pick and place (like this engine cover placement task from Xiaomi)

• Package reorientation (Figure and Robotera), covered in my blog post on recent scaling announcements for real-world robotics

• Box packing (Physical Intelligence and Ultra)

• Clothes folding (HuggingFace, Sunday, Weave, and so many more)

The reason is because, to do useful work in most real-world tasks, robots need to achieve success rates in the 97%-99.9% range or higher, and all of these tasks can be saturated during their training process, massive amounts of data collected in constrained settings, so that high-quality data can be collected and high-quality policies can be trained for them.

Achieving these high success rates takes a lot of time, too. Take this task from Xiaomi, inserting a self-tapping nut:

They initially launched with a 90.2% success rate, and got this task up to a 98% success rate in about 4 months. That’s impressively good, and probably borderline production quality, assuming they can fix any issues with retries. But it demonstrates how slow it can be to improve robot policies up to a useful level.

Part of the problem is simple; in a single task, as a policy improves, it gets harder and harder to find the rare useful failures, from which you can collect more data in order to improve the policy. And modern robot learning isn’t terribly efficient; you’ll often need a lot of data in a particular corner case to resolve it.

Why Do We Need So Much Robot Data?

Once you get deeply involved in a single robotics vertical, you start to learn all the hard things associated with that vertical. This is the implicit knowledge workers in that domain have, which is not readily captured in a blog post or a video or even ChatGPT. How exactly should you physically interact with something? This knowledge, presumably, comes from a robotics deployment.

The problem, as noted by Sumedh Sontakke here, is that the “long tail” of robotics problems is very “long and fat,” meaning that you need to solve many, many distinct problems which each show up quite frequently to get utility out of your “general purpose” robot.

Because there are so many robotics tasks out there, and because the space of robot control is so naturally high dimensional, it’s important that we try to cover the space of potential interactions fairly well.

Robotics is a fairly high-dimensional problem. Let’s consider the minimal amount of free dimensions needed to represent a dual-armed mobile manipulator interacting with the world. The robot’s torso can be represented as an SE(2) position (x, y, theta), and a torso height z. In addition, there will be two 6-DOF poses representing (roll, pitch, yaw, x, y, z) for each of the two end effectors, and two gripper states (left, right). So, 18 degrees of freedom for the robot. In a manipulation problem with N objects, we would need to add 6 degrees of freedom per object.

Now imagine evenly sampling just 10 potential values for each of these 18+6N states. We might want to consider all the potential relationships between objects. However, this explodes in complexity. This is something that we often refer to as the curse of dimensionality. For different values of N:

• At N=1, we have a manageable 10^24 samples

• At N=5 = 10^48, we are approaching the number of atoms on Earth (~10^50)

• At N=10 objects = 10^78 samples, we are approaching the number of atoms in the known universe

The useful regions of this space are relatively limited, but naive exploration is clearly not feasible. In the past, many solutions have been proposed: robot motion planning uses learned heuristics, sampling-based planning, and priors from supervised learning to make the problem more tractable.

In robotics, then, the goal is to explore this massive combinatorial space of possible world states, and as much as possible, gain coverage of all the regions of that state space which might potentially matter to the robot, at sufficient resolution to solve the problem in all cases.

Because the problem is so huge, exhaustive exploration will always be impossible, but likewise because the problem is so huge, it can be easy to miss some crucial manifold which will show up at test time. And this is before even accounting for all of the many different things that will need to be done within this huge configuration space!

The Search for Novelty

So the problem becomes one of operations: how do we explore this huge space, while finding all of the niche little regions of that space that might be relevant to our policy?

Deployment does not usually solve this problem. Deployed robots will very rarely see new situations which we care about, as they must already be performing at high success rates; when they do, they provide a very inefficient way of collecting new demonstrations as you never fully explore a failure mode.

Thus, a goal of robot data collection is to constantly find and explore new, relevant tasks — identifying a broader subspace of robot worlds which we can characterize (through demonstration data) in order to achieve more general and robust policy performance.

Finding these new tasks and examples is an art; finding more high quality task data is of high importance for robot learning, and there are few “natural” flywheels where deployment data can actually provide this.

But there are domains which do introduce a steady stream of novelty for a proper flywheel; the most obvious of these is the home. Companies like Weave Robotics and 1X are able to teleoperate robots in homes remotely when they cannot perform a task, meaning they’ll be able to solve the data problem. But of course, on-demand intervention like this is still an excruciating, difficult way of collecting the right data!

Kyle Vedder of Physical Intelligence wrote a (much shorter) blog post here which makes similar points:

Industrial Robotics Tasks

Where does this leave us? Many interesting industrial tasks can be tightly constrained, so that simple, smaller robot policies can be developed to deploy them. Take this plug insertion task by Sanctuary AI, for example.

In these tasks, the subspace of the vast, high-dimensional world that needs to be explored is extremely small, and can be fully saturated during data collection, so long as the environment is tightly constrained. These tasks, then, are perfectly suited for automation on an assembly line already.

This, notably, completely avoids a strict need for learning from deployment, or any notion of a “data flywheel,” by simply choosing a task that is straightforward to train a model for.

Final Thoughts

So, to summarize:

• Most robot deployments don’t introduce new novelty at a sufficient pace to improve your model’s general capabilities.

• Most robot deployments don’t generate new failure and correction data fast enough to improve models at a reasonable rate.

What are we to do? Well, we have two options:

• Find an environment where you actually can experience a wide range of tasks, all the time, in order to improve basic model capabilities.

• Have a cracked ops team who can go out and make up new tasks to explore on their own.

Fortunately, I think all of this will be solved via the magic of scale. There are many, many more robots in the world now; there will be even more next year, and even more the year after that. So long as these robots are doing interesting things, and so long as the data they generate can be used and shared, we will be able to mine this data more and more to create general-purpose robots, even if each individual effort is not so useful.

But it might take a while to get there.

Subscribe now

Share

Leave a comment

Inspired by a spate of recent announcements, I want to explore a thought experiment. If the price of robots keeps dropping, and their capability keeps going up, how would the world change?

The widespread success of coding models has made software easier to create than ever. You can now, with a modest investment, create your dream website or app — without being held back by years of software engineering experience. As we move towards a world where software is “free” — where frontier models like Fable 5 can one-shot a whole app — we must start wondering about what happens if we apply the same logic to robotics and, therefore, labor.

The fundamental premise of robotics is that we can automate large swathes of human manual labor and replace it with machine labor. In software engineering, abundance means more and better websites, easier data analysis, and more apps for niche applications. For robotics, universal automation — “free,” or really just radically discounted labor — means the “long tail” of jobs that just are not even worth automating right now.

What does this work look like?

Subscribe now

The Work Not Done

To understand the sheer amount of work not being done, all you need to do is to look around.

Maintenance and cleaning are constantly postponed until the last possible minute. With infinite labor, we can do actually preventative maintenance: re-grout and re-caulk tiles, repaint road lines every month so they always look brand new. Gutters, roofs, and weatherstripping can be re-done before structural issues form.

Litter would vanish. Public spaces could be maintained, by robots, at the level of a boutique hotel. The world can be made clean and beautiful, with gardens and public art and clean streets everywhere.

And it goes past this - with free labor, homes can be de-cluttered and automatically organized, their contents entered into a contextual database, instantly queryable and searchable, accessible to AI agents to make sure that, for example, the ingredients for dinner are ready and prepped the second you get home.

Things can be made to last longer. We have a culture which consumes endless, disposable goods in part because of cost of labor: a modern industrial production line can mass produce new goods, but always the same goods. What’s worse, repairing old ones requires expensive human hands — so we just don’t do that. With “free” robotic labor available at scale, we might repair old shoes or upgrade old phones instead of throwing them out, reupholster old furniture, and patch worn clothes instead of consigning them to a landfill.

Food and goods can be made incredibly customizable. Every piece of clothing custom fit; every meal freshly-prepared from fresh ingredients.

Even care work can be made easier. Even if the human elements of teaching, childcare, and eldercare aren’t replaced, free labor makes the burdens of such jobs lighter — robots could reset classrooms, prepare activities, and help carry heavy weights.

Jevons Paradox

One must then wonder if, in such a world, there will be less jobs.

Fundamentally, though, we must see that the demand for things is essentially infinite. With nigh-infinite labor, there is a massive amount of work that can be done, and therefore demand for labor will likely skyrocket.

This has been studied before, as Jevons Paradox, and it has occurred many times in human history. As travel got cheaper thanks to the automobile, people traveled more; as lighting got cheaper, thanks to the electric lightbulb, human societies generated more and more and more light.

The classic example of Jevons paradox in automation is the story of the bank teller and the ATM. Originally thought to replace bank tellers, the invention of the ATM actually saw the number of bank tellers increase until the 2010s. Human bank tellers could focus on more high-valued work, leading to the opening of new branches.

Likewise, we might think of this “free labor” as replacing not workers but enabling them — there will remain many decisions to be made, about what should be built and what work should be done. Doing the right thing — having taste — will become more important than rote labor.

Why This World Might Be Coming Soon

Good, useful robots are getting much, much cheaper, something I have written about before here. The fundamental barrier is no longer cost, but intelligence: can we make these cheap robots reliable enough, at a wide range of tasks, so that we can make this explosion of labor actually useful to people.

The other thing worth considering is that this both builds upon, and enables, an expansion of agentic AI. Endless, cheap, robotic real-world labor will be the hands and eyes and mouths of cloud-based AI, accumulating data and creating the context for future decision making.

For all of this to happen, we just need current trends to continue to hold. Hardware needs to get cheaper, robots need to become more plentiful, and AI needs to continue to get more powerful and more general as it can ingest more data.

From my point of view, this is practically guaranteed.

Subscribe now

Share

Leave a comment

Read More

When Robots Create Jobs by Nils of Auki

A couple startups have announced really exciting general-purpose robots in the sub-$10,000 range. These are finally, finally starting to look like something that could be made very capable.

This is exciting because it means affordable home robots, easily in the same range as common home appliances like a refrigerator or a dishwasher, are clearly only a few years from the market.

Let’s go over a couple of the latest announcements.

Subscribe now

Nori L2

Size: 100cm or 130cm tall for Grande (3-4 feet)

Price: $1,288 or $1,322 for the Grande model

Compute onboard: no (thin client)

Nori Robotics, founded by Antonio Li, is selling possibly the cheapest reasonably large bimanual robot I have ever seen. It’s tall enough to reach the top of a table or countertop (though definitely not a top shelf).

The arms look similar to the SO-100, a very common low-cost open-source arm by HuggingFace; probably nice if you want to build a leader setup for teleoperation for a few hundred extra dollars. The robot itself also has a nice screen useful for telepresence.

You can read about an earlier version of in the Nori robot paper, which probably owes some inspiration to the XLeRobot which I have covered before.

You can see the announcement video here.

BracketBot 2.5

Size: ~5’ 8”

Price: < $3000

Compute onboard: possibly a Jetson Nano

Another wheeled robot manipulator, this one with a distinctive tall, thin shape, and hoverboard-style drive wheels (with stabilizers running out in front). The creators want to ship “a billion robots” and with a price like this it seems totally reasonable.

There’s a blog post on a similar project here, written by Clayton Haight. Bracketbot, though, is founded by Brian Machado.

Learn more in the X post.

Weave Isaac 1

Size: probably roughly ~5’ 6”

Price: $8,000 or $450/month

Compute onboard: unknown; teleop-enabled

The successor to Isaac 0, the clothes-folding home robot, Isaac 1 is a fully capable home mobile manipulator which can put away laundry and tidy a room. This is the first robot promising the full (semi-) autonomous experience that you can pre-order this year, after last year’s launch of the 1X NEO for pre-orders.

This probably has the most robust and full-featured software stack; you can already see Isaac 0 robots in different locations, folding clothes. According to Weave, the older model folds 1000+ pounds of clothes every week.

Another note: unlike the other two, this one clearly comes with a charging dock, which is a great and really underrated feature for a robot like this.

You can pre-order Isaac 1 here.

Final Thoughts

These robots have a couple similarities that are letting them run at low cost.

• Wheels over legs. Wheels break less and require less parts. The tradeoffs in terms of going up and down stairs will most likely be worth it in terms of product lifespan without spending much more on software, sensors, compute, and model training.

• Remote inference. None of them seem likely to have much compute for models; they’re largely going to rely on the cloud for this, or some other local offboard compute. Isaac 1 may be the exception, but it does allow for teleop.

• End effector cameras, but simple grippers. No fancy tactile sensors or dexterous hands - just robust hardware.

• Minimal actuation. The heads for BracketBot and Nori seem likely to be completely fixed.

This means they share a common set of limitations - no stairs, cannot access top shelves in a kitchen. Given the added complexity of legs, though, and the safety risks of a robot autonomously going up and down stairs, it’s not clear this is such a bad thing.

All in all — an exciting set of announcements and I would love to try out all three.

Leave a comment

Share

Subscribe now

It’s really hard to tell which robot policies are better than others, and who is actually making the most progress towards embodied general intelligence. I’ve written about robotics evaluation many times before, but this time I wanted to talk about leaderboards, as inspired by a thread on X.

There are many benchmarks, but few evolving leaderboards. PolarRiS, for example, is a great tool for benchmarking which correlates well with real-world performance, but it isn’t a live leaderboard in the same way — the authors do not update it, and the set of entrants is closed to the ones they chose at paper writing time.

Instead, let’s look at what public artifacts we have over time, from people who have been updating results over time, and see if we can identify any trends in robot learning.

Updated June 25, 2026 to add new Cosmos 3 results.

Subscribe now

Why This is a Hard Problem

There are two challenges here:

• Running a benchmark is expensive

• You need buy-in, which means community excitement as well as a feeling of progress (the benchmark must be hard but not too hard; real but not too real)

I actually briefly ran a robotics competition for open-vocabulary mobile manipulation (OVMM). There are a few problems here which make this uniquely hard for robotics compared to computer vision leaderboards (even something like the chatbot arena).

First, it’s costly to run a leaderboard like this, especially if your main audience is academics. Unlike LLMs, robot evaluation is costly: you need either large-scale simulation, or actual robot hardware, neither of which is cheap. Robot hardware in the real world will also incur resetting costs which do not exist for a language or image-based benchmark.

Another issue we’ll have to contend with: once a benchmark becomes widely accepted, it loses some of its value because people start to cheat.

Benchmark Hacking

Another big issue is benchmark hacking. We have seen this before with major LLMs like Meta’s Llama 4, but it’s rarer in robotics, partly because our benchmarks aren’t of widely accepted value like Chatbot Arena is/was.

This appears to have changed:

Read the thread from Pranav Atreya on X for more info. RoboArena works by volunteers evaluating pairs of policies; some teams were volunteering to assess policies, but often not completing all the evaluations they were assigned.

You’ll notice that now they’ve added indicators to their leaderboard for which models are low-sample:

Spirit, currently vying for top spot but with low confidence, is a fast-growing Chinese AI startup. Their VLA (Vision Language Action model - basically a generalist robot policy) is in fact open source, and you can find it on Github. For a primer on VLAs, check out my previous blog post on the topic.

My Own Attempt: The HomeRobot Benchmark

When I was at Meta we wanted to build a real-world robotics benchmark for pick and place. This is one of my favorite tasks still, and one that I think the robotics community remains fairly bad at because dexterity and task-specific work has overtaken it in focus.

Here’s the problem: I want a robot which can help me out in my home. That means I need to be able to tell it to pick up any object, and move it to any other location. As a hint we also gave it an idea of where the object could be found (sofa, chair, etc), as with low-resolution cameras maybe you cannot see a small object from across the room, and in a large house this will increase problem size tremendously.

In our case, we split this between a simulation-only leaderboard, exploring a bunch of different simulated homes (in the Habitat simulation), and a real-world setting in a specific apartment built by Meta for the purpose.

The simulation component was implemented on EvalAI. Basically: submit a docker image containing your code, it gets executed by us, and we maintain full control of the test environment. This is a bit different from most modern leaderboards, which emphatically do not allow for on-site code execution — but of course it also dramatically affects hosting costs.

In the end, the experiment was a middling success; roughly 60 teams submitted, we briefly evaluated three of these in the real world. We learned a lot though.

It’s very hard to strike a balance between “too hard” and “too easy.” OVMM was a case that was actually pretty hard for learning-based methods, and in fact our winning solution in sim and real was a model-based method that merely leveraged open-vocabulary vision.

A lot of the time was spent chasing down simulator bugs or tweaking RL environments to make sure they trained properly — in order to give those using the environments something to actually work with, and make sure the environment actually transferred to the real world.

Another note is that the $25,000 Hello Robot Stretch was probably still too expensive. Today I would probably want to use something like XLERobot or BracketBot — at less than 1/10 the price, everyone could buy one and use it. This would also make it more important for us to set up “home testing” instructions, codifying the rules we could use so people could scatter objects around their own labs to get a good signal about how well they would do.

Three Leaderboards

But enough about that. Mobile robotics benchmarks are particularly difficult, as they just take up so much time, to build and to evaluate. Most of the excitement today is around robot dexterity — robots manipulating objects.

Let’s look at three interesting leaderboards for robotics models that have all been updated recently:

• RoboLab-120: a simulation-based benchmark with experiments run by NVIDIA staff (leaderboard, code)

• RoboArena: a distributed, simulation-based robotics leaderboard where experiments are run in reality (leaderboard, RoboPapers episode, code)

• MolmoSpaces: an open-source platform for robotics develeopment, supporting multiple simulations and training methods. Technically this one is a mobile robot leaderboard, with a range of different tasks (leaderboard, Robopapers episode, code)

What do we see when we compare them?

RoboLab-120

Recently, RoboLab-120’s authors added Cosmos3-Nano-Policy to the leaderboards. This pushed down pi-0.5 and DreamZero, previously the top two previous policies.

RoboLab is a procedurally generated environment simulation based on pick-and-place tasks. Most of its benchmark entrants are VLAs. You can check out the list of example tasks here: they are dominated by pick and place, all in simulation, and run in different environments.

Some examples:

The environment leaderboard is dominated largely by VLAs, and all evaluations are run in-house. This is in many ways the gold standard for a benchmark, because you can’t just submit your own run (impossible to game, as happened briefly to RoboArena), but also means that the number of individual models is very limited — no Spirit VLA, for example.

The breakdown is interesting too. We see the newest NVIDIA model at the top, perhaps unsurprising for an in-house NVIDIA benchmark. The old pi-0.5 is second place, unless you rank for complex tasks only — in which case DreamZero comes in second.

RoboArena

We can see several world action models at the top (DreamZero for example), and the Spirit VLA mentioned above. There’s a much wider range of policies here, though the “confidence” can be a lot lower - there are so many more policies being submitted, and so many fewer potential evaluators.

If you have not seen it, this image from the website is a good indicator of what the policy evaluation environment looks like:

Basically, it is a benchmark of a single Franka arm performing various tasks, again largely pick and place. It’s another leaderboard that is completely dominated by VLAs and World Action Models, with Spirit,

MolmoSpaces

This is probably the most different experiment. MolmoSpaces is large and can involve mobile robots. While AI2 provides data, it’s not going to be as well studied as the Droid dataset. Correspondingly, we see a very different set of benchmark results here — some hybrid methods, including TiPToP, a task and motion planning-based system that leverages foundation models.

The wider range of tasks makes this benchmark more interesting, but it’s a pure simulation benchmark with lower-quality rendering.

Update June 25, 2026: Cosmos 3 is now at the top of this benchmark as well!

It’s definitely interesting how different these various leaderboards still are, with TiPToP (a task and motion planner) near the top of MolmoSpaces, Cosmos crushing the simulation benchmarks, and a list of VLAs on RoboArena.

Final Thoughts

There are many other benchmarks out there. I’ve written at length about evaluating robot policies before, and won’t go into that in more detail:

But let’s look at what the top entries here actually are. When it comes to pick and place tasks, there’s not a strong consensus — the ordering of the top models seems in flux. Compared to LLMs, though, it seems older models (especially pi-0.5) have a lot more staying power. It’s also just possible that benchmarks like RoboArena are easier to saturate - that we need complex, and more challenging, benchmarks with associated resources for verification.

So, my takeaway: we still really don’t know what is the best here, and there are no clear winners who just dominate all available benchmarks. Anyone could still “win” in robotics. We see new players (Spirit) appearing at the top of established leaderboards; we also see a surprising stickiness for some older policies.

Part of that last stickiness is almost certainly data — pi-0.5 was trained on tens of thousands of hours of high-quality real robot data, which many newer entrants have not yet managed to collect. This will put them ahead, at least until the large data collection industry can be leveraged to bolster their performance.

Let me know what you think; and what you think the future robotics leaderboards should actually look like.

Subscribe now

Share

Leave a comment

Robots need to be able to interact with the world in order to be useful. They do this in many ways: by driving around and taking pictures during inspection tasks, by walking up and down stairs. Military robots often interact with the world by exploding. But increasingly, we want useful robots to manipulate their environment and perform human-level tasks, which requires a gripper or hand.

The problem is that human hands are incredibly capable — they’re a high bar. Human hands are strong, light, adaptive, and self-healing. The human fingertip has ~2,500 touch receptors per square centimeter, with roughly 22 degrees of freedom. It can move at high speeds while applying significant force.

Building a robot hand that can match human capabilities, then, is very difficult. But robot hands have seen incredibly rapid change over the last couple years. You might have seen incredible manipulation demos from companies like Sharpa Robotics. The Sharpa Wave hand is part of a new generation of robot hands which nearly match human capabilities.

Now we have hands like this one, which is $600 and precise and compliant enough to pick up a raspberry without smashing it:

This is a great task for a few reasons: a raspberry is obviously very tiny, but it’s also squishy, soft and easily damaged. The CRAFT hand was made by researchers at UIUC and UC Irvine, led by Prof. Unnat Jain. The price tag is incredible, at only $600, and the plans for building it are open source. But there are a wide range of open-source robot hands coming out now, as well as an increasing range of commercial options.

So, let’s talk about robot hands. What are the key design decisions? Which one should you get? How do you make sense of all this? If you find this interesting, please subscribe and share.

Subscribe now

Why Do We Even Need Hands?

The most common robot hands, like those mounted on the YAM or ALOHA arms, have only a single degree of freedom: opening and closing. And this is enough for an incredibly wide range of tasks, as Physical Intelligence is so adept at demonstrating. Obviously we do not need hands on our robots then:

But of course, no one sees a demo of something that doesn’t work; robots aren’t deployed doing things that cannot be automated. And indeed it turns out that lots of tasks can’t be done without multiple fingers: in-hand reorientation of an object, for example. And solidly grasping long, thin objects — even managing door handles — can be difficult.

While the limitations of simple two-finger grippers are not an issue in relatively-controlled industrial settings, and for simple pick-and-place tasks of small, low-mass objects, this describes a minority of tasks that we might care about. Things that we need higher-DOF dexterous hands to do include:

• Very stable manipulation of objects with a wide range of masses

• Robust handling of long or thin objects

• Tool use, especially for any tool that is designed to also support humans

For a truly general-purpose robot, robot hands need to get a lot better. But this itself is a huge issue; human hands have a frankly incredible mix of robustness, ability to apply force, sensor density, and dexterity, with degrees of freedom comparable to entire humanoid robot platforms.

Fortunately, many people are working on this.

Types of Robot Hands

Robot hand design has to trade off between many different conditions:

• Strength - how much force it can apply

• Compactness - how big the hand is — human size being ideal for video learning

• Precision - how accurately can it servo to different positions

• Affordability - how expensive is it?

• Robustness - how often does it break

Human hands can apply a huge amount of force in a very small and very durable package, and two of them are attached essentially to every human. They therefore max out nearly every one of these conditions. Robot hands lag behind a bit. To understand why, we need to talk about how robots actually actuate and interact with the world.

A large part of the modern “humanoid revolution” and the scalability of robot quadrupeds is the use of quasi-direct-drive (QDD) actuation pioneered by the MIT Mini Cheetah. Quasi-direct-drive here usually means a low-gear ratio planetary gear assembly, connected “directly” to the joint that it’s moving.

This gives a great tradeoff between force transparency (equivalence between applied and measured forces), and backdriveability (ability to “push” a joint backwards without exerting extra force).

These two qualities are extremely important for robot hands, which are constantly colliding with things, and thus experiencing forces all the time! These forces can easily translate to robot damage, and so a non-backdriveable hand with poor force transparency can be easily destroyed by simple interactions.

The problem is that putting the big motors that you need in order to make a robot work are not (normally) going to fit in a human hand as they would for the leg of a quadruped or the arm of a humanoid robot. To exert the forces you want to firmly grasp and hold objects, then, you need higher gear ratios, which hurts backdriveability and has downstream effects on performance. In addition, these servos also mean more weight in the fingers, which causes its own performance problems and further increases the need for powerful servos further up the kinematic chain.

Broadly, there are two classes of solutions to the problem:

• Tendon-driven hands, like the ORCA hand, put the motors outside the fingers and connect them to joints via a tendon of some kind.

• Direct-drive hands like those from market leaders like Wuji and Sharpa, whose strategy revolves around building smaller and smaller motors to actually fit into the individual joints of the hand, using a higher gear ratio plus active compliance to achieve backdrivability.

This blog post by Quanting Xie, Tongzhou Liao, and Yonatan Bisk called the Dexterity Deadlock is a good read for more on the topic.

And there’s another connected issue. Once we are starting to pack lots of actuating power into a small form factor, we might need to make trade-offs and reduce the number of motors. This cuts the number of active (actuated) degrees of freedom of the hand, but results in lower cost, weight, and can save size as well. Notably, even human hands have passive degrees of freedom (the top joint on your finger usually cannot move independently of the others) — even Mother Nature struggles to produce a perfect hand.

It’s also possible that the actuation problem above needs to be solved by wholly new technology. One example of a team trying to do this is Daxo Robotics, which is working on soft or continuum robot hands which lack rigid links and thus technically have “infinite” degrees of freedom, similar to human muscles. These technologies are currently harder to manufacture, but this might be solvable; and they can take a lot more punishment than many traditional hands. For more you can check out our podcast episode from last year:

RoboPapers

Ep#40: Daxo Robotics

How can we build robotic hands with truly superhuman dexterity? Daxo Robotics is developing a unique tendon-driven soft robot hand, which aims to be tougher and more capable than a traditional humanoid hand. Each finger consists of many different tendons, which act in concert to move or manipulate…

Listen now

10 months ago · Chris Paxton and Michael Cho

Open Source Hands

We have a very large design space of tradeoffs in robustness, strength, cost, weight, and flexibility to explore, in order to try to build robot hands that can get close to human level performance. Much of the research I’ve seen looks at tendon-driven hands: academic researchers aren’t spinning their own motors or building custom high-precision gearing in the same way as an industrial hand manufacturer might be.

Another huge issue for researchers is the high cost of current hands: the best hands like the Sharpa Wave are often over $50,000 each, meaning that the whole cost of a robot can be dwarfed by the pair of hands mounted on the ends of its arms.

So there have been a couple recent efforts to build good, high-quality open-source hands. Take, for example, the RUKA hand from NYU, a <$1300 tendon-driven hand with 6 active and 15 total degrees of freedom for compliant grasping. It’s slightly bulkier than a human hand and has less degrees of freedom, but is lightweight and much more affordable.

Another example would be the LEAP hand (v1, v2); the v2 hand has 8 active dof, which makes it slightly more capable of adapting to different tasks and means it can be used for, for example, in-hand reorientation of objects. It’s much bigger than a human hand, being more similar to the Allegro hand in size, but still costs only ~$2000. The advanced version is even better, at $3000 and 22 dof, roughly human level.

The recent CRAFT hand actually starts to approach human capability on paper, with 15 active and 5 passive degrees of freedom, plus good compliance and actuation with lightweight fingers and for about $600. It’s a 3D printed hybrid hard-soft hand, which means that its finger joints use soft thermoplastic polyurethane, aiming to make sure that it can conform and make contact smoothly and robustly without taking damage from applying too much force to an object.

There are many more researchers working on novel types of robot hands; given how difficult the problem is, it is great that so many open, printable and hackable ideas are being produced to drive the area forward.

A New Wave of Hand-Centric Startups

And of course now there are a wave of robot hand startups appearing. This will not, by any means, be an exhaustive list of hands, just a few representative examples to talk about new entrants to the space over the last couple years. There are of course an older wave of hand companies like the Shadow Robot dexterous hand series — these too are out of scope for this post.

Some examples, but by no means an all-inclusive list:

• Sharpa: Singapore-based Sharpa is probably the industry leader at the moment, building hands which are a near match for human capability. They have done a number of impressive demos, including one where they show their 63-DOF Sharpa North robot peeling an apple with an apple peeler.

• Linkerbot is seeking a $6 billion valuation for a new round of funding (as per Reuters) and is mostly known for its robot hands. There are robot hands like the Linkerbot o6 which are roughly $1500, though this one is only 6 dof.

• BrainCo’s Revo 2 DuraX hands focus heavily on durability. The hands have been demonstrated many times as prosthetics to replace amputated limbs, similar to Psyonic hands. While the Revo 2 had only six active degrees of freedom and was relatively cheap, they’re moving towards higher-DOF and thus more expensive hands with the Revo 3, which will also integrate tactile sensing.

• The Psyonic Ability hand is another durable hand with only 6 active degrees of freedom, designed first for prosthetics and featuring built-in pressure sensors.

• Inspire hands are a commonly-used cheap robot hand, also with six active degrees of freedom.

There are many others of course. But the trend here is clearly towards higher degrees of freedom and more tactile density, with the SharpaWave being very similar to a human hand in terms of raw specifications: it has roughly 1,000 tactile pixels per fingertip and 22 active degrees of freedom, vs. roughly 21 for a human (depending on how you count).

What Issues Remain

The big open challenges in hands, as I see it, are:

• High cost. Robot hands are expensive. Two $50,000 USD SharpaWave hands may be the best on the market, but they will cost you more than your entire robot.

• Reliability. Human hands take constant impacts and bounce back, partly because they are ablative (covered in regrowing skin and muscle that heals on its own). Robot hands are notoriously finicky; even very robust hands like some of the prosthetics mentioned above won’t match human reliability.

• Sensing. Part of why human hands are robust is also just that we use them better: we avoid damage to our hands, we respond very quickly to heat or pressure by instinct. This is served by a network of dense tactile sensors and built-in whole body control behaviors centered around preventing damage.

None of these issues, I think, are insurmountable, but all will take dedicated work. There’s no “perfect” hand on the market now, which is part of why so many research labs have decided to build their own.

Final Thoughts

This is a blog post without a real lesson, just an observation. There’s a lot of companies and researchers out there trying to solve the problem of human-level dexterity. Now, for the first time, there really appear to be roughly human equivalent hands, in terms of dexterity and sensing if not manufacturability, strength, and robustness.

But these remaining issues will be resolved. This has real implications; as Avik De and I wrote about in our recent article on the changing nature of robotics research, hand researchers will shortly need to “move up the value chain” away from the kinds of fundamental work they are doing now.

min{power}

The First Paradigm in Robotics & AI Research: Lessons from Computer Engineering

Read more

4 months ago · 29 likes · 8 comments · Avik De and Chris Paxton

There’s no version of the world, I think, where in 5 years there are not good, dexterous, capable hands for a few thousand dollars price, ready for integration into different robots and products.

You should probably plan accordingly.

Subscribe now

Share

Leave a comment

References

• Lin, L., Patel, S., Moon, J., Lazebnik, S., & Jain, U. (2026). CRAFT: A Tendon-Driven Hand with Hybrid Hard-Soft Compliance. arXiv:2603.12120. Project page: https://craft-hand.github.io/

• Zorin, A., Guzey, I., Yan, B., Iyer, A., Kondrich, L., Bhattasali, N. X., & Pinto, L. (2025). RUKA: Rethinking the Design of Humanoid Hands with Learning. Robotics: Science and Systems (RSS). Project Page: ruka-hand.github.io

• Xie, Q., Liao, T., & Bisk, Y. (2026, February 12). The Dexterity Deadlock. Origami Robotics. Article: origami-robotics.com/blog/dexterity-deadlocks

• Shaw, K., Agarwal, A., & Pathak, D. (2023). LEAP Hand: Low-Cost, Efficient, and Anthropomorphic Hand for Robot Learning. Robotics: Science and Systems (RSS). Project Page: leaphand.com

• LEAP Hand v2 Advanced: Dexterous, Low-cost Hybrid Rigid-Soft Hand for Robot Learning. (2024/2026). Project Pages: v2.leaphand.com / v2-adv.leaphand.com

For a long time, we have seen robots mostly in ones and twos in videos, mostly in lab environments. But there have been a spate of announcements this week that aim to change that, with evidence of promising real-world deployments and real manufacturing scale-ups starting.

A quick reaction blog post, and some thoughts. If you find this kind of thing interesting, please consider subscribing.

Subscribe now

RobotEra Logistics Deployment

Check out this post from CyberRoboooo on X, which shows a line of RobotEra L7 humanoids in action. These robots are busy aligning bar codes at logistics hubs all over China, including ones run by by China Post and others.

Aligning bar codes looks like a ridiculous task for humanoids, but as you can see here it’s actually one that’s rarely automated. This is because:

• It requires handling a surprising variety of package shapes and sizes

• Requires human-level dexterity and speed — traditional automation struggles

• Requires high reliability - every bar code NEEDS to be scanned.

At really high-end logistics companies (think Amazon), this is indeed done with traditional automation — packages go through a funnel with many cameras, so orientation does not matter. But these solutions are expensive, and most e-commerce retailers don’t use them yet, instead relying on human labor. Which may become humanoid robot labor soon.

Beijing-based RobotEra launched this logistics platform last year. RobotEra has raised around $200 million USD equivalent funding (RMB 1 billion) and builds a wide range of robots.

A Cheap Unitree Manipulator

Unitree is most famous for its G1 and H1/H2 humanoids: mobile robots which are largely not designed for manipulation. Sure, they’ve been used that way — many times — but they’re fundamentally robots for whole-body control research and entertainment. While they’ve toyed with manipulation-focused robots in the past (like the Unitree G1-D), this hasn’t really been an area of focus for the company.

So it’s really interesting that they’re launching a line of manipulation-focused robots based on their very cheap R1 series platform. The R1 is the unitree “toy” robot, the roughly $5k humanoid you can buy to cartwheel and box and, well, that’s about it, normally.

With 19 degrees of freedom (including 2 from the grippers), a mobile base, an actuated head, and ~half a meter of reach, this actually seems like a really nice little robot to get started with, especially given the low price. The payload is very low, at only ~2 kg per arm, but is still enough for lots of light manipulation work.

Information about the robots was posted on X. The price doesn’t seem to include the mobile base, but it’s still a very good start and indicates how cheap, at the low end, industrial semi-humanoids are likely to become.

Figure BotQ Manufacturing Update

Not to be outdone by the Chinese companies, a few American companies have made huge announcements with incredible high-quality videos this week. First up is Figure, which increased production of their Figure 03 humanoid up to 1 robot per hour — capable of 24 a day.

Stills from the videos in their blog post are impressive: showing a veritable army of robots, robots performing exercises, human technicians building robots, and perception-aware whole body control.

Another interesting note: Figure is now doing 3d perception. Their RGB stereo cameras are used to predict 3d structure, which they can use for navigation and moving around in the environment. Another step toward real-world deployment, as it means reliable, robust navigation should be ready to ship soon.

I’ve written about Figure 03 before if you want to know more about the robot. Currently, these robots are targeted at research, data collection, and customer discovery.

1x Hayward Factory

Home robot maker 1x released a look at their Hayward factory, built to produce 10,000 NEOs a year. The look even showed NEO robots walking around the factory floor, helping out with various tasks.

Some more points they announced:

• Vertical integration means building motors from copper on up

• 10,000+ pre-orders in the first 5 days of their NEO launch (order here)

I won’t say as much here because the videos speak for themselves. You can also read the accompanying blog post.

Tutor Intelligence and “America’s First Data Factory”

I wanted to include this: Tutor’s co-founder and CEO Josh Gruenstein shared a 100-robot data factory, where their industrial semi-humanoid Sonny is busy being used to scale robot learning.

Data Factory 1 (as they call it) aims to scale remote-VR proprioceptive teleop, so they can use it to cover the whole range of the robot’s lifecycle. This includes data annotation:

This is a nice look at what a good teleop system looks like and how to annotate images; a “peek behind the curtain” of scaling robot learning that many may not have seen before. These operational details are maybe not useful for Tutor’s competitors (the Physical Intelligences of the world) but are really useful for those getting started, so it’s great to see.

Final Thoughts

While manufacturing progress is being made, much of what is being scaled now seems to be “industrial semi-humanoids,” i.e. integrated robots with multiple arms and a head capable of a wide range of dexterous tasks. RobotEra, for example, does not bother with legs for their package-handling deployment. You’ll note that the RobotEra task - package arrangement — is very similar to one Figure has been doing, and Figure is also perhaps unsurprisingly in talks with UPS to deploy their robots.

I think we will see that there are a few tasks that modern end-to-end robot learning is extremely suited for — box packing, package sorting and reorienting, clothes folding — and that will proliferate quickly. The costs of these activities will rapidly drop as robotics production and intelligence scale.

Subscribe now

Share

Leave a comment

Large language models are a fundamentally different type of intelligence from humans; and this means that even as they approach and even exceed humans in some domains (coding!) we will still see some places where they lag behind human intelligence in surprising ways.

Spatial reasoning is one of these — telling the difference between “left” and “right” in various contexts is still often hard for even very powerful models. Says one user on X:

This shortcoming of language models has actually existed for a very long time and it’s an interesting one that, in my opinion, is illustrative of the structural weaknesses of learning principally from language, and why these weaknesses aren’t as true for, say, color. And so, a short post on left, right, colors, and why some things are harder for LLMs and VLMs to learn than others.

Subscribe now

Spatial Understanding is Hard from Language

Language does a terrible job specifically of capturing spatial information. Why is this?

Spatial concepts just aren’t obvious and well-defined from the vast corpus of text available to train on. What does it really mean to walk across a room, or to put a book on a bookshelf? The precise spatial locations that we might need to unambiguously resolve this missing information into a clear concept just isn’t a part of the training data in any real way.

I’ve written about world representations for robots in the past (a term which I use very differently from “world models”). For my world representation to be useful, it needs to capture, for example, the positions of all of the various objects in a house, so that I can navigate to them, manipulate them, and ask questions about them.

The process of using sensor measurements to answer these questions is usually referred to as grounding the concepts. The symbol grounding problem deals with how we connect these symbols (on top of a table, inside a box, lined up, clean…) into an understanding of the physical world.

There’s actually a surprising amount of information in LLMs on how objects should be arranged, but there’s always missing information too — details about geometry and contact, about object shapes and physics. Training on text is like learning from a textbook with half the pages torn out, or watching a YouTube video with captions in the wrong language. You’re probably still going to learn something, but a lot of the key context is missing.

This Missing Information Persists in VLMs

But many of the leading frontier models these days are actually multi-modal, taking in images or even short video clips. Unfortunately, this is still not enough to resolve the problem — it persists even in a vision-language model (VLM).

VLMs are trained on images from a single perspective, or in the case of a short video a handful of highly correlated perspectives. This reduces ambiguity and helps resolve some of the missing context, but hardly removes it completely. What does “left” mean in the context of the portrait? My left, or the left of the person in the picture facing me? These things are opposites.

Such ambiguities aren’t really a challenge for modern VLMs (I can just specify “my left” or “their left” in the prompt), but complexity and ambiguity remain. Problems like precise estimation of 3d object position from images remain challenging, which means that for many real-world tasks, the VLM just doesn’t work to a reasonable level of reliability.

Example: Position vs. Color

Consider how an LLM might leverage its training data to understand colors (red, purple, blue..) versus left and right.

Color is used consistently. An apple is red; the sky is blue; leaves are green. Even if I do not know what green “means” — in terms of specific RGB values — I do know which things are described as green, and because green represents a distribution of RGB values, it’s used fairly consistently and it’s easy enough for a model to understand that something will be green just based on this data.

However, as mentioned above, left and right are highly context dependent. They’re going to be used in millions of different ways, to mean different things.

Broader Implications

This is an important illustration of an idea that I’d like to write more about: I fully expect AI and robotics, now, to be able to exceed humans in many domains. This does not necessarily mean they’ll exceed in all domains; there will be some “comparative advantage” where humans remain leaders in some tasks. It’s hard to say what those are; but we can look at the types of learning and reasoning that humans do well for examples.

Humans, for example, operate off of comparatively noisy training data. We set our own learning goals rather than relying on someone else evaluating us. We get bored and do something else, managing our own time to keep us from getting stuck in pointless loops (except during grad school).

Our “training data” is based towards real, streaming, multi-modal data that’s aggressively compressed - using far more compression than works with modern transformers. Humans have to disregard millions of useless signals from various sensors just to go about our daily lives. By contrast, LLMs and VLMs are trained using mostly still images and mountains of text — they’re learning concepts like “left” and “right,” “purple” and “green” all from associations.

There are lots of cases where machine and human intelligence just end up working differently, due to these differences in compression, training data, and of course our different computational hardware.

Ethan Mollick has written about us approaching a sort of jagged AGI, where modern models exceed human intelligence in some areas but fall behind in others, making comparatively little progress. When you look at how these models are learning and what they are doing, I think we’ll find many of these shortcomings are due to fundamental differences in the way human vs machine intelligence works.

This is obscured by the fact that usually, we don’t need our AI agents to interact with the physical world. They inhabit a realm of text prediction and function calls just fine. But this still represents a fundamental limit of what they can do, which can only be overcome, I think, by figuring out how to scale embodied data.

Final Thoughts

This is all to say that certain types of intelligence, I believe, only really make sense with embodiment. Text, and even much of the image data on the internet, fundamentally can’t learn to solve all of artificial intelligence — embodiment matters.

This is also another reason to be excited about world models, which potentially can start to understand these concepts and learn to ground the various spatial relationships that we talk about in reality. World models take in and predict video data; because of this, they can actually learn an implicit model of the world, of how objects interact, and learn (implicitly) to understand the 3D space that humans and embodied agents must inhabit.

Perhaps the most interesting question, then, is this: will embodied data eventually allow our AI agents to become better at, say, using 3D modeling software, as per the tweet in the intro to this blog post? I believe so; it seems possible at least that these concepts will be best captured through large scale interaction.

After all, it would make sense if, to achieve human-level artificial intelligence, we needed agents to learn the same way humans do.

A horde of visitors just descended on San Jose, California; blocking off multiple city blocks, many wearing black and lime green, all of them talking unceasingly about artificial intelligence. Yes, it’s time for another NVIDIA GTC.

GTC is an interesting conference for me. It’s much more focused on business and sales than the conferences that I usually would go to, like the Conference on Robot Learning. Most of this is selling AI software, but there was a pretty good presence centered around physical and industrial AI — which is to say, robotics.

In this short post I’ll give some of the things I saw on the show floor.

All videos and photos in the article were taken by me.

Subscribe now

Skild Assembles GPUs

First up, demoing robotics manipulation in the NVIDIA booth, it’s Skild!

Skild has just raised an incredible $1.4 billion Series C with $30 million in revenue, meaning it probably has the most revenue of any of the current-gen robotics “neolabs.” And here they’re demonstrating why: they have an incredibly precise manipulation demo showing server rack assembly.

From CEO Deepak Pathak on X:

• This is showing fully end-to-end learning on a real industrial task

• It’s just using cameras - no fancy, expensive sensors with arduous calibration procedures

• Long-horizon reasoning and action via in-context learning

• Robust to disturbances and easy to deploy for new tasks

According to the booth staff, they had 15-20 hours of real robot data for this demo, as well as use of NVIDIA Isaac Lab with Newton; and in fact we can see the task simulation in the NVIDIA promo videos for Newton.

Noble Machines Shows End to End Humanoids

Noble Machines just recently came out of stealth showing their brand-new industrial humanoid.

They showed demonstrations of whole body control for robot teleoperation, as well as a couple autonomous manipulation policies: picking and placing 2x4’s and pipes, and loading and moving a tote. Both worked flawlessly at least while I was watching.

I also like how this robot is wearing shoes; it might hurt motion aesthetics (because it’s unlikely to be modeled properly in Isaac) but it’s an easy solution to the incredible difficulty of some shop and real-world environments. Shoes are “ablative” and easy to replace, and they have a lot of grip on the floor by design.

Watch some more video of it in action:

Generalist Shows Incredible Dexterity

Robotics learning startup Generalist has quietly been building up impressive technology. In the Universal Robots booth, they showed their manipulation capabilities live. The task: putting a phone in its box. Some impressive details:

• They only had the robots for a few days to prepare for the demo - this was a bit last minute

• They claimed mere hours of on-embodiment data - a very fast setup process, though they likely had quite a but more data from the same task on other robots

They were constantly doing things like hitting the robots with hockey sticks, interrupting the manipulation and watching it recover.

Generalist has since gone on a bit of a press tour, so I will leave further details to them.

Agile Humanoids

Agile showed both humanoid robots performing a range of manipulation tasks, and a teleoperation system. Agile Robots is one of the small but growing number of European companies in the space, showing how the robotics race is going global.

The Unitree H2, In Person

My first time seeing this robot in person. I think this will be huge — likely the biggest humanoid robot in the world in 1-2 years.

You can also see it fighting at GTC — check it out on X.

Pony AI Car

Self-driving cars continue to scale up. I won’t say much about this, but you can check out my previous blog posts on their competitors like Baidu and Waymo.

Medical Robotics

We don’t see as much work on medical robotics making waves on social media. Medical robots don’t make for flashy videos, but they’re extremely important

Cheap Robots

Seeed studio builds a variety of open-source and low-cost robotics hardware, powered by NVIDIA Jetson. It’s interesting to see this here, as GTC is mostly aimed at larger businesses.

They are preparing to launch re-Bot, which is a larger, higher-payload arm more similar to the common YAM arm. The availability of extremely cheap open-source hardware like this will make robotics research and experimentation far more accessible, which is a bright sign for the future.

Olaf Walks The Floor

NVIDIA and Disney’s robotic Olaf the Snowman looked fantastic, like a Disney character come to life.

Interactive Humanoid Demo

Humanoid had an interactive demo running in the exhibit hall, where you could speak into a microphone and ask the robot to bring you something. This company has been moving incredibly fast to get their robots up and running and start early deployments.

Dexmate Robot Wanders the Hall

Dexmate didn’t have a specific booth, but they still brought a robot. The Santa Clara-based company is one of relatively few aiming to build a platform, with high-quality general-purpose hardware that people can build applications on top of. The Dexmate Vega itself looks highly capable, and was safe enough for them to take it on the show floor.

I liked seeing the teleop rig they were using; it’s easy to imagine a version of this being used to do mobile manipulation tasks in different locations.

Final Thoughts

Compared to previous years, we saw:

• Far more end-to-end demos

• More mobile manipulation

• More teleop

Just more robots that appeared to work, in general. It’s also striking how these companies are all aggressively pursuing applications now; this did not feel like a crowd of researchers discussing their latest papers. Everything felt very substantial and tangible.

All in all, an exciting look at the current cutting edge in robotics.

Leave a comment

Share

Subscribe now

Even if we should be freaking out about AI, intelligence specifically in robots still has a long way to go — it can take weeks to deploy a robot for a new task. On the other hand, video generation has gotten insanely good recently: Veo 3 from Google and Seedance 2.0 from ByteDance seem likely to change the way Hollywood works forever.

Perhaps as a result of this incredible jump in capabilities, world models — which is to say, video-predictive model for robot tasks — are one of the most exciting trends in robotics right now. They show advanced generalization capabilities, can execute open-ended tasks, and provide an effective way to leverage large amounts of internet data to learn “priors” about how the robot should interact with the world, thus providing an effective way to close the robot data gap.

I’ve written about world models before, so think of this post as an update — looking at the developments over the last few months and a couple novel ways of employing these models, as well as some trends that have developed.

If you like this blog or these kinds of posts, please like, subscribe, and share:

Types of World Models

I’ll go over a few different types of world models that people look at. Note that I am to some extent using my own terminology here; terms like “world model” and “world action model” get used all the time for all sorts of things, so I’m going to give you three different ways of setting up the problem. There are, of course, tons of types and subtypes within each.

Action-conditioned World Models, like V-JEPA 2 [1], which predict future states based on an action vector. These are the most theoretically justified, and are the “model” in “model-based reinforcement learning” — a very well-studied but still nascent area of robot learning.

These (very, very roughly) take the form:

where f is the world model, x are state observations, a are actions, and x’ is the predicted future state sequence (image or video). They are the “classic” formulation of the problem, and have been famously championed by Yann LeCun. Other great examples are Dreamer v4 [2], which learned a “simulation” of minecraft and was able to train policies within that world model; and RISE [3], which is a recent paper that learns a “local” action-conditioned world model and uses it for reinforcement learning.

Next, there are Video World Models, which work by generating a video first, and then work backwards using an “inverse dynamics model” to recover robot actions — DreamGen from the NVIDIA GEAR Lab is an example [4], as is the most recent 1x World Model, and the LingBot VA model from RobbyAnt [5].

These have the advantage of most directly being able to leverage large-scale video data, as they do not need actions at all to train the world model, but instead recover actions from this “inverse dynamics model” g which can be trained on far less data:

Finally, joint World Action Models (WAMs) are policies which predict both video and action jointly -- DreamZero is an example [6], as is Fast WAM [7]. While the term has been used many different ways, I’ll use it here to mean a joint prediction, indicating that it learns both how the world evolves and what actions the robot is taking together:

These models seem to generalize better than alternatives, as explored in the paper “Do World Action Models Generalize Better Than VLAs?” by Zhang et al [8].

Separately, there is also the question of 2D vs. 3D. There are also 3D generative world models - which create a 3D scene, such as PointWorld [9] or Marble from World Labs. This is very broad category, deserving of its own post in the future.

Background: Planning For Robots, or: Why Do We Care About This?

Planning for robots is a field with a long history, dating back over 50 years. People have been using robots to solve complex, long-horizon problems forever. The issue is that these always rely on models of the world, which have historically always been defined by humans.

The key attribute of any of these models is the dynamics model, or how the world state x will evolve in response to the robot’s actions a. From this, all else follows: we can use a good dynamics model to plan over complex terrain and solve challenging symbol manipulation problems.

More recently, large language models have proven themselves to be reasonably good at this kind of thinking, but they remain unable to solve the symbol grounding problem which stymied the old-school planners: basically, what does it mean to “grasp a cup by the handle,” or “be careful, that’s hot,” or really convert symbolic instructions into measurable real-world values.

Fortunately, we have near infinite amounts of video data available to us in order to solve this symbol grounding problem now. And thus, world models.

Action-Conditioned World Models

Above: a screen recording of me using the Interactive World Models [11] online demo.

Action-conditioned world models predict a world state (usually a latent world state) given previous observations and robot actions. Examples include Interactive World Simulator [11], V-JEPA 2 [1], DREAMER v4 [2], and RISE [3]. Also notable is Genie 3 from Google, which generates scenes from a text prompt but then lets you explore them.

The main downside of these models is that they tend to be fairly local; errors accumulate rapidly, and they can’t do really long horizon reasoning. Even if they maintain physical consistency (which they increasingly do!), they will drift away from the real world.

I tend to think of this as the “purest” form of world model, and it is the oldest, despite its complexity and difficulty.

Alternate names:

• learned simulation

• JEPA-style model - technically a subset of action-conditioned world model which relies on specific tricks to prevent representation collapse when training on purely latent-state prediction without reconstruction

• learned dynamics model (as in the end it’s modeling the changes in world state)

Video World Models with Inverse Dynamics

But these action-conditioned world models have a crucial weakness: they fail utterly at long horizons. Prediction quality degrades rapidly. They tend to fail at “long” horizons, even to the order of seconds (though this is getting much better; again see Interactive World Simulator [11] and try the demo).

Another more subtle issue is that they require action labels. Action labels are not available for most video data: we do not know how a human’s finger joints moved to pick up the cup, only what the results looked like.

And so we have a class of models which can take advantage of the massive corpus of video data to predict robot actions. They are “ordinary” video models, trained or fine-tuned on robot data, coupled with an “inverse dynamics model” which takes in successive states and predicts robot actions from them.

I tend to think of NVIDIA’s DreamGen [4] as one of the first outstanding examples in this category, which used video models and an “inverse dynamics model” to extract actions from a fine-tuned version of WAN, though they don’t directly execute the actions but instead used them to generate neural trajectories for training.

LingBot-VA [5] is a cool bridge between these models and our next category. During inference, it interleaves prediction of video frames with an inverse dynamics model which extracts actions from them, making it a hybrid between a pure video world model and a joint world-action model.

Watch the RoboPapers episode on the 1x world model, one example of this category. We also have a podcast on DreamGen.

Joint World Action Models

At different times, people have used the term “world action models” to refer to many things, but in this paper I’ll use it as used in DreamZero to describe the class of “joint world-action models” which predict both state and action without explicit conditioning.

These are probably the current “best” models, with chart-topping results in a couple benchmarks like RoboArena. They have the advantages of both world models (being able to train on and benefit from massive amounts of video) and VLAs (straightforward architecture, direct prediction of actions without jumping through a lot of hoops).

As per above: the main advantage here appears to be that these models can learn from video how the world should evolve implicitly, while still mostly predicting actions. There’s evidence (see Fast WAM [7], above), that decoding video is not at all a necessary step; DreamZero likewise does not decode video at test time, only at train time.

Watch our RoboPapers episode on DreamZero.

Bringing It Together

DualWorld takes the unique approach of combining the “video generation” approach with the much more local JEPA model. It uses a WAN 2.2 5B video generation backbone to predict high-level actions, then uses a JEPA model for local control to track these video references.

This is a cool approach because you can inject, for example, tactile and force information into the more local, short-horizon JEPA model, while getting all of the generalization advantages of the large pretrained video model. Perhaps approaches like this could result in a world model version of “system 1, system 2” reasoning and solve the issues with both approaches.

You can check out their video results here.

Final Thoughts

There are a ton of world model - focused companies out there, all with their own particular version of what a world model “is.” Many of these are not robotics focused. Moonlake AI, for example, raised $28 million in late 2025 to build game worlds, though they have grander ambitions. Yann LeCun’s Advanced Machine Intelligence raised a staggering $1.03 billion for world models, initially targeting smart glasses like Meta’s RayBans.

A note of caution is that world models have, historically, really underperformed expectations. They’re not a new idea, and especially action-conditioned world models have always struggled at producing real value. What’s changed is these things:

• Models, especially video generation models, are much better

• More data is now available for training these models

• New architectures and improvements like video world models and the joint world action models described above

Interestingly, the disadvantages for all of these models are basically the same; as with many trendy methods like sim-to-real learning, they don’t yet achieve the high reliability that we see from on-robot reinforcement learning approaches like RL-100 or pi-0.6*. But this is likely to change, as this is an active area of research, and the models themselves seem so promising. More to come.

References

[1] Assran, M., et al. (2025). V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning. Meta FAIR.

[2] Hafner, D., et al. (2025). Training Agents Inside of Scalable World Models (DreamerV4).

[3] Yang, L., et al. (2026). RISE: Self-Improving Robot Policy with Compositional World Model.

[4] Jang, J., et al. (2025). DreamGen: Unlocking Generalization in Robot Learning through Neural Trajectories. NVIDIA GEAR Lab.

[5] Li, L., et al. (2026). Causal World Modeling for Robot Control (LingBot-VA). Robbyant.

[6] Ye, S., et al. (2026). DreamZero: World Action Models are Zero-shot Policies.

[7] Yuan, T., et al. (2026). Fast-WAM: Do World Action Models Need Test-time Future Imagination?

[8] Zhang, Q., et al. (2026). Do World Action Models Generalize Better than VLAs? A Robustness Study.

[9] Chao, Y.-W., et al. (2026). PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulation.

[10] Fikes, R. E., & Nilsson, N. J. (1971). STRIPS: A New Approach to the Application of Theorem Proving to Problem Solving. Artificial Intelligence, 2(3-4), 189-208.

[11] Wang, Y., et al. (2025). Interactive World Simulator for Robot Policy Training and Evaluation. Columbia University, Toyota Research Institute, Amazon, UIUC.

Citations produced by Gemini from the text of this blog and may have errors.

Learn More

A great writeup from the Moonlake team on world models on X

Interactive World Models Paper by Yixuan Wang et al.

Dual world: https://world-mind.github.io/DualWorld/

Some previous blog posts and podcasts from me that you can read or listen to:

RoboPapers

Ep#61: 1x World Model

Every home is different. That means that to build a useful home robot, we must be able to perform zero-shot manipulation of a wide range of tasks — which is a real challenge for robotics, since so many cutting-edge approaches require expert fine tuning on a small set of in-domain data…

Listen now

7 months ago · 4 likes · Chris Paxton

Data-first home robot company Sunday Robotics announced their series B raise for a unique and contrarian take on how to deploy in-home robots today. Their approach claims zero robot data, but still focuses on real-world-data only. This shouldn’t be a huge surprise; co-founder Cheng Chi was behind the game-changing Universal Manipulation Interface (UMI) paper. The idea is to use a tool, an exact duplicate of which is mounted on the robot arm, to collect all the robot data.

A short blog post here with some thoughts on what makes this company and their vision of how to build robotics unique. This is a “quick reaction” blog post, so apologies for any errors in advance. If you like my writing please like and subscribe.

Subscribe now

Why Zero Robot Data

Part of what makes the Sunday vision of robotics unique is their focus on glove-style data collection. Robot data is expensive, as their head of product says on X.

To collect real-world data, you need:

• Skilled operators

• Real robots, at real-robot prices

• Good teleoperation setups, which are often clones of the whole robot

• To be able to move this around into different environments to collect data

All of this is rather expensive. With tools, assuming this data is as good as real robot data, all you need is the first part - the skilled operators. This is something Sunday does; by skipping the latter three requirements, they can jump directly to paying for high-quality data at scale.

And Sunday pays very well for data, up to $60 per hour according to public job postings — likely more for other positions and for specific customer requirements.

The fact that tools are easier to carry around is also important. The key to scaling robotics learning is data diversity, something I have written about before. Tools let you scale data easily, because you can just walk around and use the tools for anything you want. There are a few Robopapers episodes with a similar premise you can listen to, if interested in this line of reasoning.

Despite how we’ve repeatedly seen that diverse scenes and tasks are important, it remains very hard to get a good distribution of data just with teleoperation data. While other approaches exist — like large-scale human video data — all of these approaches have tradeoffs. My (boring) personal take is that we’ll probably use many different data sources together, but I like to see that Sunday is committing to an extreme to see how far it can take them.

Rapid Hardware Iteration

So, how does this work in practice?

The Sunday Robotics approach seems to have centered on fast hardware iteration. Above you can see some images — shared on X — of how they iterated on the gripper. It’s a unique 3 finger design, a “mitt” which can be exactly mounted on the end of the robot. In addition, data collectors wear a “hat”, similar to the baseball hat worn by their robot, Memo, which captures wide field-of-view RGB camera images.

Their final glove is around $400, which means they can afford to have a distributed network of roughly 500 operators collecting high-quality, diverse data at massive scale.

In public comments, the company has discussed how much time and effort was spent just on data collection - they spent much of their time building the tools and collecting data, and comparatively little time training their models.

Finally, they constantly point out the importance of building a full-stack team: that iterating hardware, data, and training as a whole is crucial to deploying real-world general-purpose robots.

Final Thoughts

This is a very data-first version of robotics. They have taken a strong bet that good data at scale is crucial, and that the particular type of data that can be collected with UMI tools is ideal. It’s an interesting bet and a promising one; I’m interested to see how it goes and hope to see Memo out and about in the real world soon.

Leave a comment

Share

Subscribe now

Something strange has happened to software development over the last year: the machines have taken over. Software development is faster and easier than ever with the aid of tools like Claude Code and Cursor. Similar changes are coming in other industries; AI tool use will come for computer-aided design (CAD) tools next; it will make it easier to design a robot or other hardware in software, write the bridges to connect it to simulation, and deploy it.

This challenges a couple major assumptions that have been driving massive robotics investment over the last few years, so I think it’s worth exploring the idea.

Subscribe now

The Case for One Robot

It’s a cliche that you have the same iPhone and the same laptop as Elon Musk. Scale, right now, is king in software and hardware. Electronics products are impeccably engineered, optimized, and designed for manufacturing at scale.

One might assume the same logic would apply for robotics; in fact, one might assume it would be even more extreme for robotics. This is because robots need real-world data; robot models are data hungry, and there is a huge data gap to be overcome.

Co-designing robot sensors, hardware, and models is already very important for good real-world performance. This means that deploying one robot everywhere, an approach championed by Tesla and Figure, has a lot of appeal. You solve these hard problems once, then scale, scale, scale until the robot is affordable and incredibly capable. There are clear parallels to how humans “work,” as in how humans are employed in a wide variety of industries and to solve a wide variety of problems.

This is only sort of how humans work, though. Humans use tools. Incredibly inhuman automation is vastly more productive than humans for a wide variety of problems, from package sorting to harvesting crops. Humans provide the intelligence, but our bodies increasingly are not actually performing the labor — our minds are.

The humanoid-only champions have an answer here: that the robots will be using human tools. This is something that I am deeply skeptical of: if robots are doing all the work, why would you design tools to support legacy hardware (humans) forever?

Initial Conditions

First, let’s start with a few assumptions:

• Software development will become effectively “free” over the next year, meaning that you can use AI coding agents to produce any code you can clearly design, given about a month and a few thousand USD.

• Simulations will continue to improve, to the point where they’re operationally indistinguishable from reality, at least on a mechanical level, given some real-world data to “seed” them.

• Cross-embodiment learning — a “single brain” for all robots — works reasonably well, and it’s not hard to deliver intelligence for a new form factor.

• AI agents will improve their ability to use CAD tools for hardware and electrical design, and can “close the loop” with a simulation like NVIDIA Isaac, Mujoco, or potentially a new upstart like Lucky.

What would be possible in such a world?

The Robotic Cambrian Explosion

The Cambrian Explosion was a rapid evolutionary event, a period roughly 539-520 million years ago when a vast number of animal phyla appeared in the fossil record for the first time.

If these conditions exist — if agents can be trusted with hardware design, if they can be given reasonably-accurate simulations of real-world tasks, then instead of seeing a few humanoids appear everywhere, we would see a massive diversity of different robot form factors.

After all, the motors that power a robot dog and a humanoid and an industrial pick-and-place machine don’t need to be all that different. One might easily imagine a world where a robotics company builds modular components like LimX’s TRON 2, which can be recombined in many different ways according to the needs of a particular task. They might go farther, with a list of 4-5 supported actuators and a handful of common sensors, with shells and wiring harnesses designed on demand.

This would mean that instead of a single humanoid in a factory, you would see a family of dozens of interrelated robots; big cargo haulers, fast-moving inventory-checkers, loaders and packers, all build out of the same core parts and on the same hardware. If you want one for your home, maybe it’s using lower-powered actuators, and cheaper sensors, but again it’s still the same mind and the same parts.

Importantly, if cross-embodiment learning works — which, to an extent, it seems to — these robots are still all sharing data and still making each other more intelligent.

This to me seems like a much more “AI-first” way of viewing robotics hardware development, even if in many ways it’s still far fetched.

Why It Might Not Happen

A few potential issues:

Maybe contact-rich simulation remains intractable. Despite the best efforts of NVIDIA and World Labs and so on, perhaps contact physics never become quite good enough to fully automate the design process. This would make the economics much worse for our robotics Cambrian Explosion: you would need much more rigorous real-world testing, with more room for human engineering and intuition in the process.

Human data is more useful for humanoids. But we’ve actually seen from recent works that you can get incredibly good results just with in-domain data, and there’s also a large and growing body of work on cross-embodiment learning.

In-domain data might be too important. Basically, cross-embodiment learning might be good enough for a demo, but what if it’s never good enough for true, high-reliability cross platform performance? This would, again, make the economics of this Cambrian Explosion harder, since you’d need lots of real-world data for each one (if it can all be done in simulation, you’re fine).

And there’s some evidence for this one: in a recent blog post from Physical Intelligence, we see that they achieved their best results by mixing downstream customer data into the pretraining for their foundation model. Similarly, for the NVIDIA example above, cross-embodiment robot data may or may not have helped any more than other egocentric data.

This is a pattern we have actually seen many times; robotics problems tend to be these little isolated “islands” of data with less evidence of positive transfer between them than you would hope. On the other hand, this will almost certainly improve as we “fill in the gaps” with more robot deployments.

Final Thoughts

The truth, likely, will be a mix between the two: some general-purpose platforms, and a great deal more robots for a wide variety of tasks. For now, at least, real-world data seems irreplaceable, but this could all change fast, and it constantly is.

The real test, I think, will be if data-driven simulation really takes off and proves that it can handle complex contact dynamics. If this happens, it will be much easier to train specialized robots in simulation. This reduces the risk and means you can produce a much wider range of robots without all the expensive real-world deployments and data collection.

But please let me know what you think below.

Subscribe now

Share

Leave a comment

The team at NVIDIA recently released a set of robotics models; one that’s been driving a lot of discussion online lately has been DreamZero: a “World-Action Model” which predicts video in addition to robot actions. In particular, it achieved benchmark-topping results at two very interesting benchmarks this week: RoboArena and MolmoSpaces:

Let’s briefly go over what this means and what we might learn from these results. In this blog post I write:

• A very high level look at what makes DreamZero different from other policies and world models

• A brief look at RoboArena and MolmoSpaces

• Some speculation about what’s different and what might be causing performance gains

A lot of this was pulled from discussions on X. Upcoming content: we will have a RoboPapers podcast on DreamZero in the next few weeks, and next week on this blog I should have a more in-depth analysis of world models and how the landscape has changed in 2026.

What is DreamZero?

DreamZero is a “world-action model” from NVIDIA. In practice, this takes a lot of the concepts from world models — principally that video generation is useful for robotics — but changes the formula in a couple key ways. Most crucially, it jointly models action generation and video generation.

Usually, there are two classes of world models:

• Action-conditioned world models learn the mapping x’ = f(x, a), for state observations x and actions a. An example would be V-JEPA 2 or the world model from the recent RISE paper.

• “Inverse dynamics” world models like NVIDIA’s DreamGen or the 1x world model (RoboPapers podcast for DreamGen, RoboPapers podcast for 1x), which learn x’ = f(x) and then an inverse dynamics model a = g(x, x’)

This model is much more like a traditional robot policy, but also predicts future video - so we are learning something more like (x’, a) = f(x). All of these are a huge oversimplification of the true training objectives, of course — read the paper for details.

You can also compare to a traditional vision-language-action model: we are also predicting what the future will look like. This gives a much richer supervisory signal over how the world will evolve going forward.

The Benchmarks

RoboArena is a distributed “real world” benchmark based on Droid. Evaluators across the world have somewhat similar robots and setups, and run a set of open-ended robotics evaluations with different natural-language commands.

This means that it’s somewhat in-distribution for DreamZero, as it was trained on Droid, which contains very similar tasks and setups. But it’s still a real-world evaluation, with all the difficulties and changes that this implies; and the tasks are chosen by evaluators. It’s also a “head-to-head” comparison benchmark, similar to the Chatbot Arena which was so influential in LLM development. I wrote about this at length in my post on robot learning evaluation.

MolmoSpaces is a new benchmark, with high-fidelity physics and diverse, procedurally-generated environments. MolmoSpaces-Bench, in particular, is a test of controlled variation across a variety of pick, place, open and close tasks, plus compositions thereof.

It’s a new benchmark that’s definitely not close to saturated yet. And DreamZero performs well on both:

What Can We Learn?

Let’s compare pi-0.5 with DreamZero, specifically, as pi-0.5 is the second best.

Training data: pi 0.5 was trained on 10k+ hours of real robot data, VLM data, and data from the Droid dataset, as Mahi Shafiullah points out on X. DreamZero was trained on either DROID data or Agibot data, depending on the model checkpoint.

Training data distribution likely makes a huge difference here. Notice how in the DreamZero paper, Dreamzero radically outperforms pi-0.5 on AgiBot (not represented in the pi-0.5 training data) but is much closer on the shared DROID-Franka setup:

This also seems to suggest to me that possibly the additional 10k hours of robot data is not as useful as people think. Pretraining on the right robot data does seem very important; in another recent blog post, Physical Intelligence showed a massive gain from pretraining on in-distribution partner data:

So, maybe adding in an extra 10k hours from a different robot is actually no better than whatever random video data you happen to have. This would likely be a bad result for people who want to train cross-body robot brains. Perhaps you’re not getting anything from those other robot embodiments that you wouldn’t get from just adding cheap, plentiful egocentric video data.

Model backbone: DreamZero is based on Wan2.1-I2V-14B-480P, a 14B-parameter video generation model. It can take up to 8 frames of context. By contrast, pi-0.5 was trained based on the 3B-parameter PaliGemma open-source VLM. So, a much smaller model. It’s also one that takes only a single frame instead of an 8-frame history. This has ramifications for any task which is partially observable or has hard-to-model dynamics, which is to say any real robotics task.

Model size: DreamZero is a huge model, and much of the work in the paper is actually around making this massive, 14B parameter model run in real time. Ablations in the paper seem to show that model size makes a very large difference here:

The problem with both including more history, and with increasing model size, is that this would usually make the model both harder to train and cause it to “overfit” more in a low-data regime. Unlike LLMs, where overfitting really isn’t an issue due to abundance of data, in robotics we’re essentially always operating in a low-data environment, even now. DROID is tiny compared to even the smallest LLM datasets.

So, a proposal: the video-generation objective functions as an “auxiliary loss,” forcing structure on the DreamZero model and causing it to learn some internal model of the world. It provides a comparatively strong form of supervision compared to the sparse signal from robot actions. This in turn makes it better able to adapt to the varied MolmoSpaces environments that it was not trained on.

Final Thoughts

There is a lot we can’t determine just from these papers. We don’t have access to all the data that Physical Intelligence used; it’s still hard to access the GB200s that NVIDIA uses for inference. But to me, it’s exciting to see that perhaps we don’t need nearly as much data as was previously thought to get strong real-world robot results, and I hope to see more work in this direction.

One last question I have is this: if video generation is “just” an auxiliary loss, how long will we need it? In a world with thousands of intelligent robots deployed, will this eventually be distilled away, and will all of these models just be beaten by dedicated robot video models with robot-specific fine tuning? This seems very likely, but is potentially a couple years off.

Please let me know what you think below.

Subscribe now

Share

Leave a comment

Let me start with a vision of a potential future:

You wake up with a jolt in your foxhole, a little dug-out somewhere in the no-man’s land between two armies. A light is flashing on your beat-up tablet.

Maybe you hope for a bit that it’s a message from home, or even from command -- it’s been weeks since you’ve seen any other humans on your side of the war. Maybe there aren’t many of you left.

It’s not, though. It’s the enemy, moving through the trees a few dozen kilometers away, spotted by a little drone the size of your hand. At your command, a small army flies into action: the rotors of bomber-drones whir to life, the rolling autonomous gun-platforms taking up strategic positions in cover and to cut off any chance of escape. Robot dogs carrying guns sweep into nearby buildings, scuttling up bombed-out stairwells and setting up lines of fire. Anti-personnel drones loaded with explosives swarm through the trees.

Kill them all, you say, and your robotic troops spring into action. They know exactly what to do, coordinated by an artificial intelligence with superhuman ability to coordinate disparate data sources and make decisions.

You lie down and close your eyes. When you wake back up, the enemy are all dead.

In the battlefields of eastern Ukraine, drones are increasingly becoming autonomous. When given a target — a Russian soldier or vehicle — they will “lock on” and use their on-board artificial intelligence to pursue and kill that target. At the same time, defense startups have begun building combat-ready robots with increasing levels of autonomy.

In addition, the whole business of fighting a war seems liable for automation. The kill chain — the sequence of information-gathering, decision making, and action-taking that leads to deploying lethal force — is well suited for artificial intelligence.

Let’s talk about how and why.

Subscribe now

What is the Kill Chain?

In a war, soldiers do not cleanly agree to meet in a small area and shoot each other, as in a video game. Instead, enemies must be found; they must be identified; assets must be allocated to engage them; and then finally munitions must be used to destroy them. This is the “kill chain,” and it stretches from generals in Kyiv down to the men in the trenches on the front lines of the Ukraine war, or from US Navy admirals down to the sailors who fire missiles at boats in the Caribbean.

Large parts of this, especially in a modern military environment, look a lot like the sort of data analysis tasks for which AI is particularly suited. AI models like Anthropic’s Claude can analyze the massive amount of data produced by a modern military, identify trends, and respond faster and — one hopes — more accurately than human analysts can.

This has led the US military to use Claude for military operations, like the US strike on Venezuela that led to the capture of President Maduro. It’s also led to conflict this week between the Pentagon and Anthropic, as Anthropic has a “red line”: Claude cannot be used to make decisions to kill people on its own.

AI Agents for Combat

Deploying drone swarms will require autonomy. These simply cannot be remotely piloted in meaningful numbers, as right now, for the common FPV drones that inflict so many casualties in Ukraine, you need a single pilot per vehicle.

All of this will change. As autonomy gets better, it will be possible for one person to manage multiple drones, say in order to strike a position multiple times in rapid succession to pin them down or hit enemies attempting to reposition. One pilot will manage three drones, then ten, then a hundred.

And it won’t stop with drones. Some of these robots will be armed ground vehicles. We’ve seen recently companies like Tencore pitching their ground robots as part of combined forces which escort and work with humans on the ground, protecting them from aerial drones. Countering this threat will require superhuman reaction times, and likely push these systems to have more and more decision making authority on their own.

One of the clearest visions came from Scout AI co-founder and CEO Colby Adcock, who proudly announced the Fury Advanced Vehicle Orchestrator in an X post. This is an agentic system which can take a natural-language command and turn it into action. In their demo, they show a human commander giving a high-level command, and then AI agents orchestrating a response: finding, fixing, and destroying a target vehicle using a combination of ground and aerial robots.

To understand what’s different about Scout’s approach with Fury, you also need to understand that it’s using a fundamentally different method than military robots in the past, instead being based on a family of Vision-Language-Action foundation models.

These models can consume video feeds and sensory data from different robots, and aggregate them to build a single unified picture of a battlefield with which to make decisions. The commander says “find and destroy the red vehicle,” and the robots execute: they gather information, process it, and complete the mission.

Final Thoughts

For now, these robots are still semi-autonomous at most. The majority of combat robots in Ukraine are under full human control: they are remotely operated, their purpose is to keep a human soldier out of the line of fire while he does his job. The semi-autonomous drones in Ukraine right now are not terribly intelligent; they’re still not much smarter than a guided missile. The robot ground vehicles, as I’ve written about before, are largely holding trench lines and conducting ambushes. It will be a long journey before Fury is much more than a demo video for a VC pitch deck.

Besides, to an extent, autonomous weapons have always been with us. A land or sea mine is a crude version of an autonomous weapon, wired to blow when an enemy (or anyone else) makes contact with it.

Mines are also an example of the realistic failings of such systems: they make mistakes. A lot of mistakes. People will die for decades after a conflict ends due to mines or encounters with previously-unexploded munitions. Perhaps autonomous robots will be the same; perhaps when demolishing some derelict building ten years after the end of the Ukraine war, a robot sentry will activate and gun down a handful of workers.

There are other concerns, too. From Dario Amodei:

Of course, these weapons also have legitimate uses in the defense of democracy: they have been key to defending Ukraine and would likely be key to defending Taiwan. But they are a dangerous weapon to wield: we should worry about them in the hands of autocracies, but also worry that because they are so powerful, with so little accountability, there is a greatly increased risk of democratic governments turning them against their own people to seize power.

Autonomous military forces will require good governance, keeping them accountable and strictly under the control of civilian authorities. AI agents for military operations radically increase “AI risk.”

There are positives to these systems, too, of course — at least distant, potential positives. By taking human warfighters out of the direct line-of-fire, they could save lives on both sides: robotic systems are more disposable, after all. If a situation is ambiguous, it’s much more reasonable to lose a few $300,000 robot ground vehicles than a few of your human soldiers, so maybe you give the order to hold fire and wait for more information. These systems can also make decisions faster, using more information, and will perhaps one day be able to make those decisions better and with less error, matching the trends we have seen in other parts of society.

This is a uniquely strange and risky area for robotics, though. I write this mostly because I think it’s important everyone understand that this is here, and it is happening today.

Please like, share, and let me know what you think below.

Share

Leave a comment

Learn More

There was a saying popular in Silicon Valley around the mid 2010s: “The last driver’s license holder is already born.” There is even a blog using this as its name. After a lull, it seems finally clear that this statement is, more or less, true in 2026. Ten more years of this and no one will be driving their cars.

Right now, there are two big trends in self-driving as I see it:

• At the high end, Waymo (and its competitors) are offering a fully autonomous taxi service which scales quickly to new cities, is offering millions of rides a year, and is a full replacement for the taxi.

• Automakers like BYD and companies like Wayve are increasingly offering high-quality self-driving capabilities for consumer vehicles, in a similar vein to Tesla’s Full Self Driving.

Not only are cars getting more intelligent, but more of them don’t even need humans. And this extends beyond personal automobiles to semi trucks, with Aurora recently bragging they can make the 15 hour trip from Phoenix to Fort Worth autonomously. In China, companies like Neolix have begun deploying self-driving delivery vans on roads (to sometimes hilarious results).

In this blog post, I’ll go over some of the recent news on self driving vehciles. If you’re interested in this or other robotics content, please subscribe below.

Waymo Continues To Expand

Waymo has at many times in the past faced skepticism that it could scale quickly. But this concern looks increasingly dated; they’re currently deployed in 10 cities, either through Uber or their own app. They’ve planned expansions to over 20 additional cities, including cold-weather cities like Buffalo. To fuel all this growth, they recently raised a $16 billion funding round. The scale of the operation is unmatched outside China (Baidu is the leader globally). To couple with this expansion, they’ve announced their next-generation vehicle, the Waymo Ojai, built by Chinese automaker Geely, which should greatly bring down per-unit costs while offering a spacious interior.

Waymo’s Remote Operators

There’s been a lot of discussion lately about Waymo and its use of remote operators, especially after recent testimony to Congress. Waymo claims to use a surprisingly small number of operators (or “assistants” to use their preferred term): just 70 for a fleet of 3000+ autonomous vehicles.

These operators’ job is not to drive the vehicle. Instead, the operators will usually give fairly high-level instructions: telling a vehicle it can ignore an obstacle in the road, for example, or giving a sequence of waypoints for the vehicle to follow in order to get it out of a tight spot.

This system is not always perfect. A serious stress-test for Waymo happened during PG&E outages which resulted in traffic lights going dark:

Waymo vehicles stopped all over the city, blocking intersections and causing serious trouble for emergency vehicles. This was due to a spike in requests for help going back to their command centers from the autonomous vehicles, which seems to have taken down their system. They claim to have implemented new protocols to prevent this from happening again.

Likely, we will see growing pains like this continue. But remember this: we absolutely do know that Waymo is a lot safer than a human driver. The case for autonomy is clear.

Autonomy in Consumer Cars

Tesla’s autopilot continues to improve. But it’s not just Tesla: we see more and more consumer vehicles adding autonomy features — Rivian Driver+ and especially BYD God’s Eye are growing increasingly capable.

The case of BYD is particularly interesting. BYD is the world’s biggest automaker now, and its vehicles are consistently well-reviewed and well-liked. When it rolled out its God’s Eye 5.0 self-driving software, it did so across 2.3 million vehicles in China alone. God’s Eye uses a mixture of camera, ultrasonic, and radar sensors to achieve safe, reliable driver assistance.

According to BYD, this generates 160 million kilometers of driving data every day.

Legacy automakers are following along. Nissan, for example, is working with Wayve to deploy its self-driving software as ProPILOT on their new vehicles starting in 2027. Wayve has set ambitious goals, showing that its system is capable of driving in cities all over the world in 2025 — such as in their “90 cities in 90 days” initiative. This approach works without maps; they visited both to rural areas and to dense cities.

Wayve has a camera-only autonomy stack. What’s interesting about the Nissan approach is that it adds radar and lidar for “Ground Truth Perception,” adding extra safety guarantees on top of the camera-based autonomy from the Wayve Driver. With the low cost of lidar and radar, this would not be expected to add much to the cost of the vehicle; and the extra safety guarantees may be worth it.

Tesla Robotaxis in Austin

And of course, when speaking of consumer cars, we must proceed to discuss the elephant in the room: Tesla. Tesla has their own robotaxi service, and they recently rolled the first fully-autonomous, steering-wheel-free Cybercab off of their production line in Gigafactory Texas. They’re planning to expand their robotaxi program to seven new cities in the first half of 2026, up from just Austin and San Francisco.

However, Tesla has seen a spate of bad news lately. From AV Market Strategist:

The new crashes, from December 2025 and January 2026, include collisions with a fixed object at 17 mph, a bus while stationary, a heavy truck at 4 mph, and two backing incidents at 1-2 mph into a pole and a fixed object. Buried in the updated data is a revised report for a July 2025 crash that Tesla originally filed as “property damage only.” Five months later, Tesla quietly upgraded the injury severity to “Minor W/ Hospitalization.”

It’s not been all good news for Tesla in other areas either - for example, we saw a recent $243 million judgement against Tesla for an accident in Florida involving autopilot.

Tesla is taking a unique, extremely low cost approach, building a very cheap Cybercab that can be rapidly scaled. If the Waymo Ojai is in the $70k to $80k range, as I and others have estimated, the Cybercab is going to be probably a quarter of the price. This is a huge win for them, and is a result of a car with substantially less on board sensing and compute.

The big question is whether or not this is possible. Humans, after all, use quite a lot of compute to drive, at least; our brains are insanely powerful thinking machines. Vision-based autonomy is clearly possible, but Tesla might have trouble especially if they need to continue producing models for legacy hardware.

Still, the vision is incredible: I loved this video of a Tesla delivering itself to its new owner, for example; Tesla continues to sell the vision of the self-driving car better than anyone else.

Final Thoughts

Despite a few bumps in the metaphorical road, the progress towards ubiquitous self-driving cars seems possible. There are many forces driving this change: not just Waymo, Baidu, and the Tesla Cybercab, but the fact that the general level of autonomy on all cars seems to be going up year over year.

And I can’t wait. There are so many horrific stories like this one caused by people driving and making mistakes that cost lives. The world will be better off when this technology is everywhere. Again, these cars are a lot safer than a human driver. And with more and more cities seeing autonomous vehicles, I think a lot of the public opposition to these cars will fade away, defeated by the sheer convenience and the quality of the experience.

I hope this year we see honest, successful, high-quality competition for Waymo — from Nuro, Zoox, and Tesla — but this seems farther off, as all three companies seem significantly behind in the United States.

But my main prediction for 2026 is this: by the end of the year, self-driving cars will be boring. People’s fears will be largely assuaged, the recipe for scaling will be clear, and the technology will be broadly acknowledged as solved.

Share

Leave a comment

Learn More

The AV Market Strategist

Waymo's Shocking Data Release, Uber Invest $100M in Robotaxi Infrastructure

👋 Welcome back everyone…

Read more

6 months ago · 17 likes · 1 comment

We all have too many papers to read; in the past I have just dumped a bunch of related papers together with some light summary, to say why they’re interesting. This paper roundup is on evaluation and benchmarking, which has been a theme for me lately.

Benchmarking robotics is hard; collecting robotics datasets you can actually use is hard. This blog post is a short overview of a few datasets that might be of interest, both lesser-known ones that struck me as interesting and a few large and well-known ones for context.

You can also use this blog post to scroll through and see what the datasets people are talking about in a RoboPapers episode, actually look like.

If you like reading thoughts on robotics, please consider subscribing. Usually I write more in-depth summaries; this particular blog post is much more minimal and “stream of consciousness” than usual.

Subscribe now

Datasets

Open X Embodiment

An absolutely huge dataset of robotics data. At the time of initial publishing, it had more than 500 skills and 150,000 tasks included, with a wide variety of robot embodiments. It won the ICRA 2024 best paper, and has about half a million authors. Find it here.

DROID

An important subset of Open X Embodiment, and a dataset that’s still heavily used today. Partly because the hardware setup (above) was duplicated across so many universities. Contains a mix of office, kitchen, and home environments, with a Franka Panda arm and a zed camera on the wrist, plus a number of third-person views.

Check out the website for more.

Build: 10k Hours of Egocentric Data

The Build team has released 10,000 hours of video data. 192,900 monocular wide-angle egocentric video clips, collected by human workers in a variety of different environments.

You can find the data here on HuggingFace. And look at Eddy Xu’s X thread.

OpenEgo: 1107 Hours of Egocentric Data

600+ environments with 120 million frames. Task, language, and 3D hand pose annotations. Often these datasets do not have annotations, so this part is potentially really valuable.

Check out the original X thread from Ahad Jawaid. Github site for OpenEgo.

ShareRobot: Task Planning

A task planning dataset, with complex multi-step tasks and question answering. You can find it on HuggingFace here.

HUMOTO: Human Object Interactions

A large interesting dataset of human-object interactions. Not sure how useful it is, but given how much object interaction in whole-body robot control has accelerated recently, this seems worth a look. You can check it out here.

Benchmarks

Libero

You probably know this one; it’s one of the biggest.

With 130 tasks spread out across 4 task suites, it has a lot of variety, and has been used in a ton of research papers so far. The code is also open source.

Contains datasets with RGB information from wrist cameras, proprioception data, language, and PDDL scene descriptions here. These are all high-quality human teleop data, making this benchmark highly suited for learning-from-demonstration research.

CALVIN

A commonly-used benchmark by Oier Mees for language-conditioned task execution. It uses the Franka Panda robot and has a simple setup with some light variation in objects and coloration, together with a good variety in pick and place tasks. Find it on Github here.

RoboEval

A benchmark of 10 base tasks and 3000+ demonstrations which aims, somewhat uniquely, to provide unique, non-binary metrics of performance, in order to give an idea of how and why experiments are failing: there are trajectory smoothness metrics, it tracks environment collisions, etc.

Check it out here. Thread with thoughts; original X thread.

RoboArena

Real-world benchmarking via web socket to your policy. Grad students run evaluations in different university setups.

I’ve written about this one before, when writing about evaluations. I think it’s one of the best ideas for policy evaluation I’ve seen; though it’s still woefully limited in many ways.

See the X thread here, or check out the project site.

PolaRiS

A benchmark specifically aimed at sim-to-real correlation. Scan an environment with your phone, use Gaussian splatting to construct a scene, and use a set of tools to create a photorealistic simulation. Ships with a set of “real-to-sim” environments in which performance correlation with the real world was validated.

Check it out here. Robopapers episode.

REALM: Real-to-Sim Validated Benchmark

A benchmark specifically aiming to close the sim-to-real gap. Sedlacek et al. put together this benchmark designed to be a good simulation that closely represents the real world. See the post on this from Evangelos Kazakos on BlueSky.

Physics IQ

A video understanding benchmark - but of course video understanding is very closely related to robotics. Basically, how well can video models understand certain physical problems?

Check out the website and the thread on X. There was also an ICCV 25 challenge and associated workshop.

Conclusions

There are a lot of benchmarks and datasets out there; comparatively few (1) have gotten real uptake, and (2) show some actual correlation with real-world results. This is still an active area, and I personally am very excited about potential for real-world benchmarking like RoboArena (though it’s very expensive).

Ultimately the best benchmark is the real world — fortunately, with common platforms like the Unitree G1, we can expect people to open-source code that others can immediately use and deploy.

Subscribe now

Share

Leave a comment

Learn More

Sometimes, you need to raise a round for your robotics startup and the training run just didn’t go so well. Or maybe the CoRL deadline is a week away and the results just aren’t there. Never fear; you have options. Follow this guide and everything will work out just fine.

Let’s discuss how to make your model look as good as possible.

1. Don’t let other people run comparisons. Remember what happened with Llama 4: if other people can try your model out, they’ll quickly uncover its limitations. If you can keep your model secret, that’s best.

2. Control the environment of your demo carefully. Lighting, objects, initial robot configuration, and so on. This lets you overfit the demo scene and get really nice, smooth, high quality motions.

3. Never show your failures. This one might seem obvious — why would I show failures? — but if people can see where your model fails, they can start to see the limits of what you can do. Only strong robotics papers and results can show failures with confidence.

4. If you have to let other people run comparisons, choose the people carefully. Make sure they’re only happening in the right circumstances, ones your model supports as roughly in distribution. Absolutely don’t do what Physical Intelligence or NVIDIA do and open source your model so anyone can benchmark it.

5. When working on the results section of your research paper or blog post, you may be tempted to include some baselines. This is a good idea; just be careful to choose weak baselines so you look good. Octo is a great choice here; it was well publicized but had lots of limitations that weren’t widely discussed.

6. On the same note: cherry-pick your benchmarks. There are a ton of robotics benchmarks out there, and they all test subtly different things. Importantly, these differences are not obvious to people who are not familiar with the benchmarks involved.

Now, you may be thinking: “Chris, this is all great advice; but people will call me out if I follow these rules.” Don’t worry about that. There are so many little things which influence robotics performance: camera placement, arm configuration, object diversity, low-level controller implementation, and so on.

As long as you can make an argument your setup has to be slightly different from everyone else’s, you can get away with a lot. For example; a benchmark I like is RLBench, which turns out to be a very difficult one for many VLAs, and many successful methods on this benchmark instead use motion planning together with a higher-level goal prediction model.

Second, as a result of the above, robotics benchmarking standards are quite low relative to other areas of machine learning. Papers from very famous roboticists get away with these things all the time, and they’re under much more scrutiny than you are. You’ll be fine.

What to Look For In a Strong Robotics Release

1. Open source code and models. Not always possible, but always welcome.

2. Very diverse scenes and environments. Modern robotics learning methods are very, very good at overfitting to a small task distribution — a clean table, objects at most a few centimeters from where they started.

3. Don’t be afraid to show failures. We all know robotics methods fail a lot; showing these is a strong signal, and also helps qualify where it works and where it doesn’t.

4. Compare against the current best methods. Pick the best models head-to-head with embodiments and benchmarks they report on. Don’t cherry-pick.

On the same note, I think paper reviewers must be accepting of some results which aren’t as good as other methods; it must be allowable to fail at some benchmarks, or people will cherry-pick.

Some Caveats

Almost all of the robotics videos and results you see are, I believe, real — in that they’re doing exactly what the creators say they are. The problem is that because it’s so easy to overfit to a particular scene, and because the limitations of a model are so hard to ascertain from a 30 second clip, it’s really hard to tell whether a team is making progress toward the underlying goal of general-purpose embodied intelligence.

And robotics is hard; just because a team is employing some of the tricks I wrote here does not mean their results are invalid or their model is weak. I am certainly guilty of them all, at one time or another! One of the fundamental issues with robotics projects is that there are so many things that influence performance, that it’s very hard to distinguish the signal from the noise.

On the same note: lots of machine learning researchers from other fields don’t understand how hard robotics benchmarking is. They will often insist on building their own, usually simulated, benchmarks, which invariably don’t tell us anything and just add more options for #6.

In the end, robotics benchmarking will be solved by having lots and lots of robots, and models that actually work across most of them. More projects like Lingbot-VLA, DreamZero, and pi-0.5 — models that people can actually try out on different robots, use, and openly compare.

Share

Leave a comment

I wrote about evaluation a bit in the past, and will surely do so again: