There’s this new model floating around called “Ox Alpha”. It’s a stealth model on Open Router, OpenCode Zen, and some other places. This isn’t anything new, we’ve had stealth models before, but some things just don’t line up. For one, the providers have 100T a day worth of allocation of tokens, full zero data retention, the model is very capable, and it’s completely free.

I don’t want to spend too long guessing what this model is, go on X and you will see tons of posts about this already, or just wait a few days and we’ll know. If you want my guess, it’s GLM-5.3 Flash:

Let’s Talk About the Frontier

The real thing I wanted to talk about here is the frontier itself, especially the US v China, Open v Closed source, and the big three labs battling in the US.

Having now spent more time with this Ox Alpha model and getting a more accurate read of it’s DeepSWE performance:

This thing is very clearly not at the level of GPT-5.6 Sol or Claude Fable 5, but it is still very very good. I’m 99% sure it’s a Chinese model, and 80% sure it’s from GLM, and have heard credible rumors that this is a small model (which if that ends up being true will be as big of a deal as Mythos was, but that’s an article for after it comes out).

As such, I think the best model to compare it to is actually Grok 4.6. They’re fairly on par with each other, and as I was writing out that comparison a bunch of tangents on the industry spun off…

Grok 4.6 vs Ox Alpha

And a lot of thoughts on SpaceXAI + American lab positioning

Read more

I was lucky to be on an incredible team, there is no universe in which I could have done any of this on my own. Thor (Pirate Software), Theo, Eli, Mike, Girn, CryptoFck, and many others did insane work. My job wasn’t really to solve the puzzles, my job was to have dozens of agents going at all times trying to solve the puzzles and do a couple of other things we later discovered were extremely helpful. There were 13 total puzzles and the agents were able to solve three out of thirteen on their own.

We were really scared going in that AI was going to one shot all of the puzzles. There were a lot of other competitions this year that AI did steamroll, luckily Goldbug wasn’t one of them.

I think it’s helpful to look at a puzzle AI could one shot: Spooky Manor. The idea was to work deeper and deeper into a haunted house by solving a puzzle on the page (room), then going to the next page (room) by extending the url. Everything you needed was contained on these pages, and the AI cracked it in ~1 hour. It got stuck on day 1, but that’s just because there was a mistake on the last step from the creators. Once that was fixed the agents quickly solved it.

How did the rest of the puzzles resist? We keep seeing AI crack novel math problems, agents can now run for days on end, write better code than any human, so why are they so useless? This entire experience has made me way more bullish on AI, but also a lot more measured in my expectations. AI has a long way left to go before humans are obsolete.

How I Thought This Would Go

I went in with an agent swarm prepped and ready. It had a system for sharing resources, a message board, a locking system so that they wouldn’t clobber work, and I even made a few custom dashboards to keep track of them. Going in I had high hopes.

The challenges started, the swarm auto-spawned, and I woke up to 3 solutions ready to go. I was really impressed and figured the rest of the puzzles would probably fall soon after. It took about 1-2 hours for those first ones, so assumed the rest would be done by the end of the day.

They did not get done by the end of the day.

I tried both GPT-5.6 and Fable, I tried a bunch of different harnesses, I tried different bits of context, I tried different machines - nothing worked.

I want to be clear, we used these things to solve all 13 problems. As I’m going to get into later they were absurdly useful, but they required us steering them heavily to get anything useful done.

How the Goldbug Creators Beat AI

So why is it that AI was able to derive novel math proofs not able to solve some puzzles at a hacker convention? First let me show you a couple of the puzzles:

Yes, these are both puzzles. And that’s all the context you get. Take 12 rubiks cubes or a dress and get a solution phrase. No idea how long it needs to be, just that it’s all letter with no spaces. That’s it. That’s all we had to work with, and my god the agents failed spectacularly on these.

One of the big things that made almost all the puzzles this year so hard for AI (and often us to be honest) was that they were in person. No digital data sheet or text, just some paintings, or cubes, or a dress, or even a pile of rocks. AI is amazing at working with data and code, but it’s not great at working with images, and pretty awful at working with videos. So when I sent it a pile of pictures and videos of these puzzles it fell apart almost instantly.

Now I will say, I was really impressed at how well it worked through the data here. It was able to reconstruct an accurate dataset of the cubes (which I got a ton of use out of later), and did a mostly good job on the other ones. But it missed important things for all of them, and the data quality was a total mess.

The dress was the most extreme version of this. That thing was total data overload. Different shapes, colors, groups, numbers, and locations of beads spread all over the dress left the agents useless and confused. We had to label and curate the data ourselves to solve it, which actually ended up being one of the things AI was best at helping with, but I’ll cover that later…

One of the most remarkable things I started to notice when I had multiple different GPT-5.6 swarms going on different machines all trying to solve the same problem, they’d get the same answer. I’d watch them try different theories, go down different paths, but over time they would all independently arrive at the same wrong answer.

I started to call this “convergence”. There’s probably a real term for this from actual researchers, but it felt apt to me so I’m gonna keep calling it convergence.

The convergence was very repeatable. Theo and I were both doing this, and both of our swarms would somehow reach the same exact wrong solution. After thousands of tool calls and billions of tokens, they somehow found their way to the same place. Logically I see it, these are probability machines and eventually everything regresses to the mean.

I think a lot of the failures here came down to:

• AI has trouble with data overload

• AI’s image/video understanding has a pretty hard ceiling right now

• Hallucinations still exist

Hallucinations don’t get talked about nearly as much these days. When we’re working with agents they’ve got web search tools, the ability to write/run code, and many other ways to check their work. As a result, they’re much better about not making things up. These puzzles had no such ways to check work and things got bad fast…

I cannot tell you how many times I would grill an agent about it’s logic to get a solution, just to get some plausible sounding list of steps. Steps that are entirely based on it making something up.

Some of these these hallucinations were really insidious. These puzzles have so many letters/ciphers that it ended up being easy to get something like “THECATISBLGE”. Clearly that’s wrong, but I cannot tell you how many times an agent would come up with some plausible sounding explanation for why we can sub the “G” with “U” to get “THECATISBLUE”. It never worked, but it sure did like trying stuff like that. I said in the title that I burned 30 billion tokens while solving these, I’m guessing 90%+ was devoted to garbage rabbit holes like this.

If an agent couldn’t solve a problem within an hour or two, it probably couldn’t solve it at all. More time just meant more rabbit holes and absurd leaps in logic. This is another reason I think the math stuff’s been working. Math can be verified piece by piece keeping the agent on the path for long periods of time. The problem is most domains aren’t verifiable. The hallucination issues here combined with just how much the models would converge on themselves over long periods of time (often hallucinating the same things just getting there from different paths) makes me a hell of a lot more bearish on “the god AI” coming any time soon.

As a side note though, I think this “convergence” of behavior from models is a lot of why swarms are looking promising. It’s also why OpenAI’s unreleased model that hacked Hugging Face didn’t snitch on itself even though there were hundreds of copies. They’re all so similar that it never occurred to any of them. As such I think these swarms are gonna be great force multipliers, but probably not going to change the fundamental problem sets these things are good or bad at. Though honestly this is mostly speculation, we really will not know how this will feel until GPT Astra drops.

So Was the AI Useful at All?

Yes. Absurdly so.

I wanted to talk about the failures first for two reasons:

1. There’s a ton of existential dread going around right now about humans getting fully replaced in all domains. As I showed above this just isn’t true. Will it be someday? Possibly, probably even. But I could also see that not happening. The levels of convergence at play were staggering. GPT-5.6 was far worse about it than Fable, which does suggest that as you scale up a model it’s able to go down more diverse and unique paths through the weights, but for now that’s just speculation. I think we’ll have a much better idea of the answer to this in a couple months after Astra and Fable 5.1 are released.

2. I wanted to show what I would consider to be the “dumb” way of using AI right now first. It was basically just sending the model “fix this” and hoping the thing you want magically pops out. Sometimes it does, but it’s rare and usually worse than if you put actual effort into using it.

Below is what I want to show here, how I was able to use the Agents to actually help the team solve the puzzles.

To give some context, I’m not a puzzle solver. I’ve never done or seen anything like these before and as a result can’t do much to help the AIs solve other than asking it about things and hoping I notice something that steers it in the right direction. The team on the other hand, are puzzle solving gods. I spent the most time working with Thor (Pirate Software) so I’m just gonna be talking about him for the rest of this section, but he’s one of many incredibly talented solvers on the team.

Thor and I were working on trying to solve the dress puzzle. The AI had nothing useful to say, and we pretty quickly realized that we had to pull a dataset out of the beads. This was one of the first places where the AI was extremely useful. I had Codex quickly build this:

What you see here is the final version, but we went through dozens of iterations. First I had Codex take the pictures we had of the dress and build this dataset itself. The way the puzzle worked is that the belt was the “key”. 26 different colors, mapping A-Z that we then needed to use on the rest of the dress. The issue is that these colors were really hard to tell apart sometimes for us, and were impossible for Codex. So what ended up happening is it gave us a first draft of data, we then labeled the errors and sent it back, then it got us a fixed draft. We repeated this for all of the dress, reorganized the page over and over until we had what you see above.

A lot of my job too was just listening to what the guys were saying. I didn’t have anything intelligent to add to most of it, but I could listen, and I could pass what they said on to the agents. Every time Thor talked through a theory I was furiously typing it down into a new thread so that the swarms could research and test his theory as fast as possible. We did this over and over, resulting in some really strong feedback loops. The guys have theories, the swarms test them in minutes, repeat.

My favorite use of AI was making this for the cubes puzzle:

Trying to figure out what we were supposed to do with the cubes just from pictures was extremely painful. So I had codex throw together a 3D viewer for it. First draft was ok, but not perfect. The team was voicing frustrations, I don’t think they quite realized I could actually fix these things fast, so I just quietly sent off all the prompts to fix their issues, then sent them a fresh link 10 minutes later. We did this over and over until we had a full tool built out for it, that ended up being essential in cracking this puzzle.

To get the solution you had to solve the two cubes in the gold holders, then take the moves u did to solve those and apply them to the rest of the cubes. Doing so in a solver or on paper would have been possible, but would have made it crazy difficult to see the actual solution:

Look at the non solved cubes. See how the two faces on each make a letter? AI didn’t figure this out. It didn’t figure out how to solve the puzzle or the final answer, but it did build the viewer that got us that answer.

To go back to what I talked about earlier with having the AIs solve the puzzles, I don’t think it was a total failure. Yes the ones it solved by itself were the easiest ones, but it did actually get the answers to some of the later puzzles, just after getting a lot of help from the team.

If you’ve ever used these things for coding, you know how when you give the model a slight hint or nudge when it’s stuck that unlocks an entirely new path for it? That’s pretty much how it worked here. To use the dress as an example, we got the dataset mostly by hand but using the AI for data entry. Then once we had that, we figured out what cipher was needed. Once we had the cipher and the dataset, just throwing those into the agent’s context was enough for it to get the solution within 10 minutes.

I really think this is where things are going at least for a while, AIs are an absurdly powerful raw force, that we have to direct. The amount of direction will probably change, but honestly if I had to guess it’ll just get more high level and concise. Like we’ll get to the point where you can say three words and completely change the direction of a swarm to end up exactly where you wanted. In many ways that’s what we did at Defcon and it worked incredibly well.

I hope it feels like this again next year. I really loved the balance we had between AI and humans. All the tedious garbage part of puzzles (hand testing 20+ cipher theories) evaporated under the agents, and we spent the entire time doing the fun part: figuring out the actual puzzles.

How Much Did This Cost?

30 Billion tokens is a lot, if billed at API prices that would be ~$24,000. I can assure you, I did not spend $24,000 on this competition. The actual total price I paid was around $600.

The reason I could burn so many tokens is really just because OpenAI is subsidizing their $200/mo. Plan like crazy. I have two of their $200/mo. subscriptions as well as a $200/mo. Claude subscription. I most of the time prefer OpenAI models (as you can see above), but Claude Fable 5 was useful for helping talk through a few of the puzzles, but nothing all that useful came out of it. The vast majority of my tokens were GPT-5.6 Sol through Codex, and as you can see at least for right now you can get absurd amounts of inference from these subscriptions.

The Tools I Used

1. Codex (now ChatGPT) + GPT-5.6 Sol on high or ultra reasoning. This was my main workhorse agent throughout the puzzles. GPT is really good at driving problems into the ground, it will work for hours and hours until it has (or thinks it has) throughly solved something.

2. aDrive - https://github.com/davis7dotsh/adrive. This is a little app I put together that’s basically a self hosted Google Drive. It has a web app, an api, and most importantly a cli that makes it really easy for the agents to upload files whenever they want. All of the visualizers and documents I made were hosted on adrive and then I just sent the link to the team. If u want to set it up just send the link to the github repo to codex and ask it to help you set it up and it will. (As a side note, if you don’t use AI as much, really try doing stuff like this that feels like it shouldn’t work and you will be very surprised)

3. T3 Code - https://t3.codes. T3 Code isn’t an agent, it’s a UI for your agents. It uses Codex or Claude Code under the hood, then just gives you a really nice way to view and control them. I had T3 Code running on one of the boxes in my home server which I used to do a lot of the long running jobs. I’d just send a prompt in the web app, then come back an hour later to get it. Once again, I highly recommend trying this out and the best way to get started is just by asking Codex to help you set it up.

4. Cursor - https://cursor.com/cloud. Cursor has what they call “cloud agents” which basically just means a really powerful cloud VM that you can run an agent in, spin up on the fly, and it’s awesome. I was running a ton of big long running jobs on this as well, it’s really nice to not have to have it running on my machine directly. If you don’t have the equipment for a home server, I would start here.

What I Learned

As I said, I’m not a security researcher. My job is normal software stuff. Websites, APIs, CLIs, agents, etc. The type of nonsense you build in TypeScript. If you’ve done anything with AI in that space then you probably know just how absurdly good it’s gotten at building software. High quality professional websites can be generated in minutes with frontier models, things that used to take months now take hours, there are a lot of people out there who think this field is “solved”.

I think it’s about as solved as the security puzzles were. Sure for most normal stuff it is, but for the biggest challenges? The exceptionally complex problems that actually matter? I think myself and others have overestimated it a bit.

Because the thing is AI can do basically anything in normal software. All of the sites I showed above were entirely generated by 5.6-sol without me reading any code it just happened. And in a lot of cases that’s totally fine! In fact in this case, it was amazing! It would suck if I was sitting there reading the code it would be a total waste of time.

Remember earlier when I talked about hallucinations while solving the puzzles? I mentioned it isn’t as bad in more verifiable domains like software and that is absolutely true. But the thing is, bad decisions/assumptions can still be validated. For example, the 3d cubes viewer. That was built as a single html file with all of react and 3js vendored into it. It solved the problem, but it’s not what I would want for any real software going to users.

I’m not saying you need to go hand write a fix for that, you don’t. Just tell the agent to rebuild it in a better way with a couple specifics defined and you will get a great output. That’s really my point: we still have to make a lot of decisions, keep an eye on what it’s actually doing (what horrific assumptions have been made in the depths of subagent #31’s 5th compaction), and give it clear context/direction. But if you do that? You combine a competent set of humans with the best AI systems?

Incredible stuff will happen. You still can’t replace the years of experience the best humans have or the creativity and out of the box thinking required to solve puzzles like this. AI is really good at emulating it, sometimes accurately, but the problem at least right now is more often than not it’s creative solution is just making something up.

I think this will be the real test of scaling. RL has proven to produce unbelievable work horses like GPT-5.6. Scaling (bigger model better pre-train) does seem to be bringing out more discernment in Claude Fable/Mythos, but that’s just one data point. I don’t think we’ve really scratched the surface of what it could become so really we just have to see if it can get to the point of making better decisions than humans. I and many others in Silicon Valley are pretty confident that it will be able to, but it’s a long road to get there. For now competent human + AI is the best way to work by a mile.

The models I prefer, the harnesses I use, and where I get my tokens from.

I’ve done something like this before as a bulleted list of models with little descriptions next to them, but I think it’s honestly better in an article format.

Subscribe now

The models I’m using

I’m very happy to say that the list is growing. Last time I did something like this, there were really only two models I ever touched: GPT-5.5 and Claude Opus 4.8. It’s still not a huge number, but I’m now actively using GPT-5.6 Sol, Claude Fable 5, GPT-5.6 Luna, DeepSeek V4 Flash, and Grok 4.5.

Kimi K3 is an extremely impressive model, but it’s not at the level of the others, and I don’t have nearly enough experience with it to say anything interesting. More on open-weight models at the end.

Read more