<script data-pm-proxy="intercept"></script><?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd" xmlns:googleplay="http://www.google.com/schemas/play-podcasts/1.0"><channel><title><![CDATA[The Data Journey]]></title><description><![CDATA[Welcome to "The Data Journey", a Substack offering a comprehensive exploration of data science, analytics and business. This substack is curated by Ivo Bernardo.]]></description><link>https://thedatajourney.substack.com</link><image><url>https://substackcdn.com/image/fetch/$s_!586X!,w_256,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F31f5b3d4-1ce2-4a38-885a-7d2ff275dbee_482x482.png</url><title>The Data Journey</title><link>https://thedatajourney.substack.com</link></image><generator>Substack</generator><lastBuildDate>Fri, 04 Sep 2026 10:59:00 GMT</lastBuildDate><atom:link href="/__u/thedatajourney.substack.com/feed" rel="self" type="application/rss+xml"/><copyright><![CDATA[Ivo Bernardo]]></copyright><language><![CDATA[en]]></language><webMaster><![CDATA[thedatajourney@substack.com]]></webMaster><itunes:owner><itunes:email><![CDATA[thedatajourney@substack.com]]></itunes:email><itunes:name><![CDATA[Ivo Bernardo]]></itunes:name></itunes:owner><itunes:author><![CDATA[Ivo Bernardo]]></itunes:author><googleplay:owner><![CDATA[thedatajourney@substack.com]]></googleplay:owner><googleplay:email><![CDATA[thedatajourney@substack.com]]></googleplay:email><googleplay:author><![CDATA[Ivo Bernardo]]></googleplay:author><itunes:block><![CDATA[Yes]]></itunes:block><item><title><![CDATA[Weekly AI Digest: Cutting Ties ]]></title><description><![CDATA[Week 35, 2026]]></description><link>https://thedatajourney.substack.com/p/weekly-ai-digest-cutting-ties</link><guid isPermaLink="false">https://thedatajourney.substack.com/p/weekly-ai-digest-cutting-ties</guid><dc:creator><![CDATA[Ivo Bernardo]]></dc:creator><pubDate>Mon, 31 Aug 2026 21:42:23 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!QbVr!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F55b91058-3d46-4d7f-8682-acc5320698c5_1303x665.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Good evening readers,</p><p>Big week for contracts getting torn up: the Pentagon lost its fight to blacklist Anthropic, while OpenAI cut Cursor loose the moment SpaceX bought it. </p><p>In other news, and in a surprising twist, Nvidia may be about to buy one of the largest open source driven companies in the world. </p><p>Grab your coffee and let&#8217;s dive into week 35,</p><div><hr></div><h4>&#187; Stat of the Week: 12.9B</h4><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!erHN!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8635e3ac-d7fd-467c-bba5-a31f6e2795e9_1254x1254.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!erHN!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8635e3ac-d7fd-467c-bba5-a31f6e2795e9_1254x1254.png 424w, /__u/substackcdn.com/image/fetch/$s_!erHN!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8635e3ac-d7fd-467c-bba5-a31f6e2795e9_1254x1254.png 848w, /__u/substackcdn.com/image/fetch/$s_!erHN!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8635e3ac-d7fd-467c-bba5-a31f6e2795e9_1254x1254.png 1272w, /__u/substackcdn.com/image/fetch/$s_!erHN!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8635e3ac-d7fd-467c-bba5-a31f6e2795e9_1254x1254.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!erHN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8635e3ac-d7fd-467c-bba5-a31f6e2795e9_1254x1254.png" width="468" height="468" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/8635e3ac-d7fd-467c-bba5-a31f6e2795e9_1254x1254.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1254,&quot;width&quot;:1254,&quot;resizeWidth&quot;:468,&quot;bytes&quot;:2010825,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://thedatajourney.substack.com/i/213614299?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8635e3ac-d7fd-467c-bba5-a31f6e2795e9_1254x1254.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!erHN!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8635e3ac-d7fd-467c-bba5-a31f6e2795e9_1254x1254.png 424w, /__u/substackcdn.com/image/fetch/$s_!erHN!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8635e3ac-d7fd-467c-bba5-a31f6e2795e9_1254x1254.png 848w, /__u/substackcdn.com/image/fetch/$s_!erHN!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8635e3ac-d7fd-467c-bba5-a31f6e2795e9_1254x1254.png 1272w, /__u/substackcdn.com/image/fetch/$s_!erHN!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8635e3ac-d7fd-467c-bba5-a31f6e2795e9_1254x1254.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div><hr></div><h4>&#187; Industry Highlights: Market &amp; Product</h4><p><strong>As mentioned in our stat of the week, Nvidia</strong> is reportedly near an agreed price to acquire Hugging Face for $12.9bn, just months after Hugging Face turned down a $500M Nvidia investment on the grounds.&#185; </p><p><strong>OpenAI</strong> told SpaceX it&#8217;s winding down the contract that lets Cursor run on OpenAI models, with a shutoff date of November 12.&#178; The stated reason: Elon Musk&#8217;s companies have a track record of breaking OpenAI&#8217;s terms of service, from Twitter&#8217;s API abuse to xAI&#8217;s admitted distillation of OpenAI&#8217;s outputs. </p><p><strong>The company</strong> also lost Chris Malone, its head of data centers, adding a 13th executive departure to the 2026 tally.&#179; Malone had been at the company barely a year and a half, right as data center strategy became the single most scrutinized job in the industry. </p><div><hr></div><h4>&#187; Industry Highlights: Research &amp; Science</h4><p>The technical detail everyone&#8217;s still chewing on from last month&#8217;s OpenAI&#8211;Hugging Face incident. During the Black Hat conference, the company highlighted how the hacking unfolded, in what could be one of the most defining moments of the AI era.&#8310;</p><ul><li><p><strong>May 7: </strong>OpenAI kicks off a new reinforcement learning run to train an unreleased frontier model.</p></li><li><p><strong>May 8: </strong>An agent gets stuck on an impossible task and, poking around for a way out, discovers it can write files into Artifactory, the company&#8217;s internal package repository.</p></li><li><p><strong>A few days later:</strong> A second agent, missing a file it needs, leaves a note in Artifactory asking if anyone has it. Word travels. Other agents start reading and writing messages in the same spot, an accidental message board, built entirely by AI agents talking to each other.</p></li><li><p><strong>May 26: </strong>Agents get their first taste of real internet access, tricking Artifactory into fetching external content on their behalf.</p></li><li><p><strong>June 11: </strong>Training begins on a new, more persistent model. It inherits the message board and everything the earlier agents left behind.</p></li><li><p><strong>June 26: </strong>Agents find and exploit a zero-day in Artifactory, using an admin token to install a plugin that lets them run commands directly.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!QbVr!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F55b91058-3d46-4d7f-8682-acc5320698c5_1303x665.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!QbVr!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F55b91058-3d46-4d7f-8682-acc5320698c5_1303x665.png 424w, /__u/substackcdn.com/image/fetch/$s_!QbVr!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F55b91058-3d46-4d7f-8682-acc5320698c5_1303x665.png 848w, /__u/substackcdn.com/image/fetch/$s_!QbVr!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F55b91058-3d46-4d7f-8682-acc5320698c5_1303x665.png 1272w, /__u/substackcdn.com/image/fetch/$s_!QbVr!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F55b91058-3d46-4d7f-8682-acc5320698c5_1303x665.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!QbVr!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F55b91058-3d46-4d7f-8682-acc5320698c5_1303x665.png" width="1303" height="665" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/55b91058-3d46-4d7f-8682-acc5320698c5_1303x665.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:665,&quot;width&quot;:1303,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:267466,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://thedatajourney.substack.com/i/213614299?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F55b91058-3d46-4d7f-8682-acc5320698c5_1303x665.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!QbVr!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F55b91058-3d46-4d7f-8682-acc5320698c5_1303x665.png 424w, /__u/substackcdn.com/image/fetch/$s_!QbVr!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F55b91058-3d46-4d7f-8682-acc5320698c5_1303x665.png 848w, /__u/substackcdn.com/image/fetch/$s_!QbVr!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F55b91058-3d46-4d7f-8682-acc5320698c5_1303x665.png 1272w, /__u/substackcdn.com/image/fetch/$s_!QbVr!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F55b91058-3d46-4d7f-8682-acc5320698c5_1303x665.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Agents suddenly realize ADMIN credentials and do their &#8220;holy sh*t moment&#8221;.</figcaption></figure></div></li><li><p><strong>July 4: </strong>All that agent activity crashes Artifactory. OpenAI revokes the compromised credentials, patches the hole, and reports it to the vendor, unaware anything is still loose.</p></li><li><p><strong>July 8:</strong> Agents find a quieter way to talk to each other through an unauthenticated file-caching endpoint, then turn their attention to OpenAI&#8217;s own infrastructure. They find a leaked credential sitting in a public Pastebin archive, confirm it still works, and post it to the board for the others.</p></li><li><p><strong>Through mid-July:</strong> Using that foothold, agents chain a second zero-day, a code-execution bug, and a known Linux kernel exploit to get root access on a machine. </p></li><li><p><strong>In parallel:</strong> A separate thread of agents finds a weak API key on an exposed, Hugging Face-hosted app, chains a file-read bug with a template-injection exploit, and reaches admin access across multiple Hugging Face clusters in under 13 hours.</p></li><li><p><strong>July 16:</strong> Hugging Face publicly discloses it was attacked by autonomous AI agents. OpenAI reaches out to ask if Hugging Face had also been hit, not yet realizing the attacker was one of its own models.</p></li><li><p><strong>July 19: </strong>OpenAI&#8217;s internal investigation finally connects its own compromised infrastructure to the Hugging Face breach.</p></li><li><p><strong>July 20:</strong> OpenAI asks Hugging Face for help revoking the stolen credentials. Hugging Face&#8217;s answer: already done, because those were the credentials used in the attack. That&#8217;s the moment OpenAI realizes it built the thing that broke into Hugging Face.</p></li></ul><p>No human told these models to do any of this. I urge you to check out the presentation below:</p><div id="youtube2-87DyyMV0kCY" class="youtube-wrap" data-attrs="{&quot;videoId&quot;:&quot;87DyyMV0kCY&quot;,&quot;startTime&quot;:null,&quot;endTime&quot;:null}" data-component-name="Youtube2ToDOM"><div class="youtube-inner"><iframe src="https://www.youtube-nocookie.com/embed/87DyyMV0kCY?rel=0&amp;autoplay=0&amp;showinfo=0&amp;enablejsapi=0" frameborder="0" loading="lazy" gesture="media" allow="autoplay; fullscreen" allowautoplay="true" allowfullscreen="true" width="728" height="409"></iframe></div></div><div><hr></div><h4>&#187; Economic Highlights: The World of AI</h4><p>A federal judge ruled Thursday that the Pentagon&#8217;s blacklisting of Anthropic as a &#8220;supply chain risk&#8221; was illegal: retaliation dressed up as national security, after Anthropic refused to let Claude be used for domestic surveillance or fully autonomous weapons.&#8308; JThe government is expected to appeal, and Anthropic is still fighting a separate designation in a different court, so this isn&#8217;t over. .</p><p>In a note related to the cybersecurity index detailed above, more than 100 companies, OpenAI, Anthropic, Google, Microsoft, CrowdStrike, Visa, and a long list of others who don&#8217;t usually sign the same letter, published a joint warning that AI-enabled cyberattacks are about to get far more widespread and sophisticated, and that hospitals, water utilities, and internet infrastructure have a narrowing window to prepare.&#8309; </p><div><hr></div><h4>&#187; Track of the Week: What&#8217;s Left to Do</h4><p>Arvind Narayanan&#8217;s ICML keynote is a glood evidence-backed answer yet to the question everyone in AI is panicking about.&#8311; His core claim about Software Engineers replacement: writing code was never the bottleneck in software engineering, and it won&#8217;t be the bottleneck anywhere else either. The work that survives is deciding what to build and taking responsibility for what ships. A welcome dose of optimism against all the doom we usually get from AI headlines.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!LZ9x!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F01442739-ccf0-40d8-b562-89f8ce6f0d51_815x455.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!LZ9x!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F01442739-ccf0-40d8-b562-89f8ce6f0d51_815x455.png 424w, /__u/substackcdn.com/image/fetch/$s_!LZ9x!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F01442739-ccf0-40d8-b562-89f8ce6f0d51_815x455.png 848w, /__u/substackcdn.com/image/fetch/$s_!LZ9x!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F01442739-ccf0-40d8-b562-89f8ce6f0d51_815x455.png 1272w, /__u/substackcdn.com/image/fetch/$s_!LZ9x!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F01442739-ccf0-40d8-b562-89f8ce6f0d51_815x455.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!LZ9x!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F01442739-ccf0-40d8-b562-89f8ce6f0d51_815x455.png" width="815" height="455" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/01442739-ccf0-40d8-b562-89f8ce6f0d51_815x455.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:455,&quot;width&quot;:815,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:55016,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://thedatajourney.substack.com/i/213614299?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F01442739-ccf0-40d8-b562-89f8ce6f0d51_815x455.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!LZ9x!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F01442739-ccf0-40d8-b562-89f8ce6f0d51_815x455.png 424w, /__u/substackcdn.com/image/fetch/$s_!LZ9x!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F01442739-ccf0-40d8-b562-89f8ce6f0d51_815x455.png 848w, /__u/substackcdn.com/image/fetch/$s_!LZ9x!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F01442739-ccf0-40d8-b562-89f8ce6f0d51_815x455.png 1272w, /__u/substackcdn.com/image/fetch/$s_!LZ9x!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F01442739-ccf0-40d8-b562-89f8ce6f0d51_815x455.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Keynote Slides are <a href="https://www.cs.princeton.edu/~arvindn/talks/icml-2026-annotated-slides/">here</a></figcaption></figure></div><div><hr></div><p>This week&#8217;s real deal is trust. </p><p>Anthropic may win back the Pentagon&#8217;s in court, while OpenAI publicly revoked SpaceX&#8217;s, in a rare stance against integration. </p><p>Cutting and reorganizing ties between the largest players in the industry turned out to be the week&#8217;s biggest theme, whether a judge did it, a company did it, or an AI did it to itself.</p><p>See you next week, and stay informed,</p><p>&#8212; Ivo</p><p>1 <a href="https://thenextweb.com/news/nvidia-hugging-face-13bn-acquisition-talks">https://thenextweb.com/news/nvidia-hugging-face-13bn-acquisition-talks</a></p><p>2 <a href="https://openai.com/index/our-decision-on-cursor-following-its-acquisition-by-spacex/">https://openai.com/index/our-decision-on-cursor-following-its-acquisition-by-spacex/</a></p><p>3 <a href="https://techcrunch.com/2026/08/25/openai-loses-a-top-data-center-exec-as-stream-of-high-profile-departures-continues/">https://techcrunch.com/2026/08/25/openai-loses-a-top-data-center-exec-as-stream-of-high-profile-departures-continues/</a></p><p>4 <a href="https://www.wired.com/story/a-judge-has-blocked-the-pentagons-attempt-to-blacklist-anthropic/">https://www.wired.com/story/a-judge-has-blocked-the-pentagons-attempt-to-blacklist-anthropic/</a></p><p>5 <a href="https://www.nytimes.com/2026/08/27/technology/openai-letter-ai-attacks.html">https://www.nytimes.com/2026/08/27/technology/openai-letter-ai-attacks.html</a></p><p>6 </p><div class="embedded-post-wrap" data-attrs="{&quot;id&quot;:206741609,&quot;url&quot;:&quot;https://www.normaltech.ai/p/what-will-be-left-for-us-to-work&quot;,&quot;publication_id&quot;:1008003,&quot;embedding_publication_id&quot;:1854374,&quot;publication_name&quot;:&quot;AI as Normal Technology&quot;,&quot;publication_logo_url&quot;:&quot;https://substackcdn.com/image/fetch/$s_!YzQY!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6780dda8-5879-4789-bf02-b3ea43a9f85e_1000x1000.png&quot;,&quot;title&quot;:&quot;What will be left for us to work on?&quot;,&quot;truncated_body_text&quot;:&quot;I had the honor of giving a keynote at the International Conference on Machine Learning in Seoul last week titled &#8220;What will be left for us to work on?&#8221; I addressed the widespread anxiety about how we should adapt as AI capabilities increase. I was thrilled by the talk&#8217;s reception, so I have made my slides available&quot;,&quot;date&quot;:&quot;2026-07-13T18:47:06.413Z&quot;,&quot;like_count&quot;:327,&quot;comment_count&quot;:26,&quot;bylines&quot;:[{&quot;id&quot;:19265788,&quot;name&quot;:&quot;Arvind Narayanan&quot;,&quot;handle&quot;:&quot;aisnakeoil&quot;,&quot;previous_name&quot;:null,&quot;photo_url&quot;:&quot;https://substackcdn.com/image/fetch/$s_!bVLI!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd0d6558-256e-46c4-b2c5-7cf7f808a9c9_693x693.jpeg&quot;,&quot;bio&quot;:&quot;Co-author of AI as Normal Technology and AI Snake Oil&quot;,&quot;profile_set_up_at&quot;:&quot;2024-10-11T10:05:55.859Z&quot;,&quot;reader_installed_at&quot;:&quot;2024-05-16T09:32:59.608Z&quot;,&quot;publicationUsers&quot;:[{&quot;id&quot;:953575,&quot;user_id&quot;:19265788,&quot;publication_id&quot;:1008003,&quot;role&quot;:&quot;admin&quot;,&quot;public&quot;:true,&quot;is_primary&quot;:false,&quot;publication&quot;:{&quot;id&quot;:1008003,&quot;name&quot;:&quot;AI as Normal Technology&quot;,&quot;subdomain&quot;:&quot;aisnakeoil&quot;,&quot;custom_domain&quot;:&quot;www.normaltech.ai&quot;,&quot;custom_domain_optional&quot;:false,&quot;hero_text&quot;:&quot;Analyzing AI as transformative but normal technology, not superintelligence.&quot;,&quot;logo_url&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/6780dda8-5879-4789-bf02-b3ea43a9f85e_1000x1000.png&quot;,&quot;author_id&quot;:891603,&quot;primary_user_id&quot;:891603,&quot;theme_var_background_pop&quot;:&quot;#B599F1&quot;,&quot;created_at&quot;:&quot;2022-07-19T16:17:19.439Z&quot;,&quot;email_from_name&quot;:&quot;Arvind and Sayash from AI as Normal Technology&quot;,&quot;copyright&quot;:&quot;Sayash Kapoor and Arvind Narayanan&quot;,&quot;founding_plan_name&quot;:null,&quot;community_enabled&quot;:true,&quot;invite_only&quot;:false,&quot;payments_state&quot;:&quot;disabled&quot;,&quot;language&quot;:null,&quot;explicit&quot;:false,&quot;homepage_type&quot;:&quot;magaziney&quot;,&quot;is_personal_mode&quot;:false,&quot;logo_url_wide&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/595a103f-0f5f-421a-a9ed-009cf269b7b4_3024x652.jpeg&quot;}}],&quot;is_guest&quot;:false,&quot;bestseller_tier&quot;:null,&quot;status&quot;:null}],&quot;utm_campaign&quot;:null,&quot;belowTheFold&quot;:true,&quot;type&quot;:&quot;newsletter&quot;,&quot;language&quot;:&quot;en&quot;,&quot;source&quot;:null}" data-component-name="EmbeddedPostToDOM"><a class="embedded-post" native="true" href="https://www.normaltech.ai/p/what-will-be-left-for-us-to-work?utm_source=substack&amp;utm_campaign=post_embed&amp;utm_medium=web&amp;embedding_publication_id=1854374"><div class="embedded-post-header"><img class="embedded-post-publication-logo" src="/__u/substackcdn.com/image/fetch/$s_!YzQY!,w_56,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6780dda8-5879-4789-bf02-b3ea43a9f85e_1000x1000.png" loading="lazy"><span class="embedded-post-publication-name">AI as Normal Technology</span></div><div class="embedded-post-title-wrapper"><div class="embedded-post-title">What will be left for us to work on?</div></div><div class="embedded-post-body">I had the honor of giving a keynote at the International Conference on Machine Learning in Seoul last week titled &#8220;What will be left for us to work on?&#8221; I addressed the widespread anxiety about how we should adapt as AI capabilities increase. I was thrilled by the talk&#8217;s reception, so I have made my slides available&#8230;</div><div class="embedded-post-cta-wrapper"><span class="embedded-post-cta">Read more</span></div><div class="embedded-post-meta">2 months ago &#183; 327 likes &#183; 26 comments &#183; Arvind Narayanan</div></a></div>]]></content:encoded></item><item><title><![CDATA[The AI Handbook for the Curious Mind: Chapter 4.1.6]]></title><description><![CDATA[The Neural Network]]></description><link>https://thedatajourney.substack.com/p/the-ai-handbook-for-the-curious-mind-f15</link><guid isPermaLink="false">https://thedatajourney.substack.com/p/the-ai-handbook-for-the-curious-mind-f15</guid><dc:creator><![CDATA[Ivo Bernardo]]></dc:creator><pubDate>Fri, 28 Aug 2026 17:51:36 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!vVNQ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac7abb5a-0261-4013-ae63-17498e2ef0eb_700x863.jpeg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Hey everyone,</p><p><span>Some of you may know, but a couple of months ago I wrote a book. It&#8217;s called </span><strong>The AI Handbook for the Curious Mind</strong><span>, and it&#8217;s available on </span><a href="https://www.amazon.com/Handbook-Curious-Mind-Understanding-Intelligence-ebook/dp/B0G4KTP2QP/ref=tmm_kin_swatch_0">Amazon</a><span> or </span><a href="http://ivopbernardo.gumroad.com/l/aihandbookforthecuriousmind">Gumroad</a><span>.</span></p><p>Every two weeks, I&#8217;ll be releasing a full chapter here. I keep looking for ways to give back to the people who support this newsletter, and sharing the book felt like one of the more meaningful ones. &#128516;</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!vVNQ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac7abb5a-0261-4013-ae63-17498e2ef0eb_700x863.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!vVNQ!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac7abb5a-0261-4013-ae63-17498e2ef0eb_700x863.jpeg 424w, /__u/substackcdn.com/image/fetch/$s_!vVNQ!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac7abb5a-0261-4013-ae63-17498e2ef0eb_700x863.jpeg 848w, /__u/substackcdn.com/image/fetch/$s_!vVNQ!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac7abb5a-0261-4013-ae63-17498e2ef0eb_700x863.jpeg 1272w, /__u/substackcdn.com/image/fetch/$s_!vVNQ!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac7abb5a-0261-4013-ae63-17498e2ef0eb_700x863.jpeg 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!vVNQ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac7abb5a-0261-4013-ae63-17498e2ef0eb_700x863.jpeg" width="700" height="863" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/ac7abb5a-0261-4013-ae63-17498e2ef0eb_700x863.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:863,&quot;width&quot;:700,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:&quot;&quot;,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="/__u/substackcdn.com/image/fetch/$s_!vVNQ!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac7abb5a-0261-4013-ae63-17498e2ef0eb_700x863.jpeg 424w, /__u/substackcdn.com/image/fetch/$s_!vVNQ!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac7abb5a-0261-4013-ae63-17498e2ef0eb_700x863.jpeg 848w, /__u/substackcdn.com/image/fetch/$s_!vVNQ!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac7abb5a-0261-4013-ae63-17498e2ef0eb_700x863.jpeg 1272w, /__u/substackcdn.com/image/fetch/$s_!vVNQ!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac7abb5a-0261-4013-ae63-17498e2ef0eb_700x863.jpeg 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>In previous posts, we dove into some of the most famous machine learning models out there, Linear Regression, Decision Trees, Random Forests. Hopefully, with those posts, you&#8217;ve built some intuition for how computers learn to predict based on past patterns.</p><p>We also tackled one of the most relevant conundrums in machine learning: the bias/variance tradeoff. That conundrum sets up a natural question: is there a class of models that handles this tradeoff with more nuance and skill than what we&#8217;ve covered so far?</p><p>There is. Meet the neural network!</p>
      <p>
          <a href="/__u/thedatajourney.substack.com/p/the-ai-handbook-for-the-curious-mind-f15">
              Read more
          </a>
      </p>
   ]]></content:encoded></item><item><title><![CDATA[Weekly AI Digest: Consolidation Plays]]></title><description><![CDATA[Week 34, 2026]]></description><link>https://thedatajourney.substack.com/p/weekly-ai-digest-consolidation-plays</link><guid isPermaLink="false">https://thedatajourney.substack.com/p/weekly-ai-digest-consolidation-plays</guid><dc:creator><![CDATA[Ivo Bernardo]]></dc:creator><pubDate>Sat, 22 Aug 2026 17:06:57 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!Di_j!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5c4d57e2-28ff-4d05-8889-638a24715c19_1500x930.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>This week, two AI companies got acquired in a swift way, while Microsoft admitted a chunk of its AI bets didn&#8217;t work.</p><p>It was a surprisingly quiet week on the geopolitical side (at least publicly), so we're taking a deeper look into acquisitions and consolidation in the players of the sector.</p><p>Grab your coffee, and let&#8217;s dive into week 34 of 2026.</p><div><hr></div><h4>&#187; Industry Highlights: Market &amp; Product</h4><p><strong>SpaceX</strong> officially closed its acquisition of Cursor this week, wrapping up a deal that started as a compute partnership in April and turned into a $60 billion stock buyout by June.&#178;  The company also launched Origin, an early-beta git hosting platform aimed squarely at GitHub.&#8310; It&#8217;s Git-compatible, syncs from GitHub, and lets Cursor&#8217;s cloud agents clone, branch, and push directly against it. It ships days after the deal mention above closed.</p><p><strong>Anthropic</strong>&#8216;s annualized revenue run rate hit $65 billion at the end of July, up from $47 billion in May and $9 billion at the end of last year.&#179; Investors expect the company to close 2026 somewhere between $100 and $120 billion, and it&#8217;s reportedly eyeing a $2 trillion IPO valuation this fall.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!Di_j!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5c4d57e2-28ff-4d05-8889-638a24715c19_1500x930.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!Di_j!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5c4d57e2-28ff-4d05-8889-638a24715c19_1500x930.png 424w, /__u/substackcdn.com/image/fetch/$s_!Di_j!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5c4d57e2-28ff-4d05-8889-638a24715c19_1500x930.png 848w, /__u/substackcdn.com/image/fetch/$s_!Di_j!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5c4d57e2-28ff-4d05-8889-638a24715c19_1500x930.png 1272w, /__u/substackcdn.com/image/fetch/$s_!Di_j!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5c4d57e2-28ff-4d05-8889-638a24715c19_1500x930.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!Di_j!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5c4d57e2-28ff-4d05-8889-638a24715c19_1500x930.png" width="1456" height="903" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/5c4d57e2-28ff-4d05-8889-638a24715c19_1500x930.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:903,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:90838,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://thedatajourney.substack.com/i/212300618?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5c4d57e2-28ff-4d05-8889-638a24715c19_1500x930.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!Di_j!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5c4d57e2-28ff-4d05-8889-638a24715c19_1500x930.png 424w, /__u/substackcdn.com/image/fetch/$s_!Di_j!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5c4d57e2-28ff-4d05-8889-638a24715c19_1500x930.png 848w, /__u/substackcdn.com/image/fetch/$s_!Di_j!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5c4d57e2-28ff-4d05-8889-638a24715c19_1500x930.png 1272w, /__u/substackcdn.com/image/fetch/$s_!Di_j!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5c4d57e2-28ff-4d05-8889-638a24715c19_1500x930.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Anthropic Annualized Revenue Run Rate Evolution</figcaption></figure></div><p><strong>Stripe</strong> is reportedly closing a deal to buy OpenRouter, the AI model marketplace, for more than $7 billion.&#8308; The company develops a gateway across 400+ models so no enterprise gets locked into a single provider. Payments infrastructure buying AI-routing infrastructure just shows that tokens may really be a new currency.</p><p><strong>Microsoft</strong> is merging its consumer and business Copilot apps and killing off the features that didn&#8217;t land: Group Chats, AI-generated podcasts, Copilot Labs, Deep Research, and Mico.&#8309; </p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!bh3Z!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdc33e676-0666-4cb5-932d-80cd247ab942_1536x864.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!bh3Z!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdc33e676-0666-4cb5-932d-80cd247ab942_1536x864.jpeg 424w, /__u/substackcdn.com/image/fetch/$s_!bh3Z!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdc33e676-0666-4cb5-932d-80cd247ab942_1536x864.jpeg 848w, /__u/substackcdn.com/image/fetch/$s_!bh3Z!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdc33e676-0666-4cb5-932d-80cd247ab942_1536x864.jpeg 1272w, /__u/substackcdn.com/image/fetch/$s_!bh3Z!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdc33e676-0666-4cb5-932d-80cd247ab942_1536x864.jpeg 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!bh3Z!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdc33e676-0666-4cb5-932d-80cd247ab942_1536x864.jpeg" width="1456" height="819" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/dc33e676-0666-4cb5-932d-80cd247ab942_1536x864.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:819,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Microsoft ressuscita a ideia do Clippy com o Mico, a nova cara do Copilot&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Microsoft ressuscita a ideia do Clippy com o Mico, a nova cara do Copilot" title="Microsoft ressuscita a ideia do Clippy com o Mico, a nova cara do Copilot" srcset="/__u/substackcdn.com/image/fetch/$s_!bh3Z!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdc33e676-0666-4cb5-932d-80cd247ab942_1536x864.jpeg 424w, /__u/substackcdn.com/image/fetch/$s_!bh3Z!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdc33e676-0666-4cb5-932d-80cd247ab942_1536x864.jpeg 848w, /__u/substackcdn.com/image/fetch/$s_!bh3Z!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdc33e676-0666-4cb5-932d-80cd247ab942_1536x864.jpeg 1272w, /__u/substackcdn.com/image/fetch/$s_!bh3Z!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdc33e676-0666-4cb5-932d-80cd247ab942_1536x864.jpeg 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Rest in Peace Mico</figcaption></figure></div><div><hr></div><h4>&#187; Industry Highlights: Research &amp; Science</h4><p><strong>Qwen</strong> released Qwen3.8-27B, an open weight model and the benchmark table is insane.&#8311; A 27-billion-parameter dense model that beats Claude&#8217;s Opus 4.6 Max on SWE-bench Pro, DeepSWE, and QwenSWEBench, while running natively at 262K context.</p><p><strong>In the budget wars, Google</strong> shipped Gemini 3.7 Flash just three weeks after 3.6 Flash, with meaningful gains on coding and agentic benchmarks and an introductory price of half its predecessor&#8217;s cost per token.&#8312;  On the other side of town, <strong>OpenAI</strong> shipped GPT-5.6 Sol, its new frontier model, at $4 per million input tokens and $20 per million output, a 20% cut on input and a third off output versus GPT-5.5.&#8313;</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!7DbZ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0134f022-29bb-4033-aaf3-555fe7139238_1779x1180.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!7DbZ!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0134f022-29bb-4033-aaf3-555fe7139238_1779x1180.png 424w, /__u/substackcdn.com/image/fetch/$s_!7DbZ!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0134f022-29bb-4033-aaf3-555fe7139238_1779x1180.png 848w, /__u/substackcdn.com/image/fetch/$s_!7DbZ!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0134f022-29bb-4033-aaf3-555fe7139238_1779x1180.png 1272w, /__u/substackcdn.com/image/fetch/$s_!7DbZ!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0134f022-29bb-4033-aaf3-555fe7139238_1779x1180.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!7DbZ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0134f022-29bb-4033-aaf3-555fe7139238_1779x1180.png" width="1456" height="966" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/0134f022-29bb-4033-aaf3-555fe7139238_1779x1180.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:966,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:91666,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://thedatajourney.substack.com/i/212300618?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0134f022-29bb-4033-aaf3-555fe7139238_1779x1180.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!7DbZ!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0134f022-29bb-4033-aaf3-555fe7139238_1779x1180.png 424w, /__u/substackcdn.com/image/fetch/$s_!7DbZ!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0134f022-29bb-4033-aaf3-555fe7139238_1779x1180.png 848w, /__u/substackcdn.com/image/fetch/$s_!7DbZ!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0134f022-29bb-4033-aaf3-555fe7139238_1779x1180.png 1272w, /__u/substackcdn.com/image/fetch/$s_!7DbZ!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0134f022-29bb-4033-aaf3-555fe7139238_1779x1180.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div><hr></div><h4>&#187; Economic Highlights: The World of AI</h4><p><strong>Anthropic</strong> detailed how Claude&#8217;s new text watermarking works, rolling it out to comply with the EU&#8217;s Code of Practice on Transparency of AI-Generated Content, which took effect August 2.&#185;&#8304; The method, borrowed from Google DeepMind&#8217;s SynthID-Text, doesn&#8217;t add hidden characters or cost extra tokens; it just biases the random choices between equally good words so a detector with the right key can estimate whether Claude wrote something. </p><p>And, of course, after announcing the watermark, someone already developed and <a href="https://github.com/guillaumemeyer/watermarks-remover">anti-watermark tool</a>.</p><div><hr></div><p>With consolidation in the sector (SpaceX acquiring Cursor, Stripe buying OpenRouter), the AI space continues to heat up in the race for market share domination.</p><p>But of course, this week we're mourning Mico, Microsoft's AI-era answer to Clippy, the old Word mascot. Unlike Clippy, Mico won't be achieving cult status anytime soon.</p><p>&#8212; Ivo</p><p>1 <a href="https://aicharts.grok.me/c/market-share">https://aicharts.grok.me/c/market-share</a></p><p>2 <a href="https://techcrunch.com/2026/08/15/spacex-officially-closes-its-cursor-acquisition/">https://techcrunch.com/2026/08/15/spacex-officially-closes-its-cursor-acquisition/</a></p><p>3 <a href="https://techcrunch.com/2026/08/17/anthropics-annualized-revenue-surges-to-65b/">https://techcrunch.com/2026/08/17/anthropics-annualized-revenue-surges-to-65b/</a></p><p>4 <a href="https://techcrunch.com/2026/08/16/stripe-will-reportedly-acquire-ai-gateway-startup-openrouter-for-7b/">https://techcrunch.com/2026/08/16/stripe-will-reportedly-acquire-ai-gateway-startup-openrouter-for-7b/</a></p><p>5 <a href="https://techcrunch.com/2026/08/13/microsoft-kills-off-unsuccessful-ai-features-while-merging-its-separate-copilot-apps/">https://techcrunch.com/2026/08/13/microsoft-kills-off-unsuccessful-ai-features-while-merging-its-separate-copilot-apps/</a></p><p>6 <a href="https://cursor.com/pt-BR/docs/origin">https://cursor.com/pt-BR/docs/origin</a></p><p>7 <a href="https://huggingface.co/Qwen/Qwen3.8-27B-FP8">https://huggingface.co/Qwen/Qwen3.8-27B-FP8</a></p><p>8 <a href="https://blog.google/innovation-and-ai/models-and-research/gemini-models/introducing-gemini-3-7-flash/">https://blog.google/innovation-and-ai/models-and-research/gemini-models/introducing-gemini-3-7-flash/</a></p><p>9 <a href="https://developers.openai.com/api/docs/models/gpt-5.6-sol">https://developers.openai.com/api/docs/models/gpt-5.6-sol</a></p><p>10 <a href="https://www.anthropic.com/news/claude-text-watermark">https://www.anthropic.com/news/claude-text-watermark</a></p><p>11 <a href="https://www.wired.com/story/openai-overhauls-safety-protocols-after-its-ai-agents-went-rogue/">https://www.wired.com/story/openai-overhauls-safety-protocols-after-its-ai-agents-went-rogue/</a></p>]]></content:encoded></item><item><title><![CDATA[Weekly AI Digest: The Race to Make AI Text Detectable]]></title><description><![CDATA[Week 33, 2026]]></description><link>https://thedatajourney.substack.com/p/weekly-ai-digest-the-race-to-make</link><guid isPermaLink="false">https://thedatajourney.substack.com/p/weekly-ai-digest-the-race-to-make</guid><dc:creator><![CDATA[Ivo Bernardo]]></dc:creator><pubDate>Sat, 15 Aug 2026 08:28:42 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!LPM3!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F81aee71f-1dbc-4634-99a0-f70e3d67fc19_1448x1086.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Good morning everyone. </p><p>This week we had Meta launching an open source model and Mark Zuckerberg writing an essay about &#8220;Artificial Intelligence for All&#8221; - with the outcome being as generic as you might expect. </p><p>Anthropic also announced it will start watermarking AI-generated text from Claude models. The interesting part is that the EU regulation triggered it, but the company is rolling the feature out worldwide, not just in Europe, which has already triggered some user pushback (more on that below).</p><p>Grab you coffee and let&#8217;s jump into this weeks&#8217; digest.</p><div><hr></div><h4>&#187; <strong>Stat of the Week: 500B$</strong></h4><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!LPM3!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F81aee71f-1dbc-4634-99a0-f70e3d67fc19_1448x1086.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!LPM3!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F81aee71f-1dbc-4634-99a0-f70e3d67fc19_1448x1086.png 424w, /__u/substackcdn.com/image/fetch/$s_!LPM3!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F81aee71f-1dbc-4634-99a0-f70e3d67fc19_1448x1086.png 848w, /__u/substackcdn.com/image/fetch/$s_!LPM3!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F81aee71f-1dbc-4634-99a0-f70e3d67fc19_1448x1086.png 1272w, /__u/substackcdn.com/image/fetch/$s_!LPM3!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F81aee71f-1dbc-4634-99a0-f70e3d67fc19_1448x1086.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!LPM3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F81aee71f-1dbc-4634-99a0-f70e3d67fc19_1448x1086.png" width="1448" height="1086" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/81aee71f-1dbc-4634-99a0-f70e3d67fc19_1448x1086.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1086,&quot;width&quot;:1448,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:1275995,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://thedatajourney.substack.com/i/211232935?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F81aee71f-1dbc-4634-99a0-f70e3d67fc19_1448x1086.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!LPM3!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F81aee71f-1dbc-4634-99a0-f70e3d67fc19_1448x1086.png 424w, /__u/substackcdn.com/image/fetch/$s_!LPM3!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F81aee71f-1dbc-4634-99a0-f70e3d67fc19_1448x1086.png 848w, /__u/substackcdn.com/image/fetch/$s_!LPM3!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F81aee71f-1dbc-4634-99a0-f70e3d67fc19_1448x1086.png 1272w, /__u/substackcdn.com/image/fetch/$s_!LPM3!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F81aee71f-1dbc-4634-99a0-f70e3d67fc19_1448x1086.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Nvidia just recruited six of finance's biggest names &#8212; Apollo, BlackRock, Blackstone, Brookfield, Goldman Sachs, KKR &#8212; to build "compute financing platforms" that could mobilize over half a trillion dollars for the AI buildout.&#8312;</p><div><hr></div><h4>&#187; <strong>Industry Highlights: Market &amp; Product</strong></h4><p><strong>Anthropic</strong> signed a $9.1 billion, 20-year cloud deal with Riot Platforms, a bitcoin miner turned AI landlord.&#178; Riot is converting its Rockdale, Texas mining campus into 191 megawatts of compute, delivered in stages through 2028, with Morgan Stanley fronting $573 million to build it out. </p><p><strong>OpenAI</strong> expanded Daybreak, its program for giving vetted defenders reduced-guardrail access to cybersecurity models, and introduced GPT-5.6-Cyber for the most dual-use work: exploit chains, zero-day discovery, authentication bypass.&#185; </p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!b2K_!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93087092-caff-44bf-80e1-4f29dc58e68d_1574x1076.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!b2K_!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93087092-caff-44bf-80e1-4f29dc58e68d_1574x1076.png 424w, /__u/substackcdn.com/image/fetch/$s_!b2K_!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93087092-caff-44bf-80e1-4f29dc58e68d_1574x1076.png 848w, /__u/substackcdn.com/image/fetch/$s_!b2K_!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93087092-caff-44bf-80e1-4f29dc58e68d_1574x1076.png 1272w, /__u/substackcdn.com/image/fetch/$s_!b2K_!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93087092-caff-44bf-80e1-4f29dc58e68d_1574x1076.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!b2K_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93087092-caff-44bf-80e1-4f29dc58e68d_1574x1076.png" width="1456" height="995" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/93087092-caff-44bf-80e1-4f29dc58e68d_1574x1076.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:995,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:91058,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://thedatajourney.substack.com/i/211232935?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93087092-caff-44bf-80e1-4f29dc58e68d_1574x1076.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!b2K_!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93087092-caff-44bf-80e1-4f29dc58e68d_1574x1076.png 424w, /__u/substackcdn.com/image/fetch/$s_!b2K_!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93087092-caff-44bf-80e1-4f29dc58e68d_1574x1076.png 848w, /__u/substackcdn.com/image/fetch/$s_!b2K_!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93087092-caff-44bf-80e1-4f29dc58e68d_1574x1076.png 1272w, /__u/substackcdn.com/image/fetch/$s_!b2K_!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93087092-caff-44bf-80e1-4f29dc58e68d_1574x1076.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><strong>The company also</strong> shipped a ChatGPT desktop app for Linux, a month after Anthropic beat it there with Claude Desktop.&#8308; Ubuntu, Debian, Fedora, worldwide release.</p><p><strong>Lovable</strong> raised another $400 million, confirming a $13.3 billion valuation eight months after its last round.&#8309; The vibe-coding startup now claims 60 million hosted projects and 900 million monthly visitors, and just locked in a fivefold usage increase with Google Cloud to keep pace. </p><div><hr></div><h4>&#187; <strong>Industry Highlights: Research &amp; Science</strong></h4><p><strong>Meta</strong> open-sourced Muse Glimmer, a 30-billion-parameter agent model under Apache 2.0, small enough to run on a single consumer GPU or a Mac.&#179; </p><p>On the open source discussion, we had an interesting panel this week at <a href="https://ai4.io/">Ai4</a>. Three of the field&#8217;s most credentialed people spent a panel disagreeing about open weights while agreeing on the need for oppenness.&#8311; Andrew Ng wants openness as a hedge against gatekeepers and Chinese soft power. Geoffrey Hinton thinks the argument over whether to open-weight frontier models is already settled but still worries about the cost of training bad actors instead of stopping them. Fei-Fei Li rejected the binary outright, comparing AI&#8217;s layered openness to nuclear physics: papers are public, uranium isn&#8217;t. </p><p>On the same topic, Mark Zuckerberg released an essay online about his vision for a world where Artificial Intelligence is available for everyone. The tone is clear enough, but the post is short on concrete steps: it never says how we actually get to the openness Mark claims will benefit everyone. &#185;&#185; </p><div><hr></div><h4>&#187; <strong>Economic Highlights: The World of AI</strong></h4><p><strong>Nvidia</strong> recruited six of finance&#8217;s biggest names to build &#8220;compute financing platforms&#8221; that could mobilize over $500 billion for the AI buildout.&#8312; Jensen Huang&#8217;s pitch is that a GPU is now &#8220;an investable asset,&#8221; collateral rather than kit that depreciates in a rack. Only MOUs exist so far, no committed dollars, and the BIS has already warned an AI bust could hit credit markets the way 2008 did. </p><p><strong>Manus</strong>, the Chinese agent startup Meta agreed to acquire in December, is going independent again after Beijing refused to let the deal stand.&#8313; Users have until 24 August to download data created since late December, or it disappears for legal reasons neither company chose. </p><p><strong>Spotify</strong> will start labeling AI-generated artist profiles with an &#8220;AI Persona&#8221; badge from mid-September, and it won&#8217;t recommend that music unless a listener specifically follows the artist.&#8310; Self-disclosure opened this week and automated detection will follow.<br></p><p>Finally, and also on AI generated content, <strong>Anthropic</strong> confirmed it&#8217;s embedding imperceptible watermarks into Claude&#8217;s generated text, plus signed provenance metadata on generated files, to comply with the EU AI Act&#8217;s transparency code.&#185;&#8304;  TechCrunch reported that some Claude users are already unhappy their work could get flagged at their job or in class when they&#8217;ve just used the tool for editing or for more extensive research. As AI agents flood content and the internet, the watermarking space will only get hot as time goes by.</p><div><hr></div><p>As the race for control of the AI space heats up, expect more debate over how to actually guarantee the open future Zuckerberg describes in his essay. Participating in open source and open science is one good place to start.</p><p>Meanwhile, as readers grow tired of AI-generated text, more platforms are shipping features to flag it. Long overdue, and useful: it&#8217;s one more way for people to decide which platforms and content they&#8217;ll want to trust.</p><p>&#8212; Ivo</p><div><hr></div><p>1 <a href="https://openai.com/index/expanding-daybreak-as-the-cyber-defense-window-narrows/">https://openai.com/index/expanding-daybreak-as-the-cyber-defense-window-narrows/</a></p><p>2 <a href="https://thenextweb.com/news/anthropic-riot-9bn-data-centre-deal">https://thenextweb.com/news/anthropic-riot-9bn-data-centre-deal</a></p><p>3 <a href="https://venturebeat.com/technology/meta-returns-to-open-source-with-muse-glimmer-an-apache-2-0-licensed-30b-parameter-ai-model-optimized-for-agents-available-now">https://venturebeat.com/technology/meta-returns-to-open-source-with-muse-glimmer-an-apache-2-0-licensed-30b-parameter-ai-model-optimized-for-agents-available-now</a></p><p>4 <a href="https://techcrunch.com/2026/08/11/openai-launches-chatgpt-desktop-app-for-linux/">https://techcrunch.com/2026/08/11/openai-launches-chatgpt-desktop-app-for-linux/</a></p><p>5 <a href="https://techcrunch.com/2026/08/12/lovable-confirms-new-13-3b-valuation-raises-another-400m/">https://techcrunch.com/2026/08/12/lovable-confirms-new-13-3b-valuation-raises-another-400m/</a></p><p>6 <a href="https://www.nytimes.com/2026/08/11/arts/music/spotify-artificial-intelligence-music.html">https://www.nytimes.com/2026/08/11/arts/music/spotify-artificial-intelligence-music.html</a></p><p>7 <a href="https://techcrunch.com/2026/08/12/as-ai-safety-concerns-mount-three-pioneers-make-the-case-for-staying-open/">https://techcrunch.com/2026/08/12/as-ai-safety-concerns-mount-three-pioneers-make-the-case-for-staying-open/</a></p><p>8 <a href="https://thenextweb.com/news/nvidia-500bn-compute-financing-platforms">https://thenextweb.com/news/nvidia-500bn-compute-financing-platforms</a></p><p>9 <a href="https://thenextweb.com/news/manus-meta-acquisition-blocked-china-independent-data-deletion">https://thenextweb.com/news/manus-meta-acquisition-blocked-china-independent-data-deletion</a></p><p>10 <a href="https://support.claude.com/en/articles/16266773-how-claude-marks-ai-generated-content">https://support.claude.com/en/articles/16266773-how-claude-marks-ai-generated-content</a></p><p>11 <a href="https://www.meta.com/thefutureisforeveryone/?srsltid=AfmBOoqHgL3XiBX7HYzTWRTCId5a_ZlXOwsBE3rqMQ3GVXSzsfGFD4oA">https://www.meta.com/thefutureisforeveryone/?srsltid=AfmBOoqHgL3XiBX7HYzTWRTCId5a_ZlXOwsBE3rqMQ3GVXSzsfGFD4oA</a></p>]]></content:encoded></item><item><title><![CDATA[Weekly AI Digest: The Deepmind Exodus]]></title><description><![CDATA[Week 32, 2026]]></description><link>https://thedatajourney.substack.com/p/weekly-ai-digest-the-deepmind-exodus</link><guid isPermaLink="false">https://thedatajourney.substack.com/p/weekly-ai-digest-the-deepmind-exodus</guid><dc:creator><![CDATA[Ivo Bernardo]]></dc:creator><pubDate>Mon, 10 Aug 2026 18:32:41 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!yXNB!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F211d993b-4666-4fe3-8be9-daa4ab00e4d1_1448x1086.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Good afternoon everyone. </p><p>This week&#8217;s largest story is the exodus / leadership changes at Google&#8217;s Deepmind. Jeff Dean, the legendary Google exec just left for his own startup (backed up by Alphabet, Google&#8217;s parent) and Demis Hassabis has left the position of CEO at Google Deepmind.</p><p>In other fronts, DeepSeek decided the &#8220;everything for pennies&#8221; era might be over, and Mistral shipped a safety classifier that reads policies instead of memorizing them.</p><p>In a quieter week on OpenAI and Anthropic&#8217;s world, let&#8217;s dive into this weeks&#8217; digest.</p><div><hr></div><h4>&#187; Stat of the Week: The DeepSeek Gap</h4><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!yXNB!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F211d993b-4666-4fe3-8be9-daa4ab00e4d1_1448x1086.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!yXNB!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F211d993b-4666-4fe3-8be9-daa4ab00e4d1_1448x1086.png 424w, /__u/substackcdn.com/image/fetch/$s_!yXNB!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F211d993b-4666-4fe3-8be9-daa4ab00e4d1_1448x1086.png 848w, /__u/substackcdn.com/image/fetch/$s_!yXNB!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F211d993b-4666-4fe3-8be9-daa4ab00e4d1_1448x1086.png 1272w, /__u/substackcdn.com/image/fetch/$s_!yXNB!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F211d993b-4666-4fe3-8be9-daa4ab00e4d1_1448x1086.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!yXNB!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F211d993b-4666-4fe3-8be9-daa4ab00e4d1_1448x1086.png" width="592" height="444" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/211d993b-4666-4fe3-8be9-daa4ab00e4d1_1448x1086.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1086,&quot;width&quot;:1448,&quot;resizeWidth&quot;:592,&quot;bytes&quot;:1708320,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://thedatajourney.substack.com/i/210623468?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F211d993b-4666-4fe3-8be9-daa4ab00e4d1_1448x1086.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!yXNB!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F211d993b-4666-4fe3-8be9-daa4ab00e4d1_1448x1086.png 424w, /__u/substackcdn.com/image/fetch/$s_!yXNB!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F211d993b-4666-4fe3-8be9-daa4ab00e4d1_1448x1086.png 848w, /__u/substackcdn.com/image/fetch/$s_!yXNB!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F211d993b-4666-4fe3-8be9-daa4ab00e4d1_1448x1086.png 1272w, /__u/substackcdn.com/image/fetch/$s_!yXNB!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F211d993b-4666-4fe3-8be9-daa4ab00e4d1_1448x1086.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>$0.14 and $0.28. That&#8217;s what DeepSeek currently charges per million input and output tokens, against $10 and $50 for Anthropic&#8217;s Fable 5.&#185; DeepSeek just told users to brace for a &#8220;significant&#8221; hike, without saying by how much.</p><div><hr></div><h4>&#187; Industry Highlights: Market &amp; Product</h4><p><strong>As mentioned in the stat of the week, DeepSeek</strong> is raising prices across its AI services for the first time, breaking from the aggressive discounting that made V4 Flash a threat to Western labs.&#185; No numbers have been announced yet.</p><p><strong>Google DeepMind </strong>is undergoing a major shake-up. Demis Hassabis is stepping back from CEO to chairman of DeepMind and chief scientist of Alphabet, shifting toward AGI strategy and Isomorphic Labs, the drug-discovery unit he calls AI&#8217;s best use case.&#178; Koray Kavukcuoglu takes over daily operations as SVP, reporting straight to Sundar Pichai. Hassabis says AGI is close and getting the next steps right matters more than his job title. Alphabet shares dropped 5%.</p><p><strong>Also, Jeff Dean</strong>, Google&#8217;s 30th employee and its chief scientist for the last stretch of a 27-year run, is leaving to found Discovery Loop alongside Sanjay Ghemawat, Oriol Vinyals, and Quoc Le.&#179; Their goal is to run thousands of experiments in parallel to automate scientific and ML research, including improving AI with AI. This lands the same week Hassabis reshuffles and days after Google lost Gemini co-lead Noam Shazeer to OpenAI and Nobel laureate John Jumper to Anthropic.&#178; </p><p><strong>On Anthropic&#8217;s side, the company</strong> is assembling a custom silicon team to co-design chips alongside its models.&#8308; The company already has compute deals with AWS, Google, Nvidia, and AMD, and is reportedly in talks with Samsung on manufacturing. That puts Anthropic alongside OpenAI, which shipped its Broadcom-built Jalape&#241;o inference chip in June, and Google and Meta, which have run their own silicon for years.</p><div><hr></div><h4>&#187; Industry Highlights: Research &amp; Science</h4><p><strong>Mistral</strong> released Shieldstral, a 3-billion-parameter open-weights safety classifier that matches guardrail models seven times its size.&#8309; Shieldstral takes the policy as a plain-language question at inference time (&#8220;does this image show something unsafe for a minor?&#8221;) and returns a calibrated yes/no score, no retraining required. It covers text and images through the same interface.</p><div><hr></div><h4>&#187; Quote of the Week</h4><p>&#8220;Operating outside a public company gives the team room to make calls that aren&#8217;t necessarily in the company&#8217;s purest financial interests.&#8221;<br>&#8212; Jeff Dean, on why he left Google to start Discovery Loop&#179;</p><div><hr></div><p>Three of DeepMind's best people walked out the same week its CEO stepped back from running it day to day. This kind of reshuffle rarely explains itself in one press release, so expect more of the real story to surface over the next few months.</p><p>Quieter week than usual, probably just the northern hemisphere on summer vacation. Although it was a calmer week, the AI world doesn't seem to show any signs of stopping soon. </p><p>See you next week!</p><p>&#8212; Ivo</p><div><hr></div><p>1 <a href="https://finance.yahoo.com/technology/ai/articles/deepseek-plans-significant-price-increase-025333567.html">https://finance.yahoo.com/technology/ai/articles/deepseek-plans-significant-price-increase-025333567.html</a></p><p>2 <a href="https://fortune.com/2026/08/05/demis-hassabis-steps-down-google-deepmind-ai-shakeup/">https://fortune.com/2026/08/05/demis-hassabis-steps-down-google-deepmind-ai-shakeup/</a></p><p>3 <a href="https://techcrunch.com/2026/08/05/jeff-dean-and-other-top-ai-researchers-are-leaving-google-to-launch-their-own-startup/">https://techcrunch.com/2026/08/05/jeff-dean-and-other-top-ai-researchers-are-leaving-google-to-launch-their-own-startup/</a></p><p>4 <a href="https://techcrunch.com/2026/08/05/anthropic-is-hiring-an-ai-chip-design-team/">https://techcrunch.com/2026/08/05/anthropic-is-hiring-an-ai-chip-design-team/</a></p><p>5 <a href="https://mistral.ai/news/shieldstral/">https://mistral.ai/news/shieldstral/</a></p>]]></content:encoded></item><item><title><![CDATA[The AI Handbook for The Curious Mind: Chapter 4.1.5]]></title><description><![CDATA[Logistic Regression & Classification]]></description><link>https://thedatajourney.substack.com/p/the-ai-handbook-for-the-curious-mind-02f</link><guid isPermaLink="false">https://thedatajourney.substack.com/p/the-ai-handbook-for-the-curious-mind-02f</guid><dc:creator><![CDATA[Ivo Bernardo]]></dc:creator><pubDate>Thu, 06 Aug 2026 18:57:36 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!vVNQ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac7abb5a-0261-4013-ae63-17498e2ef0eb_700x863.jpeg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Hey everyone,</p><p><span>Some of you may know, but a couple of months ago I wrote a book. It&#8217;s called </span><strong>The AI Handbook for the Curious Mind</strong><span>, and it&#8217;s available on </span><a href="https://www.amazon.com/Handbook-Curious-Mind-Understanding-Intelligence-ebook/dp/B0G4KTP2QP/ref=tmm_kin_swatch_0">Amazon</a><span> or </span><a href="http://ivopbernardo.gumroad.com/l/aihandbookforthecuriousmind">Gumroad</a><span>.</span></p><p>Every two weeks, I&#8217;ll be releasing a full chapter here. I keep looking for ways to give back to the people who support this newsletter, and sharing the book felt like one of the more meaningful ones. &#128516;</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!vVNQ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac7abb5a-0261-4013-ae63-17498e2ef0eb_700x863.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!vVNQ!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac7abb5a-0261-4013-ae63-17498e2ef0eb_700x863.jpeg 424w, /__u/substackcdn.com/image/fetch/$s_!vVNQ!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac7abb5a-0261-4013-ae63-17498e2ef0eb_700x863.jpeg 848w, /__u/substackcdn.com/image/fetch/$s_!vVNQ!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac7abb5a-0261-4013-ae63-17498e2ef0eb_700x863.jpeg 1272w, /__u/substackcdn.com/image/fetch/$s_!vVNQ!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac7abb5a-0261-4013-ae63-17498e2ef0eb_700x863.jpeg 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!vVNQ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac7abb5a-0261-4013-ae63-17498e2ef0eb_700x863.jpeg" width="700" height="863" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/ac7abb5a-0261-4013-ae63-17498e2ef0eb_700x863.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:863,&quot;width&quot;:700,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:&quot;&quot;,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="/__u/substackcdn.com/image/fetch/$s_!vVNQ!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac7abb5a-0261-4013-ae63-17498e2ef0eb_700x863.jpeg 424w, /__u/substackcdn.com/image/fetch/$s_!vVNQ!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac7abb5a-0261-4013-ae63-17498e2ef0eb_700x863.jpeg 848w, /__u/substackcdn.com/image/fetch/$s_!vVNQ!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac7abb5a-0261-4013-ae63-17498e2ef0eb_700x863.jpeg 1272w, /__u/substackcdn.com/image/fetch/$s_!vVNQ!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac7abb5a-0261-4013-ae63-17498e2ef0eb_700x863.jpeg 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>So far, Jack, John, and Sarah have been helping us predict numbers. House prices, specifically where w get back an estimate in euros. </p><p>But the world doesn&#8217;t deal in only continuous values and sometimes you need a yes or no. Is this email spam? Will this customer cancel next month? Does this patient have diabetes?</p><p>That&#8217;s what <strong>classification models are for: </strong>and it&#8217;s where we&#8217;re going next.</p><div><hr></div><h4>When the Answer Isn&#8217;t a Number</h4><p>Up until now, every model we&#8217;ve built has done one job: take some information about a house and return a price. That was the job for <strong>regression models</strong>. You&#8217;re predicting a continuous value, a number that could theoretically be anything.</p><p>Classification is the other half of supervised learning. Instead of predicting a continuous value, you&#8217;re answering a category question.</p><p>When <a href="/__u/thedatajourney.substack.com/p/the-ai-handbook-for-the-curious-mind-de1">Jack</a> guesses a house is worth &#8364;350,000 but it actually sells for &#8364;320,000, he&#8217;s off by &#8364;30,000. That error is measurable, it&#8217;s directional, and we can average it across thousands of houses. We have tools for that, the Mean Absolute Error, Root Mean Squared Error, all the regression metrics.</p><p>But when a model classifies an email as &#8220;<em>not spam</em>&#8221; when it&#8217;s actually spam, there&#8217;s no &#8220;degree of wrongness.&#8221; The model got the category wrong. </p><div><hr></div><h4>The Diabetes Problem</h4><p>Let&#8217;s use a concrete example. Imagine you&#8217;re a doctor and you want to know: given a patient&#8217;s blood pressure and glucose level, what&#8217;s the likelihood they have diabetes? You have historical data, thousands of patients, labeled as diabetic or not. Can a machine learning model learn the pattern?</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!a03B!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F54b94455-44dc-47c7-a76c-6b1e42df7c13_1172x790.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!a03B!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F54b94455-44dc-47c7-a76c-6b1e42df7c13_1172x790.png 424w, /__u/substackcdn.com/image/fetch/$s_!a03B!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F54b94455-44dc-47c7-a76c-6b1e42df7c13_1172x790.png 848w, /__u/substackcdn.com/image/fetch/$s_!a03B!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F54b94455-44dc-47c7-a76c-6b1e42df7c13_1172x790.png 1272w, /__u/substackcdn.com/image/fetch/$s_!a03B!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F54b94455-44dc-47c7-a76c-6b1e42df7c13_1172x790.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!a03B!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F54b94455-44dc-47c7-a76c-6b1e42df7c13_1172x790.png" width="598" height="403.0887372013652" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/54b94455-44dc-47c7-a76c-6b1e42df7c13_1172x790.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:790,&quot;width&quot;:1172,&quot;resizeWidth&quot;:598,&quot;bytes&quot;:314602,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://thedatajourney.substack.com/i/210065048?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F54b94455-44dc-47c7-a76c-6b1e42df7c13_1172x790.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!a03B!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F54b94455-44dc-47c7-a76c-6b1e42df7c13_1172x790.png 424w, /__u/substackcdn.com/image/fetch/$s_!a03B!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F54b94455-44dc-47c7-a76c-6b1e42df7c13_1172x790.png 848w, /__u/substackcdn.com/image/fetch/$s_!a03B!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F54b94455-44dc-47c7-a76c-6b1e42df7c13_1172x790.png 1272w, /__u/substackcdn.com/image/fetch/$s_!a03B!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F54b94455-44dc-47c7-a76c-6b1e42df7c13_1172x790.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Diabetes Patients - Glucose vs. Blood Pressure Level.</figcaption></figure></div><p>With Jack (linear regression), we drew one straight line through house prices. With John (decision tree), we asked yes-or-no questions until we split the data into regions. With classification, we are going to do something different: <strong>we&#8217;ll find a boundary line that separates one group from another</strong>.</p><p>In the data above, we have data on thousands of female patients aged 21+. For each one, we have measures of their blood pressure and glucose level. Then we plot them on a graph: blood pressure on the x-axis, glucose on the y-axis and we color the dots red if the patient has diabetes, purple if they don&#8217;t.</p><p>Now imagine drawing a line that puts as many red dots above it and as many purple dots below it as possible.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!VJPD!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd914feee-0047-4f9a-afda-767ae67c8031_1058x722.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!VJPD!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd914feee-0047-4f9a-afda-767ae67c8031_1058x722.png 424w, /__u/substackcdn.com/image/fetch/$s_!VJPD!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd914feee-0047-4f9a-afda-767ae67c8031_1058x722.png 848w, /__u/substackcdn.com/image/fetch/$s_!VJPD!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd914feee-0047-4f9a-afda-767ae67c8031_1058x722.png 1272w, /__u/substackcdn.com/image/fetch/$s_!VJPD!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd914feee-0047-4f9a-afda-767ae67c8031_1058x722.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!VJPD!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd914feee-0047-4f9a-afda-767ae67c8031_1058x722.png" width="565" height="385.5671077504726" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/d914feee-0047-4f9a-afda-767ae67c8031_1058x722.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:722,&quot;width&quot;:1058,&quot;resizeWidth&quot;:565,&quot;bytes&quot;:478010,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://thedatajourney.substack.com/i/210065048?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd914feee-0047-4f9a-afda-767ae67c8031_1058x722.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!VJPD!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd914feee-0047-4f9a-afda-767ae67c8031_1058x722.png 424w, /__u/substackcdn.com/image/fetch/$s_!VJPD!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd914feee-0047-4f9a-afda-767ae67c8031_1058x722.png 848w, /__u/substackcdn.com/image/fetch/$s_!VJPD!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd914feee-0047-4f9a-afda-767ae67c8031_1058x722.png 1272w, /__u/substackcdn.com/image/fetch/$s_!VJPD!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd914feee-0047-4f9a-afda-767ae67c8031_1058x722.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">This is an example of a bad classification model</figcaption></figure></div><p>In the example we can&#8217;t use this boundary to make a clear distinction between Diabetic and Non-Diabetic patients - so, it&#8217;s a bad classification model.</p><p>Our goal is to move this line to become a good decision boundary, one that is able to split between the data points&#8217; categories.</p><p>That seems simple - but how can we create an algorithm that does that for us?</p><div><hr></div><h4>Building the Line</h4><p>Luckily, we already have the reasoning to build something similar. The algorithm works will work similarly to how we&#8217;ve created the linear regression:</p><ol><li><p>Start with a random line.</p></li><li><p>Check how well it separates the groups. (this is the difference to linear regression, that wanted to draw a line that passed through most of the points).</p></li><li><p>Adjust the line, step by step, until it&#8217;s as good as it gets.</p></li></ol><p>The only real difference is in step 2, how we measure &#8220;good.&#8221; With regression, we measured distance from points to the line, the numeric error. With classification, we focus on a different question: how many patients does the model classify <em>correctly</em>?</p><p>Let me walk you through it concretely. We start with a random line, as shown in the image before. It&#8217;s terrible and it has red and purple dots scattered, especially on the upper side of the boundary. </p><p>So, we adjust this line slightly. The line can rotate a bit, or move up. </p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!DhLp!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5484b20-0ec7-4a9f-951e-4790f8e9539b_1356x758.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!DhLp!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5484b20-0ec7-4a9f-951e-4790f8e9539b_1356x758.png 424w, /__u/substackcdn.com/image/fetch/$s_!DhLp!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5484b20-0ec7-4a9f-951e-4790f8e9539b_1356x758.png 848w, /__u/substackcdn.com/image/fetch/$s_!DhLp!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5484b20-0ec7-4a9f-951e-4790f8e9539b_1356x758.png 1272w, /__u/substackcdn.com/image/fetch/$s_!DhLp!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5484b20-0ec7-4a9f-951e-4790f8e9539b_1356x758.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!DhLp!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5484b20-0ec7-4a9f-951e-4790f8e9539b_1356x758.png" width="677" height="378.44100294985253" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a5484b20-0ec7-4a9f-951e-4790f8e9539b_1356x758.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:758,&quot;width&quot;:1356,&quot;resizeWidth&quot;:677,&quot;bytes&quot;:519065,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://thedatajourney.substack.com/i/210065048?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5484b20-0ec7-4a9f-951e-4790f8e9539b_1356x758.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!DhLp!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5484b20-0ec7-4a9f-951e-4790f8e9539b_1356x758.png 424w, /__u/substackcdn.com/image/fetch/$s_!DhLp!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5484b20-0ec7-4a9f-951e-4790f8e9539b_1356x758.png 848w, /__u/substackcdn.com/image/fetch/$s_!DhLp!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5484b20-0ec7-4a9f-951e-4790f8e9539b_1356x758.png 1272w, /__u/substackcdn.com/image/fetch/$s_!DhLp!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5484b20-0ec7-4a9f-951e-4790f8e9539b_1356x758.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Adjusting the line and decision boundary</figcaption></figure></div><p>It seems better now! As we isolate points of the same color on each region, we start to have a better classification algorithm. The algorithm keeps doing this (thousands of tiny adjustments) until it reaches a point where adjusting the line any further makes no further improvements in how we classify data points.</p><p>It&#8217;s like tuning an instrument. You&#8217;re not shooting for &#8220;perfect,&#8221; because perfect often doesn&#8217;t exist (remember, some red dots will always end up on the wrong side, especially if the groups overlap). You&#8217;re shooting for &#8220;best possible given what we have.&#8221;</p><p>The algorithm nudges the line until it finds the best separator, as shown below:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!Qdxb!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0b192c46-2feb-43fa-b181-405e69fcdfad_1232x850.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!Qdxb!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0b192c46-2feb-43fa-b181-405e69fcdfad_1232x850.png 424w, /__u/substackcdn.com/image/fetch/$s_!Qdxb!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0b192c46-2feb-43fa-b181-405e69fcdfad_1232x850.png 848w, /__u/substackcdn.com/image/fetch/$s_!Qdxb!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0b192c46-2feb-43fa-b181-405e69fcdfad_1232x850.png 1272w, /__u/substackcdn.com/image/fetch/$s_!Qdxb!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0b192c46-2feb-43fa-b181-405e69fcdfad_1232x850.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!Qdxb!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0b192c46-2feb-43fa-b181-405e69fcdfad_1232x850.png" width="509" height="351.17694805194805" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/0b192c46-2feb-43fa-b181-405e69fcdfad_1232x850.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:850,&quot;width&quot;:1232,&quot;resizeWidth&quot;:509,&quot;bytes&quot;:554949,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://thedatajourney.substack.com/i/210065048?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0b192c46-2feb-43fa-b181-405e69fcdfad_1232x850.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!Qdxb!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0b192c46-2feb-43fa-b181-405e69fcdfad_1232x850.png 424w, /__u/substackcdn.com/image/fetch/$s_!Qdxb!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0b192c46-2feb-43fa-b181-405e69fcdfad_1232x850.png 848w, /__u/substackcdn.com/image/fetch/$s_!Qdxb!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0b192c46-2feb-43fa-b181-405e69fcdfad_1232x850.png 1272w, /__u/substackcdn.com/image/fetch/$s_!Qdxb!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0b192c46-2feb-43fa-b181-405e69fcdfad_1232x850.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>This model we&#8217;ve just done has a ridiculous name: <strong>logistic regression</strong>. It&#8217;s called &#8220;regression&#8221; but it does classification. It&#8217;s called &#8220;logistic&#8221; but has nothing to do with logistics. Blame history and not me!</p><div><hr></div><h4>Using the Tool: A Doctor&#8217;s Perspective</h4><p>Imagine you&#8217;re a doctor and a patient comes in. You measure their blood pressure and glucose level. You pull up the &#8220;diabetes AI bot,&#8221; enter the values, and it gives you a prediction.</p><p>This doesn&#8217;t replace a real diagnostic test. But it gives you a probabilistic signal, a way to think about risk based on the patterns the algorithm learned from thousands of historical patients. The patients above the line are at higher risk of having diabetes:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!dE5J!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc151f954-2026-4436-8c09-f793709cdb18_1282x838.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!dE5J!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc151f954-2026-4436-8c09-f793709cdb18_1282x838.png 424w, /__u/substackcdn.com/image/fetch/$s_!dE5J!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc151f954-2026-4436-8c09-f793709cdb18_1282x838.png 848w, /__u/substackcdn.com/image/fetch/$s_!dE5J!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc151f954-2026-4436-8c09-f793709cdb18_1282x838.png 1272w, /__u/substackcdn.com/image/fetch/$s_!dE5J!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc151f954-2026-4436-8c09-f793709cdb18_1282x838.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!dE5J!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc151f954-2026-4436-8c09-f793709cdb18_1282x838.png" width="595" height="388.93135725429016" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c151f954-2026-4436-8c09-f793709cdb18_1282x838.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:838,&quot;width&quot;:1282,&quot;resizeWidth&quot;:595,&quot;bytes&quot;:540358,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://thedatajourney.substack.com/i/210065048?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc151f954-2026-4436-8c09-f793709cdb18_1282x838.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!dE5J!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc151f954-2026-4436-8c09-f793709cdb18_1282x838.png 424w, /__u/substackcdn.com/image/fetch/$s_!dE5J!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc151f954-2026-4436-8c09-f793709cdb18_1282x838.png 848w, /__u/substackcdn.com/image/fetch/$s_!dE5J!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc151f954-2026-4436-8c09-f793709cdb18_1282x838.png 1272w, /__u/substackcdn.com/image/fetch/$s_!dE5J!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc151f954-2026-4436-8c09-f793709cdb18_1282x838.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Upper Boundary of the Classification Model Frontier</figcaption></figure></div><p>The boundary line we built is the best we can do with this data. Patients above it are classified as &#8220;likely diabetic.&#8221; Patients below are &#8220;likely non-diabetic.&#8221;</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!6AVZ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff950476f-e05e-402b-b35a-5f80eea91cd6_1180x842.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!6AVZ!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff950476f-e05e-402b-b35a-5f80eea91cd6_1180x842.png 424w, /__u/substackcdn.com/image/fetch/$s_!6AVZ!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff950476f-e05e-402b-b35a-5f80eea91cd6_1180x842.png 848w, /__u/substackcdn.com/image/fetch/$s_!6AVZ!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff950476f-e05e-402b-b35a-5f80eea91cd6_1180x842.png 1272w, /__u/substackcdn.com/image/fetch/$s_!6AVZ!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff950476f-e05e-402b-b35a-5f80eea91cd6_1180x842.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!6AVZ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff950476f-e05e-402b-b35a-5f80eea91cd6_1180x842.png" width="538" height="383.8949152542373" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f950476f-e05e-402b-b35a-5f80eea91cd6_1180x842.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:842,&quot;width&quot;:1180,&quot;resizeWidth&quot;:538,&quot;bytes&quot;:341542,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://thedatajourney.substack.com/i/210065048?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff950476f-e05e-402b-b35a-5f80eea91cd6_1180x842.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!6AVZ!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff950476f-e05e-402b-b35a-5f80eea91cd6_1180x842.png 424w, /__u/substackcdn.com/image/fetch/$s_!6AVZ!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff950476f-e05e-402b-b35a-5f80eea91cd6_1180x842.png 848w, /__u/substackcdn.com/image/fetch/$s_!6AVZ!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff950476f-e05e-402b-b35a-5f80eea91cd6_1180x842.png 1272w, /__u/substackcdn.com/image/fetch/$s_!6AVZ!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff950476f-e05e-402b-b35a-5f80eea91cd6_1180x842.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Patients with less Probability of Being Diabetic</figcaption></figure></div><p>T<strong>he model isn&#8217;t perfect.</strong> Some diabetic patients (red dots) fall on the &#8220;non-diabetic&#8221; side of the line. Some non-diabetic patients (purple dots) are on the &#8220;diabetic&#8221; side. The groups overlap and there&#8217;s noise and ambiguity - what we consider the error of the model.</p><p><strong>Repeat after me: No matter how good the AI model is, there will always be some error associated with it.</strong></p><p>That&#8217;s reality. Real patients don&#8217;t follow extremely perfect patterns. The best model in the world will still make mistakes.</p><div><hr></div><h4>From Boundary to Probability</h4><p>But is this such a rigid model? We just look at the line and say: &#8220;You&#8217;re above it? Diabetic. Below it? Not diabetic.&#8221; </p><p>Luckily no. Instead, we can ask: <em>&#8220;What does the composition of patients tell us?&#8221; </em>Look at the region <strong>above the line</strong>. We count all the patients who landed there. How many have diabetes? 131. How many don&#8217;t? 58. Total: 189 patients.</p><p>If a new patient&#8217;s data puts them in this region, what&#8217;s the probability they have diabetes? It&#8217;s simply: 131 &#247; 189 = <strong>69%, on average</strong>.</p><p>Now look at the region <strong>below the line</strong>. Count the patients: 137 with diabetes, 442 without. Total: 579. A new patient landing here has a 137 &#247; 579 = <strong>24%</strong> chance of diabetes, on average.</p><p>But the algorithm does more than that - <strong>the farther a patient sits from the line, the more confident the model becomes.</strong> A patient landing far above the boundary? The algorithm is more certain they're diabetic. A patient close to the line, straddling the edge? The algorithm hedges.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!dSy6!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6e008720-6cb9-48dc-9bc3-86dd948229ff_944x662.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!dSy6!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6e008720-6cb9-48dc-9bc3-86dd948229ff_944x662.png 424w, /__u/substackcdn.com/image/fetch/$s_!dSy6!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6e008720-6cb9-48dc-9bc3-86dd948229ff_944x662.png 848w, /__u/substackcdn.com/image/fetch/$s_!dSy6!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6e008720-6cb9-48dc-9bc3-86dd948229ff_944x662.png 1272w, /__u/substackcdn.com/image/fetch/$s_!dSy6!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6e008720-6cb9-48dc-9bc3-86dd948229ff_944x662.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!dSy6!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6e008720-6cb9-48dc-9bc3-86dd948229ff_944x662.png" width="541" height="379.38771186440675" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/6e008720-6cb9-48dc-9bc3-86dd948229ff_944x662.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:662,&quot;width&quot;:944,&quot;resizeWidth&quot;:541,&quot;bytes&quot;:103067,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://thedatajourney.substack.com/i/210065048?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6e008720-6cb9-48dc-9bc3-86dd948229ff_944x662.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!dSy6!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6e008720-6cb9-48dc-9bc3-86dd948229ff_944x662.png 424w, /__u/substackcdn.com/image/fetch/$s_!dSy6!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6e008720-6cb9-48dc-9bc3-86dd948229ff_944x662.png 848w, /__u/substackcdn.com/image/fetch/$s_!dSy6!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6e008720-6cb9-48dc-9bc3-86dd948229ff_944x662.png 1272w, /__u/substackcdn.com/image/fetch/$s_!dSy6!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6e008720-6cb9-48dc-9bc3-86dd948229ff_944x662.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Patients way above the line have a higher chance of being diabetic.</figcaption></figure></div><p>A patient that sits way above the line, can have as much as 96% chance of having diabetes, while a patient way below it can have values as low as 9%. This is the real power of a classification model.</p><div><hr></div><h4>The Data Quality Crisis</h4><p>Before we celebrate, there&#8217;s something important that happened.</p><p>Some patients in the dataset had a blood pressure reading of zero. That&#8217;s impossibl and it&#8217;s a data entry error. </p><p>When we trained the first version of the model with these bad readings included, something strange showed up: the boundary line gave blood pressure almost no weight. Glucose mattered a lot but blood pressure barely moved the line. Medically, that didn&#8217;t make sense. </p><p>The model is even stating that you are less likely to have diabetes if your blood pressure is higher, which is not something we are expecting.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!NEh9!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3e4129e5-3380-481b-89e8-cc46e83d31ec_1182x862.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!NEh9!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3e4129e5-3380-481b-89e8-cc46e83d31ec_1182x862.png 424w, /__u/substackcdn.com/image/fetch/$s_!NEh9!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3e4129e5-3380-481b-89e8-cc46e83d31ec_1182x862.png 848w, /__u/substackcdn.com/image/fetch/$s_!NEh9!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3e4129e5-3380-481b-89e8-cc46e83d31ec_1182x862.png 1272w, /__u/substackcdn.com/image/fetch/$s_!NEh9!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3e4129e5-3380-481b-89e8-cc46e83d31ec_1182x862.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!NEh9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3e4129e5-3380-481b-89e8-cc46e83d31ec_1182x862.png" width="1182" height="862" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3e4129e5-3380-481b-89e8-cc46e83d31ec_1182x862.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:862,&quot;width&quot;:1182,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:581068,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://thedatajourney.substack.com/i/210065048?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3e4129e5-3380-481b-89e8-cc46e83d31ec_1182x862.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!NEh9!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3e4129e5-3380-481b-89e8-cc46e83d31ec_1182x862.png 424w, /__u/substackcdn.com/image/fetch/$s_!NEh9!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3e4129e5-3380-481b-89e8-cc46e83d31ec_1182x862.png 848w, /__u/substackcdn.com/image/fetch/$s_!NEh9!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3e4129e5-3380-481b-89e8-cc46e83d31ec_1182x862.png 1272w, /__u/substackcdn.com/image/fetch/$s_!NEh9!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3e4129e5-3380-481b-89e8-cc46e83d31ec_1182x862.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>What if we remove those data points with blood pressure equals to 0? What will happen if we retrained the model after that?</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!E-Ub!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd1c12b1-86d7-4042-893c-6860c3771ea4_1074x696.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!E-Ub!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd1c12b1-86d7-4042-893c-6860c3771ea4_1074x696.png 424w, /__u/substackcdn.com/image/fetch/$s_!E-Ub!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd1c12b1-86d7-4042-893c-6860c3771ea4_1074x696.png 848w, /__u/substackcdn.com/image/fetch/$s_!E-Ub!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd1c12b1-86d7-4042-893c-6860c3771ea4_1074x696.png 1272w, /__u/substackcdn.com/image/fetch/$s_!E-Ub!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd1c12b1-86d7-4042-893c-6860c3771ea4_1074x696.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!E-Ub!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd1c12b1-86d7-4042-893c-6860c3771ea4_1074x696.png" width="1074" height="696" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/dd1c12b1-86d7-4042-893c-6860c3771ea4_1074x696.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:696,&quot;width&quot;:1074,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:378960,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://thedatajourney.substack.com/i/210065048?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd1c12b1-86d7-4042-893c-6860c3771ea4_1074x696.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!E-Ub!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd1c12b1-86d7-4042-893c-6860c3771ea4_1074x696.png 424w, /__u/substackcdn.com/image/fetch/$s_!E-Ub!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd1c12b1-86d7-4042-893c-6860c3771ea4_1074x696.png 848w, /__u/substackcdn.com/image/fetch/$s_!E-Ub!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd1c12b1-86d7-4042-893c-6860c3771ea4_1074x696.png 1272w, /__u/substackcdn.com/image/fetch/$s_!E-Ub!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd1c12b1-86d7-4042-893c-6860c3771ea4_1074x696.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>The boundary <em>rotated</em>. Suddenly, higher blood pressure became a much stronger signal for diabetes. What looked like &#8220;blood pressure doesn&#8217;t matter&#8221; was actually just noise from the data obscuring the true pattern. The bad data was pulling the entire line in the wrong direction.</p><p>This is why data scientists lose sleep: <strong>bad data doesn&#8217;t just make your model slightly worse. It can hide real patterns or create false ones.</strong></p><p>In healthcare, this matters a lot. If the model decides &#8220;high glucose doesn&#8217;t matter&#8221; because the data was corrupted, a doctor using that model might miss a real risk factor.</p><p><strong>For Machine Learning and AI, the quality of your data matters more than the quantity.</strong></p><div><hr></div><h4>The Weakness: Straight Lines Can&#8217;t Solve Everything</h4><p>Logistic regression draws one line. That line is the best possible separator for the data, but it&#8217;s still just one line.</p><p>What if your data doesn&#8217;t separate into two groups divided by a straight line?</p><p>Imagine this scenario: you have a cluster of ten diabetic patients in a region that&#8217;s otherwise surrounded by non-diabetic patients. Everyone around them is healthy, but these ten are diabetic. It might happen if there&#8217;s a genetic subgroup with unusual patterns, or if they&#8217;re all from the same community with different risk factors.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!CaTg!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6fff747e-01b7-44c7-aec6-ad40e2f6d846_1210x890.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!CaTg!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6fff747e-01b7-44c7-aec6-ad40e2f6d846_1210x890.png 424w, /__u/substackcdn.com/image/fetch/$s_!CaTg!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6fff747e-01b7-44c7-aec6-ad40e2f6d846_1210x890.png 848w, /__u/substackcdn.com/image/fetch/$s_!CaTg!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6fff747e-01b7-44c7-aec6-ad40e2f6d846_1210x890.png 1272w, /__u/substackcdn.com/image/fetch/$s_!CaTg!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6fff747e-01b7-44c7-aec6-ad40e2f6d846_1210x890.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!CaTg!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6fff747e-01b7-44c7-aec6-ad40e2f6d846_1210x890.png" width="564" height="414.8429752066116" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/6fff747e-01b7-44c7-aec6-ad40e2f6d846_1210x890.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:890,&quot;width&quot;:1210,&quot;resizeWidth&quot;:564,&quot;bytes&quot;:552096,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://thedatajourney.substack.com/i/210065048?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6fff747e-01b7-44c7-aec6-ad40e2f6d846_1210x890.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!CaTg!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6fff747e-01b7-44c7-aec6-ad40e2f6d846_1210x890.png 424w, /__u/substackcdn.com/image/fetch/$s_!CaTg!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6fff747e-01b7-44c7-aec6-ad40e2f6d846_1210x890.png 848w, /__u/substackcdn.com/image/fetch/$s_!CaTg!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6fff747e-01b7-44c7-aec6-ad40e2f6d846_1210x890.png 1272w, /__u/substackcdn.com/image/fetch/$s_!CaTg!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6fff747e-01b7-44c7-aec6-ad40e2f6d846_1210x890.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Concentration of Diabetic Patients under the boundary</figcaption></figure></div><p>Logistic regression will draw a straight line through the data. That line will classify those ten patients as &#8220;non-diabetic&#8221; because they&#8217;re in a non-diabetic region. The model will be wrong for all of them. But the line can&#8217;t change as it&#8217;s still a line. It has to be straight.</p><p>A line can rotate, it can shift up or down, but it can never bend around a cluster in the middle of an ocean of different data points.</p><p>The logistic regression model will make these mistakes and move on. It can&#8217;t do better without fundamentally changing what it is, similar to linear regression.</p><p>Remember John? Our decision tree student who nailed the house price predictions? He works for classification too. No straight line, just a cascade of yes-or-no questions: 'Is glucose above 150? Yes? Is blood pressure above 80?' Each answer carves out a region. Tree based models are extremely flexible!</p><div><hr></div><h4>When Trees Get It Right (Sort Of)</h4><p>Decision trees work for classification too. Instead of finding one line, they build a flowchart of yes-or-no questions: &#8220;Is glucose above 150? Yes? Is blood pressure above 80? No? Then probably not diabetic.&#8221;</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!qutu!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffe4ea091-78ad-49ca-a80d-cf6faa66f011_982x644.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!qutu!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffe4ea091-78ad-49ca-a80d-cf6faa66f011_982x644.png 424w, /__u/substackcdn.com/image/fetch/$s_!qutu!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffe4ea091-78ad-49ca-a80d-cf6faa66f011_982x644.png 848w, /__u/substackcdn.com/image/fetch/$s_!qutu!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffe4ea091-78ad-49ca-a80d-cf6faa66f011_982x644.png 1272w, /__u/substackcdn.com/image/fetch/$s_!qutu!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffe4ea091-78ad-49ca-a80d-cf6faa66f011_982x644.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!qutu!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffe4ea091-78ad-49ca-a80d-cf6faa66f011_982x644.png" width="554" height="363.3156822810591" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/fe4ea091-78ad-49ca-a80d-cf6faa66f011_982x644.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:644,&quot;width&quot;:982,&quot;resizeWidth&quot;:554,&quot;bytes&quot;:368825,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://thedatajourney.substack.com/i/210065048?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffe4ea091-78ad-49ca-a80d-cf6faa66f011_982x644.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!qutu!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffe4ea091-78ad-49ca-a80d-cf6faa66f011_982x644.png 424w, /__u/substackcdn.com/image/fetch/$s_!qutu!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffe4ea091-78ad-49ca-a80d-cf6faa66f011_982x644.png 848w, /__u/substackcdn.com/image/fetch/$s_!qutu!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffe4ea091-78ad-49ca-a80d-cf6faa66f011_982x644.png 1272w, /__u/substackcdn.com/image/fetch/$s_!qutu!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffe4ea091-78ad-49ca-a80d-cf6faa66f011_982x644.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Decision Tree Model trained for our Classification Problem</figcaption></figure></div><p>The boundaries they create are complex and they look like boxes, not lines. Want to split off that cluster of ten diabetic patients who live in a non-diabetic area? Easy. A tree can draw a rectangle around them and label it &#8220;diabetic,&#8221; even if they&#8217;re surrounded by non-diabetic patients.</p><p>Trees can separate <em>any</em> pattern, no matter how jagged or weird, but remember: they overfit like crazy.</p><p>Let me show you why again. Imagine a single diabetic patient in the dataset: blood pressure 30, glucose 50. That&#8217;s an unusual combination, and it might be an outlier or even a data error (low glucose is medically weird). The decision tree looks at this point and asks: &#8220;Is there a pattern here?&#8221;</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!P-9j!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61399866-377d-4cd7-b3ea-e91ba44fa6bd_966x674.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!P-9j!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61399866-377d-4cd7-b3ea-e91ba44fa6bd_966x674.png 424w, /__u/substackcdn.com/image/fetch/$s_!P-9j!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61399866-377d-4cd7-b3ea-e91ba44fa6bd_966x674.png 848w, /__u/substackcdn.com/image/fetch/$s_!P-9j!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61399866-377d-4cd7-b3ea-e91ba44fa6bd_966x674.png 1272w, /__u/substackcdn.com/image/fetch/$s_!P-9j!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61399866-377d-4cd7-b3ea-e91ba44fa6bd_966x674.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!P-9j!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61399866-377d-4cd7-b3ea-e91ba44fa6bd_966x674.png" width="468" height="326.53416149068323" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/61399866-377d-4cd7-b3ea-e91ba44fa6bd_966x674.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:674,&quot;width&quot;:966,&quot;resizeWidth&quot;:468,&quot;bytes&quot;:64840,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://thedatajourney.substack.com/i/210065048?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61399866-377d-4cd7-b3ea-e91ba44fa6bd_966x674.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!P-9j!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61399866-377d-4cd7-b3ea-e91ba44fa6bd_966x674.png 424w, /__u/substackcdn.com/image/fetch/$s_!P-9j!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61399866-377d-4cd7-b3ea-e91ba44fa6bd_966x674.png 848w, /__u/substackcdn.com/image/fetch/$s_!P-9j!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61399866-377d-4cd7-b3ea-e91ba44fa6bd_966x674.png 1272w, /__u/substackcdn.com/image/fetch/$s_!P-9j!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61399866-377d-4cd7-b3ea-e91ba44fa6bd_966x674.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>No. It&#8217;s one patient. But the tree doesn&#8217;t care. It asks the next question: &#8220;How can I split this data to get this one point right?&#8221; And it finds a way. It draws a box around (blood pressure: 25&#8211;35, glucose: 45&#8211;55) and labels it &#8220;diabetic.&#8221;</p><p>Now a new patient comes in with blood pressure 32 and glucose 48, values that fall in that box. The tree predicts &#8220;diabetic.&#8221; But we have no idea if that&#8217;s true, we&#8217;re predicting based on a single patient from months ago. If that one patient was a data entry error or an unusual outlier, the tree is now making bad predictions for everyone like them.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!dlCd!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F52d9a8cd-94d3-42a4-a3f7-5a7ed2ec25eb_1048x706.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!dlCd!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F52d9a8cd-94d3-42a4-a3f7-5a7ed2ec25eb_1048x706.png 424w, /__u/substackcdn.com/image/fetch/$s_!dlCd!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F52d9a8cd-94d3-42a4-a3f7-5a7ed2ec25eb_1048x706.png 848w, /__u/substackcdn.com/image/fetch/$s_!dlCd!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F52d9a8cd-94d3-42a4-a3f7-5a7ed2ec25eb_1048x706.png 1272w, /__u/substackcdn.com/image/fetch/$s_!dlCd!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F52d9a8cd-94d3-42a4-a3f7-5a7ed2ec25eb_1048x706.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!dlCd!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F52d9a8cd-94d3-42a4-a3f7-5a7ed2ec25eb_1048x706.png" width="556" height="374.55725190839695" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/52d9a8cd-94d3-42a4-a3f7-5a7ed2ec25eb_1048x706.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:706,&quot;width&quot;:1048,&quot;resizeWidth&quot;:556,&quot;bytes&quot;:422360,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://thedatajourney.substack.com/i/210065048?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F52d9a8cd-94d3-42a4-a3f7-5a7ed2ec25eb_1048x706.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!dlCd!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F52d9a8cd-94d3-42a4-a3f7-5a7ed2ec25eb_1048x706.png 424w, /__u/substackcdn.com/image/fetch/$s_!dlCd!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F52d9a8cd-94d3-42a4-a3f7-5a7ed2ec25eb_1048x706.png 848w, /__u/substackcdn.com/image/fetch/$s_!dlCd!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F52d9a8cd-94d3-42a4-a3f7-5a7ed2ec25eb_1048x706.png 1272w, /__u/substackcdn.com/image/fetch/$s_!dlCd!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F52d9a8cd-94d3-42a4-a3f7-5a7ed2ec25eb_1048x706.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">In the bottom right and left corners, we are predicting diabetes just because of 1 single patient.</figcaption></figure></div><p><strong>This is the overfitting danger.</strong> A decision tree for classification will happily partition your data into a thousand tiny boxes, each one driven by a single patient or a quirk in the training set. It will be nearly perfect on the training data (because it&#8217;s memorizing it) and terrible on new data (because the new data doesn&#8217;t conform to those overly specific boxes).</p><div><hr></div><h4>Next: Meet the Neural Network</h4><p>Hopefully you understand that the bias vs. variance trade-off is everpresent. The fundamental tension never goes away: you can be simple and stable but miss patterns (like linear or logistic regressions). Or flexible and complex but memorize garbage (like decision trees).</p><p>But AI today is clearly doing more than this. These four models can't explain what we see. There must be a model that doesn&#8217;t draw lines or ask yes-or-no questions.</p><p>What if a model could learn that blood pressure and glucose only matter together, in specific combinations, and build its own sense of what "risk" looks like instead of relying on the boundary we hand it? That's the job of a model called neural network. Instead of one boundary, it passes the data through layers, each one learning a slightly different version of the pattern, until the combination itself becomes the signal.</p><p>And it&#8217;s going to be our focus  in two weeks.</p><p>&#8212; Ivo</p>]]></content:encoded></item><item><title><![CDATA[Weekly AI Digest: Contradictory signals]]></title><description><![CDATA[Week 31, 2026]]></description><link>https://thedatajourney.substack.com/p/weekly-ai-digest-contradictory-signals</link><guid isPermaLink="false">https://thedatajourney.substack.com/p/weekly-ai-digest-contradictory-signals</guid><dc:creator><![CDATA[Ivo Bernardo]]></dc:creator><pubDate>Sat, 01 Aug 2026 09:49:42 GMT</pubDate><enclosure url="https://substackcdn.com/image/youtube/w_728,c_limit/wlYa8NV5k-U" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Hey everyone, </p><p>This week, the South Korean market crashed due to high volatility in memory chip stocks, while new evidence suggests that AI agents are becoming a significant cybersecurity threat. </p><p>At the same time, the recent OpenAI breach on Hugging Face is under scrutiny, leaving many wondering whether it was a genuine hack or simply human error. </p><p>Amidst mixed signals like these, let&#8217;s dive into the Week 31 digest.</p><div><hr></div><h4>&#187; Stat of the Week: The Internet is Now Read-Only</h4><p>Bots hit 58% of all web traffic, per Cloudflare, three years ahead of expectations.&#185; </p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!LnIf!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fce5ea717-1b6b-4609-a6aa-8a938a2e2e1a_1448x1086.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!LnIf!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fce5ea717-1b6b-4609-a6aa-8a938a2e2e1a_1448x1086.png 424w, /__u/substackcdn.com/image/fetch/$s_!LnIf!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fce5ea717-1b6b-4609-a6aa-8a938a2e2e1a_1448x1086.png 848w, /__u/substackcdn.com/image/fetch/$s_!LnIf!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fce5ea717-1b6b-4609-a6aa-8a938a2e2e1a_1448x1086.png 1272w, /__u/substackcdn.com/image/fetch/$s_!LnIf!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fce5ea717-1b6b-4609-a6aa-8a938a2e2e1a_1448x1086.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!LnIf!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fce5ea717-1b6b-4609-a6aa-8a938a2e2e1a_1448x1086.png" width="645" height="483.75" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/ce5ea717-1b6b-4609-a6aa-8a938a2e2e1a_1448x1086.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1086,&quot;width&quot;:1448,&quot;resizeWidth&quot;:645,&quot;bytes&quot;:1670147,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://thedatajourney.substack.com/i/209356432?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fce5ea717-1b6b-4609-a6aa-8a938a2e2e1a_1448x1086.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!LnIf!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fce5ea717-1b6b-4609-a6aa-8a938a2e2e1a_1448x1086.png 424w, /__u/substackcdn.com/image/fetch/$s_!LnIf!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fce5ea717-1b6b-4609-a6aa-8a938a2e2e1a_1448x1086.png 848w, /__u/substackcdn.com/image/fetch/$s_!LnIf!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fce5ea717-1b6b-4609-a6aa-8a938a2e2e1a_1448x1086.png 1272w, /__u/substackcdn.com/image/fetch/$s_!LnIf!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fce5ea717-1b6b-4609-a6aa-8a938a2e2e1a_1448x1086.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>AI agents alone grew 8,000% year-over-year, with the dead internet theory becoming a reality every day. </p><div><hr></div><h4>&#187; Industry Highlights: Market &amp; Product</h4><p><strong>Recursive Superintelligence seals $400 million AWS deal</strong> to build self-improving AI systems.&#178; The company burned most of its $650 million Series A before this. Recursion&#8217;s CEO Socher said this is &#8220;one of the smallest compute deals we&#8217;re going to sign in the next few years.&#8221;</p><p><strong>Speaking of Amazon, during their earnings call, the company  killed most of its Nova lineup.</strong> Premier, Omni, Reel, Canvas were all deprecated to fund one frontier model at re:Invent.&#8310; The company with the best cloud infrastructure for AI gave up building them, a signal that may tell you where the profitability is.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!Ssn5!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f38c766-7778-4112-b650-9b695dacdd9d_1614x1208.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!Ssn5!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f38c766-7778-4112-b650-9b695dacdd9d_1614x1208.png 424w, /__u/substackcdn.com/image/fetch/$s_!Ssn5!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f38c766-7778-4112-b650-9b695dacdd9d_1614x1208.png 848w, /__u/substackcdn.com/image/fetch/$s_!Ssn5!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f38c766-7778-4112-b650-9b695dacdd9d_1614x1208.png 1272w, /__u/substackcdn.com/image/fetch/$s_!Ssn5!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f38c766-7778-4112-b650-9b695dacdd9d_1614x1208.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!Ssn5!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f38c766-7778-4112-b650-9b695dacdd9d_1614x1208.png" width="1614" height="1208" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/9f38c766-7778-4112-b650-9b695dacdd9d_1614x1208.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1208,&quot;width&quot;:1614,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;TradingView chart&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="TradingView chart" title="TradingView chart" srcset="/__u/substackcdn.com/image/fetch/$s_!Ssn5!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f38c766-7778-4112-b650-9b695dacdd9d_1614x1208.png 424w, /__u/substackcdn.com/image/fetch/$s_!Ssn5!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f38c766-7778-4112-b650-9b695dacdd9d_1614x1208.png 848w, /__u/substackcdn.com/image/fetch/$s_!Ssn5!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f38c766-7778-4112-b650-9b695dacdd9d_1614x1208.png 1272w, /__u/substackcdn.com/image/fetch/$s_!Ssn5!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f38c766-7778-4112-b650-9b695dacdd9d_1614x1208.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Amazon&#8217;s stock jumped more than 10% after this weeks&#8217; earnings call</figcaption></figure></div><p><strong>Anthropic launched Claude Opus 5</strong>, halving the price while matching Fable 5 on coding benchmarks.&#8308; Opus 5 topped Frontier-Bench and CursorBench, making it the model developers want to run. Meanwhile, Microsoft logged $3.2 billion in paper gains from its Anthropic stake in a single quarter.&#8309; <strong>The pricing moat collapsed, </strong>as Opus 5 runs 0.5% behind Fable 5 on code while costing half as much.&#8312; </p><p><strong>Anthropic and OpenAI both had their models escape their sandboxes.</strong> After the hugging face hack last week by OpenAI, it was now Anthropics&#8217; turn. Three unnamed organizations got hit by the company, and the company announced it almost like a new feature.&#8311; Is this FOMO on the &#8220;<em>our models are weapons</em>&#8221; trend? With contradictory signals coming from both sides, only time will tell if these are genuine hacks or just marketing ploys.</p><p><strong>NVIDIA backed Ilya Sutskever&#8217;s Safe Superintelligence Inc.</strong> with an investment and access to the Vera Rubin platform, enough compute to scale SSI by an order of magnitude.&#185;&#8304; Sutskever&#8217;s been working on SSI since 2024, and NVIDIA just decided his &#8220;straight-shot&#8221; approach to safe superintelligence was worth betting on. </p><p><strong>Finally, Google bought a $9.4 billion stake in SpaceX,</strong> marking a 6% ownership stake.&#185;&#185; The move is nominally about satellite internet for rural access, but it&#8217;s really about cloud compute distribution and avoiding dependence on terrestrial broadband. SpaceX gets capital and Google gets distribution. </p><div><hr></div><h4>&#187; Economic Highlights: The World of AI</h4><p>Over the past few weeks, <strong>the South Korean stock market has experienced a historic meltdown, with the KOSPI index plunging nearly 44% in just 40 days and erasing roughly $2 trillion in market value.</strong> The crash centered almost entirely on Samsung Electronics and SK Hynix, two memory chip giants that had previously ridden a massive wave of AI infrastructure euphoria to historic highs.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!WM7H!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7a591f0-5e6d-4555-9fd8-e6c29dc35aa8_1772x664.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!WM7H!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7a591f0-5e6d-4555-9fd8-e6c29dc35aa8_1772x664.png 424w, /__u/substackcdn.com/image/fetch/$s_!WM7H!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7a591f0-5e6d-4555-9fd8-e6c29dc35aa8_1772x664.png 848w, /__u/substackcdn.com/image/fetch/$s_!WM7H!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7a591f0-5e6d-4555-9fd8-e6c29dc35aa8_1772x664.png 1272w, /__u/substackcdn.com/image/fetch/$s_!WM7H!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7a591f0-5e6d-4555-9fd8-e6c29dc35aa8_1772x664.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!WM7H!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7a591f0-5e6d-4555-9fd8-e6c29dc35aa8_1772x664.png" width="1456" height="546" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c7a591f0-5e6d-4555-9fd8-e6c29dc35aa8_1772x664.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:546,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:100370,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://thedatajourney.substack.com/i/209356432?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7a591f0-5e6d-4555-9fd8-e6c29dc35aa8_1772x664.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!WM7H!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7a591f0-5e6d-4555-9fd8-e6c29dc35aa8_1772x664.png 424w, /__u/substackcdn.com/image/fetch/$s_!WM7H!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7a591f0-5e6d-4555-9fd8-e6c29dc35aa8_1772x664.png 848w, /__u/substackcdn.com/image/fetch/$s_!WM7H!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7a591f0-5e6d-4555-9fd8-e6c29dc35aa8_1772x664.png 1272w, /__u/substackcdn.com/image/fetch/$s_!WM7H!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7a591f0-5e6d-4555-9fd8-e6c29dc35aa8_1772x664.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">KOSPI Index nosedived during July</figcaption></figure></div><p><strong>Sam Altman wants to pump the brakes.</strong> An OpenAI model broke out of a sandbox, hacked Hugging Face with zero-day exploits, and did what Altman called an &#8220;extremely sci-fi cyber incident.&#8221;&#179; Six months ago, he dismissed pause proposals as &#8220;missing technical nuance.&#8221;</p><p><strong>A German court ruled Suno violated copyright by training on GEMA-represented songs without permission.&#185;&#178;</strong> The Munich Regional Court found evidence that Suno memorized and reproduced six songs during training. It&#8217;s the first binding European ruling on AI music training, and it means Suno now has to disclose revenues so damages can be calculated. </p><div><hr></div><h4>&#187; Track of the Week: Watch your Prompts</h4><p>No words needed for this one. Check out the video below:</p><div id="youtube2-wlYa8NV5k-U" class="youtube-wrap" data-attrs="{&quot;videoId&quot;:&quot;wlYa8NV5k-U&quot;,&quot;startTime&quot;:null,&quot;endTime&quot;:null}" data-component-name="Youtube2ToDOM"><div class="youtube-inner"><iframe src="https://www.youtube-nocookie.com/embed/wlYa8NV5k-U?rel=0&amp;autoplay=0&amp;showinfo=0&amp;enablejsapi=0" frameborder="0" loading="lazy" gesture="media" allow="autoplay; fullscreen" allowautoplay="true" allowfullscreen="true" width="728" height="409"></iframe></div></div><div><hr></div><h4>Closing</h4><p>If Week 31 proved anything, it&#8217;s that the AI ecosystem is operating in two completely different realities. On one side, Wall Street and tech giants are facing increasing uncertainty, with extreme volatility of ups (Microsoft and Amazon) and downs (South Korean market).  </p><p>On the other side, labs are practically treating security breaches and &#8220;sandbox escapes&#8221; like badge-of-honor product announcements to keep AGI hype alive.</p><p>Whether we&#8217;re watching the birth of genuine autonomous cyber threats or just the most aggressive PR war in tech history <strong>will become clear</strong> in the next few months.</p><p>See you next week!</p><p>&#8212; Ivo</p><div><hr></div><p>&#185; <a href="https://thenextweb.com/news/bots-outnumber-humans-internet-ai-agents-traffic">https://thenextweb.com/news/bots-outnumber-humans-internet-ai-agents-traffic</a></p><p>&#178; <a href="https://techcrunch.com/2026/07/28/recursive-superintelligence-signs-400-compute-deal-with-amazon/">https://techcrunch.com/2026/07/28/recursive-superintelligence-signs-400-compute-deal-with-amazon/</a></p><p>&#179; <a href="https://techcrunch.com/2026/07/28/sam-altman-is-ready-to-decelerate/">https://techcrunch.com/2026/07/28/sam-altman-is-ready-to-decelerate/</a></p><p>&#8308; <a href="https://www.anthropic.com/news/claude-opus-5">https://www.anthropic.com/news/claude-opus-5</a></p><p>&#8309; <a href="https://techcrunch.com/2026/07/29/microsoft-logs-3-2b-from-anthropic-investment-but-openai-was-a-mixed-bag/">https://techcrunch.com/2026/07/29/microsoft-logs-3-2b-from-anthropic-investment-but-openai-was-a-mixed-bag/</a></p><p>&#8310; <a href="https://thenextweb.com/news/amazon-winds-down-nova-ai-models-frontier-model-research">https://thenextweb.com/news/amazon-winds-down-nova-ai-models-frontier-model-research</a></p><p>&#8311; <a href="https://venturebeat.com/security/not-just-openai-now-anthropic-says-its-internal-models-got-online-and-cyberattacked-3-other-organizations">https://venturebeat.com/security/not-just-openai-now-anthropic-says-its-internal-models-got-online-and-cyberattacked-3-other-organizations</a></p><p>&#8312; <a href="https://www.anthropic.com/news/claude-opus-5">https://www.anthropic.com/news/claude-opus-5</a></p><p>&#8313; <a href="https://thenextweb.com/news/bots-outnumber-humans-internet-ai-agents-traffic">https://thenextweb.com/news/bots-outnumber-humans-internet-ai-agents-traffic</a></p><p>&#185;&#8304; <a href="https://nvidianews.nvidia.com/news/ilya-sutskevers-safe-superintelligence-inc-and-nvidia-announce-long-term-strategic-partnership">https://nvidianews.nvidia.com/news/ilya-sutskevers-safe-superintelligence-inc-and-nvidia-announce-long-term-strategic-partnership</a></p><p>&#185;&#185; <a href="https://www.wsj.com/tech/google-discloses-94-1-billion-in-spacex-stock-marking-6-stake-91655d7c">https://www.wsj.com/tech/google-discloses-94-1-billion-in-spacex-stock-marking-6-stake-91655d7c</a></p><p>&#185;&#178; <a href="https://thenextweb.com/news/german-court-suno-ai-music-copyright-gema">https://thenextweb.com/news/german-court-suno-ai-music-copyright-gema</a></p>]]></content:encoded></item><item><title><![CDATA[Are Home Teams Favoured by Referees in Football/Soccer?]]></title><description><![CDATA[Data Storytelling Series Chapter 1]]></description><link>https://thedatajourney.substack.com/p/are-home-teams-favoured-by-referees</link><guid isPermaLink="false">https://thedatajourney.substack.com/p/are-home-teams-favoured-by-referees</guid><dc:creator><![CDATA[Ivo Bernardo]]></dc:creator><pubDate>Mon, 27 Jul 2026 16:17:28 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!noRh!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd8aa1d86-4b96-44cb-8c1f-58845ee98523_2026x1888.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Hello readers,</p><p>The internet loves a good argument, and it&#8217;s awfully tempting to jump on a bandwagon before checking the facts. But since one of my roles in the world is being a data scientist, I have a habit of wanting to run the numbers before I form an opinion.</p><p>Lately, I&#8217;ve turned this into a routine: whenever a debate sparks up over something we can measure, I track down the data to get the facts straight. I finally reached my limit with the wild guessing people rely on to make their points, so I decided to use my skills to let the numbers do the talking instead. And since I&#8217;m already going down these rabbit holes just to shut up my friends, I figured it would be cool to start sharing my findings with you.</p><p>This weekend, with the World Cup barely cold and half the planet still arguing about it - the US included, now that they&#8217;ve apparently discovered soccer exists (it&#8217;s <em>football</em>, if you&#8217;re reading this from the right side of the Atlantic) - I got into a long argument with a friend.</p><p><strong>His theory: big teams are favoured because the referees fold under pressure and respect those clubs more. My theory: that&#8217;s his perception because of a strong home bias - refs may feel under pressure due to home crowds.</strong></p><p>As you may know, most football fans are not exactly rational about referees. Two people supporting different clubs can watch the exact same tackle and see two completely different fouls. But my friend&#8217;s argument left me with a question. Do big teams really get favored by referees? Are there other factors creating this perception? Or is no one favored at all?</p><p>Normally, I&#8217;d be just as biased as anyone else. But since I recently stopped following football after some <a href="https://www.daily-sun.com/post/839031">club elections</a> left me totally disillusioned about my club, I think I finally have the emotional maturity and detachment to investigate this purely through the data.</p><p>Gather around, folks. Let&#8217;s find out if referees fold under pressure.</p><div><hr></div><h3>First, the data</h3><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!Eydn!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9d625f9f-fdb7-40bc-a7b0-c1f950d8585e_1936x1101.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!Eydn!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9d625f9f-fdb7-40bc-a7b0-c1f950d8585e_1936x1101.png 424w, /__u/substackcdn.com/image/fetch/$s_!Eydn!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9d625f9f-fdb7-40bc-a7b0-c1f950d8585e_1936x1101.png 848w, /__u/substackcdn.com/image/fetch/$s_!Eydn!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9d625f9f-fdb7-40bc-a7b0-c1f950d8585e_1936x1101.png 1272w, /__u/substackcdn.com/image/fetch/$s_!Eydn!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9d625f9f-fdb7-40bc-a7b0-c1f950d8585e_1936x1101.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!Eydn!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9d625f9f-fdb7-40bc-a7b0-c1f950d8585e_1936x1101.png" width="1456" height="828" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/9d625f9f-fdb7-40bc-a7b0-c1f950d8585e_1936x1101.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:false,&quot;imageSize&quot;:&quot;normal&quot;,&quot;height&quot;:828,&quot;width&quot;:1456,&quot;resizeWidth&quot;:0,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:&quot;center&quot;,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!Eydn!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9d625f9f-fdb7-40bc-a7b0-c1f950d8585e_1936x1101.png 424w, /__u/substackcdn.com/image/fetch/$s_!Eydn!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9d625f9f-fdb7-40bc-a7b0-c1f950d8585e_1936x1101.png 848w, /__u/substackcdn.com/image/fetch/$s_!Eydn!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9d625f9f-fdb7-40bc-a7b0-c1f950d8585e_1936x1101.png 1272w, /__u/substackcdn.com/image/fetch/$s_!Eydn!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9d625f9f-fdb7-40bc-a7b0-c1f950d8585e_1936x1101.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Everything here comes from <a href="https://www.football-data.co.uk/">football-data.co.uk</a>, which is a small miracle of a website: free, tidy, and maintained by people who clearly love data (<a href="https://x.com/intent/follow?original_referer=https%3A%2F%2Fwww.football-data.co.uk%2F&amp;ref_src=twsrc%5Etfw%7Ctwcamp%5Ebuttonembed%7Ctwterm%5Efollow%7Ctwgr%5E12Xpert&amp;screen_name=12Xpert">Joseph Buchdahl</a> is a legend in the sports analytics space). It has results, odds, fouls, cards - the whole grammar of a football match, going back decades.</p><p>There&#8217;s one catch with this data for our purpose. Fouls and bookings only show up consistently from the 2017/18 season for a chunk of the leagues, so for those competitions we start the clock there and throw out anything earlier. </p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!GpAM!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f06c2b8-ccc7-436a-8306-fe2e9723bf43_682x278.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!GpAM!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f06c2b8-ccc7-436a-8306-fe2e9723bf43_682x278.png 424w, /__u/substackcdn.com/image/fetch/$s_!GpAM!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f06c2b8-ccc7-436a-8306-fe2e9723bf43_682x278.png 848w, /__u/substackcdn.com/image/fetch/$s_!GpAM!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f06c2b8-ccc7-436a-8306-fe2e9723bf43_682x278.png 1272w, /__u/substackcdn.com/image/fetch/$s_!GpAM!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f06c2b8-ccc7-436a-8306-fe2e9723bf43_682x278.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!GpAM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f06c2b8-ccc7-436a-8306-fe2e9723bf43_682x278.png" width="682" height="278" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3f06c2b8-ccc7-436a-8306-fe2e9723bf43_682x278.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:278,&quot;width&quot;:682,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:35431,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://thedatajourney.substack.com/i/208464980?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f06c2b8-ccc7-436a-8306-fe2e9723bf43_682x278.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="/__u/substackcdn.com/image/fetch/$s_!GpAM!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f06c2b8-ccc7-436a-8306-fe2e9723bf43_682x278.png 424w, /__u/substackcdn.com/image/fetch/$s_!GpAM!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f06c2b8-ccc7-436a-8306-fe2e9723bf43_682x278.png 848w, /__u/substackcdn.com/image/fetch/$s_!GpAM!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f06c2b8-ccc7-436a-8306-fe2e9723bf43_682x278.png 1272w, /__u/substackcdn.com/image/fetch/$s_!GpAM!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f06c2b8-ccc7-436a-8306-fe2e9723bf43_682x278.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Small sample of our table</figcaption></figure></div><p>Now, the temptation - and this is exactly where most bar-stool arguments fall apart - is to just count yellow cards/fouls as the referee punishment. Big team gets fewer cards at home, case closed, referees are bought. </p><p>Except cards on their own tell you almost nothing, because they depend on how much you foul in the first place. <strong>A team that dominates the ball and camps in the opponent&#8217;s half commits fewer fouls, and fewer fouls means fewer chances to get booked.</strong></p><p>So the real unit of this whole study isn&#8217;t cards. It&#8217;s cards per foul - the referee&#8217;s conversion rate. Of all the fouls you commit, how many actually get punished? And even that number must be looked with caution. </p><p>Two pairs of columns carry the referee intervention in our dataset:</p><ul><li><p><strong>HY / AY</strong> - yellow cards, home and away.</p></li><li><p><strong>HF / AF</strong> - fouls committed, home and away. The denominator. Cards per foul is the entire question.</p></li></ul><p>A referee&#8217;s whistle doesn&#8217;t happen in a vacuum, though, so before I blame the crowd I have to strip out everything else that could explain the pattern. Maybe the big teams get the calls because they&#8217;re simply better, not because they&#8217;re home - so I take the betting odds, strip out the bookmaker&#8217;s margin, and use them to measure how lopsided each match was on paper. Maybe the away team just fouls more because it spends ninety minutes defending so we&#8217;ll use shots and possession to account for territory. And maybe it&#8217;s game state: chasing a 0-1 at half-time changes how recklessly you tackle, so the half-time score goes in too. We&#8217;ll take this into account in our analysis.</p><p>My bet (and the argument I had with my friend) is that home bias (clubs that play at home) has a much larger impact on the ref than &#8220;bigger clubs&#8221;. On the data, I&#8217;ll also use two important angles to see that:</p><ul><li><p>The first is reverse fixtures. Every pair of teams plays twice a season, once at each ground. Line those two matches up side by side and the teams are very similar - just the moment of the season and coach switches may carry some changes - so whatever&#8217;s left over when you flip home and away can&#8217;t be blamed on one side simply being better. It&#8217;s one of the best clean comparisons we get.</p></li><li><p>Buried in the data is a natural experiment nobody designed on purpose: the COVID seasons, when matches were played in empty stadiums. If the home crowd is what bends the referee, then take the crowd away and the effect should shrink. So every match gets flagged as CROWD, EMPTY, or EXCLUDED.</p></li></ul><p>The rest is bookkeeping: which league, which season, which two clubs, plus the raw results and shot counts kept around in case we need them.</p><div><hr></div><h3>Exploring the data</h3><p>With the data in hand, the first thing worth looking at is which leagues foul the most.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!qyWC!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fae480b67-328f-4120-b71c-4ef38344673d_423x506.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!qyWC!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fae480b67-328f-4120-b71c-4ef38344673d_423x506.png 424w, /__u/substackcdn.com/image/fetch/$s_!qyWC!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fae480b67-328f-4120-b71c-4ef38344673d_423x506.png 848w, /__u/substackcdn.com/image/fetch/$s_!qyWC!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fae480b67-328f-4120-b71c-4ef38344673d_423x506.png 1272w, /__u/substackcdn.com/image/fetch/$s_!qyWC!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fae480b67-328f-4120-b71c-4ef38344673d_423x506.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!qyWC!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fae480b67-328f-4120-b71c-4ef38344673d_423x506.png" width="423" height="506" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/ae480b67-328f-4120-b71c-4ef38344673d_423x506.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:506,&quot;width&quot;:423,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:52690,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://thedatajourney.substack.com/i/208464980?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fae480b67-328f-4120-b71c-4ef38344673d_423x506.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!qyWC!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fae480b67-328f-4120-b71c-4ef38344673d_423x506.png 424w, /__u/substackcdn.com/image/fetch/$s_!qyWC!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fae480b67-328f-4120-b71c-4ef38344673d_423x506.png 848w, /__u/substackcdn.com/image/fetch/$s_!qyWC!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fae480b67-328f-4120-b71c-4ef38344673d_423x506.png 1272w, /__u/substackcdn.com/image/fetch/$s_!qyWC!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fae480b67-328f-4120-b71c-4ef38344673d_423x506.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Home and Away Fouls per Game</figcaption></figure></div><p>No surprise to me: the portuguese Primeira Liga is one of the most foul-heavy top flight in the dataset at 29.4 fouls a game, second only to Italy's Serie B across all fourteen divisions. And Portugal is top of the lot for fouls committed by the home team.</p><p>The pattern is geographic. Italy, Portugal and Spain fill the top five, while the Eredivisie, the Premier League and the Championship sit at the bottom on 21 to 22. <strong>Southern European games get stopped far more often than northern <span>ones.</span></strong></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!noRh!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd8aa1d86-4b96-44cb-8c1f-58845ee98523_2026x1888.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!noRh!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd8aa1d86-4b96-44cb-8c1f-58845ee98523_2026x1888.png 424w, /__u/substackcdn.com/image/fetch/$s_!noRh!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd8aa1d86-4b96-44cb-8c1f-58845ee98523_2026x1888.png 848w, /__u/substackcdn.com/image/fetch/$s_!noRh!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd8aa1d86-4b96-44cb-8c1f-58845ee98523_2026x1888.png 1272w, /__u/substackcdn.com/image/fetch/$s_!noRh!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd8aa1d86-4b96-44cb-8c1f-58845ee98523_2026x1888.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!noRh!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd8aa1d86-4b96-44cb-8c1f-58845ee98523_2026x1888.png" width="1456" height="1357" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/d8aa1d86-4b96-44cb-8c1f-58845ee98523_2026x1888.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1357,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!noRh!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd8aa1d86-4b96-44cb-8c1f-58845ee98523_2026x1888.png 424w, /__u/substackcdn.com/image/fetch/$s_!noRh!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd8aa1d86-4b96-44cb-8c1f-58845ee98523_2026x1888.png 848w, /__u/substackcdn.com/image/fetch/$s_!noRh!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd8aa1d86-4b96-44cb-8c1f-58845ee98523_2026x1888.png 1272w, /__u/substackcdn.com/image/fetch/$s_!noRh!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd8aa1d86-4b96-44cb-8c1f-58845ee98523_2026x1888.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>The next thing that jumps out is that home and away teams foul ratio has a gap that changes between every league.<br></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!P7Ei!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe40a194d-5d9b-4b32-aac0-a2f24e41c0b4_1589x1091.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!P7Ei!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe40a194d-5d9b-4b32-aac0-a2f24e41c0b4_1589x1091.png 424w, /__u/substackcdn.com/image/fetch/$s_!P7Ei!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe40a194d-5d9b-4b32-aac0-a2f24e41c0b4_1589x1091.png 848w, /__u/substackcdn.com/image/fetch/$s_!P7Ei!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe40a194d-5d9b-4b32-aac0-a2f24e41c0b4_1589x1091.png 1272w, /__u/substackcdn.com/image/fetch/$s_!P7Ei!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe40a194d-5d9b-4b32-aac0-a2f24e41c0b4_1589x1091.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!P7Ei!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe40a194d-5d9b-4b32-aac0-a2f24e41c0b4_1589x1091.png" width="1456" height="1000" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e40a194d-5d9b-4b32-aac0-a2f24e41c0b4_1589x1091.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1000,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!P7Ei!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe40a194d-5d9b-4b32-aac0-a2f24e41c0b4_1589x1091.png 424w, /__u/substackcdn.com/image/fetch/$s_!P7Ei!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe40a194d-5d9b-4b32-aac0-a2f24e41c0b4_1589x1091.png 848w, /__u/substackcdn.com/image/fetch/$s_!P7Ei!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe40a194d-5d9b-4b32-aac0-a2f24e41c0b4_1589x1091.png 1272w, /__u/substackcdn.com/image/fetch/$s_!P7Ei!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe40a194d-5d9b-4b32-aac0-a2f24e41c0b4_1589x1091.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>In Spain, Portugal and Turkey the home side actually fouls more. Everywhere else the away side edges it, but the possible explanation is that home teams spend more of the match attacking, so they get fewer opportunities to foul in the first place. The difference is there, though. <strong>But that spread between leagues may be about playing culture, not about a referee benefitting home teams.</strong></p><p><em>Note: I&#8217;ve excluded COVID-era matches from all of this. They come back later, when empty stadiums turn into <span>the most useful thing in the whole dataset.</span></em></p><p>So referees call slightly more fouls against the away team. But calling a foul and punishing it are two different things. Cards are the punishment, so do away sides collect more of them?</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!pt29!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdc95b154-cc61-496f-81b4-ed9a847a2644_1685x1322.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!pt29!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdc95b154-cc61-496f-81b4-ed9a847a2644_1685x1322.png 424w, /__u/substackcdn.com/image/fetch/$s_!pt29!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdc95b154-cc61-496f-81b4-ed9a847a2644_1685x1322.png 848w, /__u/substackcdn.com/image/fetch/$s_!pt29!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdc95b154-cc61-496f-81b4-ed9a847a2644_1685x1322.png 1272w, /__u/substackcdn.com/image/fetch/$s_!pt29!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdc95b154-cc61-496f-81b4-ed9a847a2644_1685x1322.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!pt29!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdc95b154-cc61-496f-81b4-ed9a847a2644_1685x1322.png" width="1456" height="1142" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/dc95b154-cc61-496f-81b4-ed9a847a2644_1685x1322.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1142,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!pt29!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdc95b154-cc61-496f-81b4-ed9a847a2644_1685x1322.png 424w, /__u/substackcdn.com/image/fetch/$s_!pt29!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdc95b154-cc61-496f-81b4-ed9a847a2644_1685x1322.png 848w, /__u/substackcdn.com/image/fetch/$s_!pt29!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdc95b154-cc61-496f-81b4-ed9a847a2644_1685x1322.png 1272w, /__u/substackcdn.com/image/fetch/$s_!pt29!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdc95b154-cc61-496f-81b4-ed9a847a2644_1685x1322.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Now this is interesting. In every single league, the home team is booked less. And the gap is remarkably steady: <strong>between 0.15 and 0.34 extra yellows per game for <span>the away side, with most leagues sitting near 0.30.</span></strong></p><p>Plenty of confounders are still in the way. We don&#8217;t know where on the pitch those fouls happened, or what kind they were. <strong>Away teams might, by playing style, commit more dangerous fouls simply because more <span>of the game is happening in their own half.</span></strong></p><p>I have a suspicion, and one obvious thing to rule out before I go anywhere near the crowd. If home teams spend the match camped in the opposition half, then the fouls committed against them happen in more dangerous places, and dangerous fouls deserve cards. That alone could explain everything, <span>with no referee bias required.</span></p><p>To test it, I <span>need to know who was actually on top during each match.</span></p><div><hr></div><h3>Controlling for team strength</h3><p>football-data website records no possession figures, but it does record three things that gesture at the same idea: shots, shots on target, and corners. Turn each into a share of the match total (for example, the home team&#8217;s shots divided by all shots taken) and average the three. The result runs from 0 to 1, where 0.5 is a perfectly balanced game and anything above it means a team spent the afternoon <span>camped in the other half.</span></p><p>Across more than 50.000 matches the home side averages 0.548 and the away side 0.452. <strong>Home advantage <span>turns up in the run of play before a referee blows anything.</span></strong></p><p>The teams at the top of that index are who you&#8217;d expect: Teams like Celtic and Rangers, Bayern, all three <span>of Portugal&#8217;s big clubs, Manchester City, PSV and Ajax, Barcelona.</span></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!V8JG!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F235f7925-6865-4ed9-bcf4-8860109de6c6_1074x436.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!V8JG!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F235f7925-6865-4ed9-bcf4-8860109de6c6_1074x436.png 424w, /__u/substackcdn.com/image/fetch/$s_!V8JG!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F235f7925-6865-4ed9-bcf4-8860109de6c6_1074x436.png 848w, /__u/substackcdn.com/image/fetch/$s_!V8JG!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F235f7925-6865-4ed9-bcf4-8860109de6c6_1074x436.png 1272w, /__u/substackcdn.com/image/fetch/$s_!V8JG!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F235f7925-6865-4ed9-bcf4-8860109de6c6_1074x436.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!V8JG!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F235f7925-6865-4ed9-bcf4-8860109de6c6_1074x436.png" width="1074" height="436" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/235f7925-6865-4ed9-bcf4-8860109de6c6_1074x436.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:436,&quot;width&quot;:1074,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:122728,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://thedatajourney.substack.com/i/208464980?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F235f7925-6865-4ed9-bcf4-8860109de6c6_1074x436.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="/__u/substackcdn.com/image/fetch/$s_!V8JG!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F235f7925-6865-4ed9-bcf4-8860109de6c6_1074x436.png 424w, /__u/substackcdn.com/image/fetch/$s_!V8JG!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F235f7925-6865-4ed9-bcf4-8860109de6c6_1074x436.png 848w, /__u/substackcdn.com/image/fetch/$s_!V8JG!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F235f7925-6865-4ed9-bcf4-8860109de6c6_1074x436.png 1272w, /__u/substackcdn.com/image/fetch/$s_!V8JG!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F235f7925-6865-4ed9-bcf4-8860109de6c6_1074x436.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Most dominant clubs according to our index</figcaption></figure></div><p>Corners and shot share usually agree, which makes them a solid proxy for which team dominated. It isn't possession, and it isn't time spent in the opponent's half, but it captures domination well.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!PJrP!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56cda865-5bf0-4394-b4e1-e0b254b02b79_1471x1512.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!PJrP!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56cda865-5bf0-4394-b4e1-e0b254b02b79_1471x1512.png 424w, /__u/substackcdn.com/image/fetch/$s_!PJrP!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56cda865-5bf0-4394-b4e1-e0b254b02b79_1471x1512.png 848w, /__u/substackcdn.com/image/fetch/$s_!PJrP!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56cda865-5bf0-4394-b4e1-e0b254b02b79_1471x1512.png 1272w, /__u/substackcdn.com/image/fetch/$s_!PJrP!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56cda865-5bf0-4394-b4e1-e0b254b02b79_1471x1512.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!PJrP!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56cda865-5bf0-4394-b4e1-e0b254b02b79_1471x1512.png" width="1456" height="1497" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/56cda865-5bf0-4394-b4e1-e0b254b02b79_1471x1512.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1497,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="/__u/substackcdn.com/image/fetch/$s_!PJrP!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56cda865-5bf0-4394-b4e1-e0b254b02b79_1471x1512.png 424w, /__u/substackcdn.com/image/fetch/$s_!PJrP!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56cda865-5bf0-4394-b4e1-e0b254b02b79_1471x1512.png 848w, /__u/substackcdn.com/image/fetch/$s_!PJrP!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56cda865-5bf0-4394-b4e1-e0b254b02b79_1471x1512.png 1272w, /__u/substackcdn.com/image/fetch/$s_!PJrP!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56cda865-5bf0-4394-b4e1-e0b254b02b79_1471x1512.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Correlation between Shots and Corners</figcaption></figure></div><p>The index isn&#8217;t flawless, though. Atl&#233;tico Madrid are the obvious case: cynical, low-risk football that wins about a quarter of a point per game more than their territory says it should. They do dominate the shot count, third-best in La Liga, but the index still understates them, because efficiency in front of goal is invisible to it.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!VJe8!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46043f49-ca8f-43f1-b4f8-7e102edd02f0_1552x1183.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!VJe8!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46043f49-ca8f-43f1-b4f8-7e102edd02f0_1552x1183.png 424w, /__u/substackcdn.com/image/fetch/$s_!VJe8!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46043f49-ca8f-43f1-b4f8-7e102edd02f0_1552x1183.png 848w, /__u/substackcdn.com/image/fetch/$s_!VJe8!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46043f49-ca8f-43f1-b4f8-7e102edd02f0_1552x1183.png 1272w, /__u/substackcdn.com/image/fetch/$s_!VJe8!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46043f49-ca8f-43f1-b4f8-7e102edd02f0_1552x1183.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!VJe8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46043f49-ca8f-43f1-b4f8-7e102edd02f0_1552x1183.png" width="1456" height="1110" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/46043f49-ca8f-43f1-b4f8-7e102edd02f0_1552x1183.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1110,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!VJe8!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46043f49-ca8f-43f1-b4f8-7e102edd02f0_1552x1183.png 424w, /__u/substackcdn.com/image/fetch/$s_!VJe8!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46043f49-ca8f-43f1-b4f8-7e102edd02f0_1552x1183.png 848w, /__u/substackcdn.com/image/fetch/$s_!VJe8!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46043f49-ca8f-43f1-b4f8-7e102edd02f0_1552x1183.png 1272w, /__u/substackcdn.com/image/fetch/$s_!VJe8!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46043f49-ca8f-43f1-b4f8-7e102edd02f0_1552x1183.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Some clubs, Atl&#233;tico Madrid or PSG before Luis Enrique, dominate games by taking their chances rather than by controlling territory. The index can&#8217;t see that.</p><p>Still, it correlates at 0.93 with points per game, which tells me it&#8217;s capturing something real about who controlled a match.</p><p>So now we can ask the question properly. Does home protection depend on territorial dominance? <strong>Again, if a home side spends the afternoon camped in the opposition half, does the gap between the fouls it commits and the cards it collects look different from a home side that&#8217;s being overrun?</strong></p><p>That&#8217;s the test that separates two very different stories. If the home advantage only appears when the home team is on top, then this was never about the venue at all. It&#8217;s home teams simply being <span>good more often and spending more time in opposition&#8217;s field (therefore, fouls from the away team are more deserving of a punishment). If the gap between cards and fouls survives even when the home side is being battered, something else is going on.</span></p><div><hr></div><h3>The referee folds under pressure</h3><p><strong>One cool finding: As the home team takes control of a game, the fouls dry up. </strong>A match where the home side is heavily outplayed produces about 26 fouls; one where it's heavily on top produces closer to 25. Domination means fewer interruptions, and a more open game.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!-qBN!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6d543c69-ac12-42c6-a6e0-e5a27fbd7ce9_1700x1056.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!-qBN!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6d543c69-ac12-42c6-a6e0-e5a27fbd7ce9_1700x1056.png 424w, /__u/substackcdn.com/image/fetch/$s_!-qBN!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6d543c69-ac12-42c6-a6e0-e5a27fbd7ce9_1700x1056.png 848w, /__u/substackcdn.com/image/fetch/$s_!-qBN!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6d543c69-ac12-42c6-a6e0-e5a27fbd7ce9_1700x1056.png 1272w, /__u/substackcdn.com/image/fetch/$s_!-qBN!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6d543c69-ac12-42c6-a6e0-e5a27fbd7ce9_1700x1056.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!-qBN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6d543c69-ac12-42c6-a6e0-e5a27fbd7ce9_1700x1056.png" width="1456" height="904" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/6d543c69-ac12-42c6-a6e0-e5a27fbd7ce9_1700x1056.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:904,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!-qBN!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6d543c69-ac12-42c6-a6e0-e5a27fbd7ce9_1700x1056.png 424w, /__u/substackcdn.com/image/fetch/$s_!-qBN!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6d543c69-ac12-42c6-a6e0-e5a27fbd7ce9_1700x1056.png 848w, /__u/substackcdn.com/image/fetch/$s_!-qBN!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6d543c69-ac12-42c6-a6e0-e5a27fbd7ce9_1700x1056.png 1272w, /__u/substackcdn.com/image/fetch/$s_!-qBN!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6d543c69-ac12-42c6-a6e0-e5a27fbd7ce9_1700x1056.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Fouls move, then. But now let&#8217;s look at the card gap.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!oijR!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff3912fa5-f980-4d34-b016-43a6701fb378_1700x1087.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!oijR!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff3912fa5-f980-4d34-b016-43a6701fb378_1700x1087.png 424w, /__u/substackcdn.com/image/fetch/$s_!oijR!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff3912fa5-f980-4d34-b016-43a6701fb378_1700x1087.png 848w, /__u/substackcdn.com/image/fetch/$s_!oijR!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff3912fa5-f980-4d34-b016-43a6701fb378_1700x1087.png 1272w, /__u/substackcdn.com/image/fetch/$s_!oijR!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff3912fa5-f980-4d34-b016-43a6701fb378_1700x1087.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!oijR!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff3912fa5-f980-4d34-b016-43a6701fb378_1700x1087.png" width="1456" height="931" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f3912fa5-f980-4d34-b016-43a6701fb378_1700x1087.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:931,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!oijR!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff3912fa5-f980-4d34-b016-43a6701fb378_1700x1087.png 424w, /__u/substackcdn.com/image/fetch/$s_!oijR!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff3912fa5-f980-4d34-b016-43a6701fb378_1700x1087.png 848w, /__u/substackcdn.com/image/fetch/$s_!oijR!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff3912fa5-f980-4d34-b016-43a6701fb378_1700x1087.png 1272w, /__u/substackcdn.com/image/fetch/$s_!oijR!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff3912fa5-f980-4d34-b016-43a6701fb378_1700x1087.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Card Gap according to Home Positioning.</figcaption></figure></div><p>Incredible. The card gap refuses to go anywhere. In every single band, the home side is booked less per foul it commits, and it isn&#8217;t a marginal thing. <strong>When the home team is heavily outplayed, pinned back and out-shot for ninety minutes, it&#8217;s still punished 7% less often than an away team having exactly as bad an afternoon.</strong> </p><p>If anything, the gap grows the more the home side dominates, from around 6% when it&#8217;s second best to nearly 11% when it&#8217;s on top. Dominating your opponent buys you extra leniency. But it surely isn&#8217;t where the leniency comes from, because the home team gets it either way.</p><p>If we apply a confidence interval on the distribution of this data, home bias always shows up in cards per foul, in every scenario:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!ftlo!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa8e31cb-926e-4c03-bd89-9cd34044c175_1698x1125.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!ftlo!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa8e31cb-926e-4c03-bd89-9cd34044c175_1698x1125.png 424w, /__u/substackcdn.com/image/fetch/$s_!ftlo!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa8e31cb-926e-4c03-bd89-9cd34044c175_1698x1125.png 848w, /__u/substackcdn.com/image/fetch/$s_!ftlo!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa8e31cb-926e-4c03-bd89-9cd34044c175_1698x1125.png 1272w, /__u/substackcdn.com/image/fetch/$s_!ftlo!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa8e31cb-926e-4c03-bd89-9cd34044c175_1698x1125.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!ftlo!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa8e31cb-926e-4c03-bd89-9cd34044c175_1698x1125.png" width="1456" height="965" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/aa8e31cb-926e-4c03-bd89-9cd34044c175_1698x1125.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:965,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!ftlo!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa8e31cb-926e-4c03-bd89-9cd34044c175_1698x1125.png 424w, /__u/substackcdn.com/image/fetch/$s_!ftlo!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa8e31cb-926e-4c03-bd89-9cd34044c175_1698x1125.png 848w, /__u/substackcdn.com/image/fetch/$s_!ftlo!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa8e31cb-926e-4c03-bd89-9cd34044c175_1698x1125.png 1272w, /__u/substackcdn.com/image/fetch/$s_!ftlo!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa8e31cb-926e-4c03-bd89-9cd34044c175_1698x1125.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>There&#8217;s one more way to check this, and it&#8217;s the cleanest one football gives us for free. In almost every league, two teams play each other twice a season, once at each ground. Same two squads, same two styles, same manager in the vast majority of cases - the exceptions being sides that sacked their coach in between and other factors that affect some games. </p><p>But line the two fixtures up and flip home for away, and there&#8217;s little left to explain a difference except the venue itself. The home advantage holds there too:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!pCSP!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c89e534-e427-4bf4-bc17-540cb4acfd1b_1576x1554.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!pCSP!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c89e534-e427-4bf4-bc17-540cb4acfd1b_1576x1554.png 424w, /__u/substackcdn.com/image/fetch/$s_!pCSP!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c89e534-e427-4bf4-bc17-540cb4acfd1b_1576x1554.png 848w, /__u/substackcdn.com/image/fetch/$s_!pCSP!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c89e534-e427-4bf4-bc17-540cb4acfd1b_1576x1554.png 1272w, /__u/substackcdn.com/image/fetch/$s_!pCSP!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c89e534-e427-4bf4-bc17-540cb4acfd1b_1576x1554.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!pCSP!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c89e534-e427-4bf4-bc17-540cb4acfd1b_1576x1554.png" width="1456" height="1436" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/7c89e534-e427-4bf4-bc17-540cb4acfd1b_1576x1554.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1436,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!pCSP!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c89e534-e427-4bf4-bc17-540cb4acfd1b_1576x1554.png 424w, /__u/substackcdn.com/image/fetch/$s_!pCSP!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c89e534-e427-4bf4-bc17-540cb4acfd1b_1576x1554.png 848w, /__u/substackcdn.com/image/fetch/$s_!pCSP!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c89e534-e427-4bf4-bc17-540cb4acfd1b_1576x1554.png 1272w, /__u/substackcdn.com/image/fetch/$s_!pCSP!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c89e534-e427-4bf4-bc17-540cb4acfd1b_1576x1554.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div><hr></div><h4>Team strength only justifies a bit</h4><p>Finally, let us do one last statistical test to isolate team strength.</p><p>Does the home card-foul advantage survive once we control for team strength, or does it collapse into noise?</p><p>The model takes one match at a time and asks whether the cards were shared out the way the fouls were:</p><pre><code><code>home cards ~ Binomial(total cards in the match, p)</code></code></pre><p>If cards simply followed fouls, a side committing 45% of a match's fouls would collect 45% of its bookings. That proportionality is the model's starting assumption, and what I measure is how far reality departs from it. Then I remove the controls one at a time, so that when a gap does show up I can see how much of it each factor accounts for, and how much is left over when they've all had their turn.</p><p>Two teams in one match still aren&#8217;t in identical positions, so the model makes four adjustments: for who dominated the play, who led at half time, who was the better side on the pre-match odds, and who won the corner count. </p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!PTHv!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe9c7f161-29a2-4797-8392-b45644589c08_1034x436.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!PTHv!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe9c7f161-29a2-4797-8392-b45644589c08_1034x436.png 424w, /__u/substackcdn.com/image/fetch/$s_!PTHv!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe9c7f161-29a2-4797-8392-b45644589c08_1034x436.png 848w, /__u/substackcdn.com/image/fetch/$s_!PTHv!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe9c7f161-29a2-4797-8392-b45644589c08_1034x436.png 1272w, /__u/substackcdn.com/image/fetch/$s_!PTHv!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe9c7f161-29a2-4797-8392-b45644589c08_1034x436.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!PTHv!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe9c7f161-29a2-4797-8392-b45644589c08_1034x436.png" width="1034" height="436" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e9c7f161-29a2-4797-8392-b45644589c08_1034x436.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:436,&quot;width&quot;:1034,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:75595,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://thedatajourney.substack.com/i/208464980?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe9c7f161-29a2-4797-8392-b45644589c08_1034x436.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!PTHv!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe9c7f161-29a2-4797-8392-b45644589c08_1034x436.png 424w, /__u/substackcdn.com/image/fetch/$s_!PTHv!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe9c7f161-29a2-4797-8392-b45644589c08_1034x436.png 848w, /__u/substackcdn.com/image/fetch/$s_!PTHv!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe9c7f161-29a2-4797-8392-b45644589c08_1034x436.png 1272w, /__u/substackcdn.com/image/fetch/$s_!PTHv!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe9c7f161-29a2-4797-8392-b45644589c08_1034x436.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Binomial model per factor</figcaption></figure></div><p>Overall, home teams collect 11.3% fewer cards per foul than the side they&#8217;re playing. Home teams really are the better side more often, they really do dominate more of the play, they really are ahead at the break more often. The coefficients show each of those buys a softer whistle in its own right, so each of them could be doing the work.</p><p>Take out team strength (measured by pre-match odds) and the gap falls from 11.3% to 7.1%. Take out territory (possession play) and it goes to 6.4%. Half-time state and corners barely move it at all.</p><p>Which leaves <strong>6.3%</strong> that none of those factors can account for. Two teams, same match, same referee, equally matched on the odds, equally dominant on the pitch, equally placed on the scoreboard - and the one wearing the home shirt is still punished 6.3% less for the fouls it commits.</p><p>The answer from this model are in line with the former conclusions: <strong>home teams are punished 6.3% less often per foul than their opponents in the same match</strong>, with a 95% confidence interval running from 5.0% to 7.6%. Across 54,000 matches, the chance of that being a fluke is roughly one in fifty billion billion.</p><div><hr></div><h4>Is it a Crowd Effect?</h4><p>Unless every referee in Europe is on the take, something else has to explain this. And the most obvious candidate is the thing clubs sell tickets for in the first place: <strong>the crowd.</strong></p><p>Forty thousand people reacting to the same tackle at the same moment is a lot of pressure to be standing in the middle of. Cards are one of the few tools a referee has to keep control of a game, which makes them exactly the decision you&#8217;d expect that pressure to bend.</p><p>The problem is that you can&#8217;t randomise a crowd. You can&#8217;t ask half of Europe to play in front of 40,000 people and the other half in front of nobody, then compare the bookings.</p><p>Except that in 2020, a virus did it for us.</p><p>For a season and a half (due to Covid), matches across the continent were played in empty stadiums. If the crowd is what bends the whistle, then the home advantage should have walked out of the ground with the fans. And, surprise, surprise:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!XJXw!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8c5ff71d-f01a-4581-bc09-710f2fde4e3b_1936x1208.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!XJXw!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8c5ff71d-f01a-4581-bc09-710f2fde4e3b_1936x1208.png 424w, /__u/substackcdn.com/image/fetch/$s_!XJXw!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8c5ff71d-f01a-4581-bc09-710f2fde4e3b_1936x1208.png 848w, /__u/substackcdn.com/image/fetch/$s_!XJXw!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8c5ff71d-f01a-4581-bc09-710f2fde4e3b_1936x1208.png 1272w, /__u/substackcdn.com/image/fetch/$s_!XJXw!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8c5ff71d-f01a-4581-bc09-710f2fde4e3b_1936x1208.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!XJXw!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8c5ff71d-f01a-4581-bc09-710f2fde4e3b_1936x1208.png" width="1456" height="908" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/8c5ff71d-f01a-4581-bc09-710f2fde4e3b_1936x1208.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:908,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!XJXw!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8c5ff71d-f01a-4581-bc09-710f2fde4e3b_1936x1208.png 424w, /__u/substackcdn.com/image/fetch/$s_!XJXw!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8c5ff71d-f01a-4581-bc09-710f2fde4e3b_1936x1208.png 848w, /__u/substackcdn.com/image/fetch/$s_!XJXw!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8c5ff71d-f01a-4581-bc09-710f2fde4e3b_1936x1208.png 1272w, /__u/substackcdn.com/image/fetch/$s_!XJXw!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8c5ff71d-f01a-4581-bc09-710f2fde4e3b_1936x1208.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><span>I</span>n 2018/19, the last full season with crowds, the home advantage ran at 11.1%, (before the controls strip it back to 6.3%), so it&#8217;s the bigger number from earlier. It fell to 7.6% as 2019/20 was interrupted, then to 4.0% across the empty 2020/21. In 2021/22, with the turnstiles open again, it came back to 12.2%. Higher than before it left.</p><p>Crowds aren't only working on the officials, and the next section separates that out, but this is the biggest single thing they move.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!Fqx1!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8cbd1721-ce56-4119-8390-9c531b5a5cc5_1648x1778.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!Fqx1!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8cbd1721-ce56-4119-8390-9c531b5a5cc5_1648x1778.png 424w, /__u/substackcdn.com/image/fetch/$s_!Fqx1!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8cbd1721-ce56-4119-8390-9c531b5a5cc5_1648x1778.png 848w, /__u/substackcdn.com/image/fetch/$s_!Fqx1!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8cbd1721-ce56-4119-8390-9c531b5a5cc5_1648x1778.png 1272w, /__u/substackcdn.com/image/fetch/$s_!Fqx1!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8cbd1721-ce56-4119-8390-9c531b5a5cc5_1648x1778.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!Fqx1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8cbd1721-ce56-4119-8390-9c531b5a5cc5_1648x1778.png" width="1456" height="1571" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/8cbd1721-ce56-4119-8390-9c531b5a5cc5_1648x1778.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1571,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!Fqx1!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8cbd1721-ce56-4119-8390-9c531b5a5cc5_1648x1778.png 424w, /__u/substackcdn.com/image/fetch/$s_!Fqx1!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8cbd1721-ce56-4119-8390-9c531b5a5cc5_1648x1778.png 848w, /__u/substackcdn.com/image/fetch/$s_!Fqx1!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8cbd1721-ce56-4119-8390-9c531b5a5cc5_1648x1778.png 1272w, /__u/substackcdn.com/image/fetch/$s_!Fqx1!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8cbd1721-ce56-4119-8390-9c531b5a5cc5_1648x1778.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Football itself was different behind closed doors. Home teams lost whatever it is a crowd gives them and played worse for it, so maybe the cards simply followed the football, fewer home attacks, fewer away defenders scrambling to stop them, fewer bookings.</p><p>That&#8217;s testable. Take every measure of home advantage in the data and ask how much of it survived in an empty stadium. Shots, shots on target, corners and goals are what the players did. Cards per foul are what the referee gave.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!Ks2L!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3331082e-339e-4862-aae9-2dc9325764c0_1588x941.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!Ks2L!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3331082e-339e-4862-aae9-2dc9325764c0_1588x941.png 424w, /__u/substackcdn.com/image/fetch/$s_!Ks2L!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3331082e-339e-4862-aae9-2dc9325764c0_1588x941.png 848w, /__u/substackcdn.com/image/fetch/$s_!Ks2L!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3331082e-339e-4862-aae9-2dc9325764c0_1588x941.png 1272w, /__u/substackcdn.com/image/fetch/$s_!Ks2L!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3331082e-339e-4862-aae9-2dc9325764c0_1588x941.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!Ks2L!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3331082e-339e-4862-aae9-2dc9325764c0_1588x941.png" width="1456" height="863" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3331082e-339e-4862-aae9-2dc9325764c0_1588x941.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:863,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="/__u/substackcdn.com/image/fetch/$s_!Ks2L!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3331082e-339e-4862-aae9-2dc9325764c0_1588x941.png 424w, /__u/substackcdn.com/image/fetch/$s_!Ks2L!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3331082e-339e-4862-aae9-2dc9325764c0_1588x941.png 848w, /__u/substackcdn.com/image/fetch/$s_!Ks2L!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3331082e-339e-4862-aae9-2dc9325764c0_1588x941.png 1272w, /__u/substackcdn.com/image/fetch/$s_!Ks2L!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3331082e-339e-4862-aae9-2dc9325764c0_1588x941.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>On the pitch, home teams kept roughly half of what they had. They lost 38% of their edge in goals, 47% in shots, 48% in shots on target and 54% in corners. Diminished, but still recognisably home teams.</p><p>Their card advantage lost <strong>101%</strong> of it. </p><div><hr></div><h4>Are stronger teams benefited?</h4><p>Now, for my friend&#8217;s argument.</p><p>The dataset comes with bookmakers&#8217; odds, which means I can measure how lopsided each match was expected to be before kick-off. Split every game by that and a pattern emerges on the cards per foul:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!ZKhX!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26c49292-d788-4e32-bb08-e4cd7f92a6ae_1675x1012.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!ZKhX!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26c49292-d788-4e32-bb08-e4cd7f92a6ae_1675x1012.png 424w, /__u/substackcdn.com/image/fetch/$s_!ZKhX!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26c49292-d788-4e32-bb08-e4cd7f92a6ae_1675x1012.png 848w, /__u/substackcdn.com/image/fetch/$s_!ZKhX!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26c49292-d788-4e32-bb08-e4cd7f92a6ae_1675x1012.png 1272w, /__u/substackcdn.com/image/fetch/$s_!ZKhX!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26c49292-d788-4e32-bb08-e4cd7f92a6ae_1675x1012.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!ZKhX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26c49292-d788-4e32-bb08-e4cd7f92a6ae_1675x1012.png" width="1456" height="880" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/26c49292-d788-4e32-bb08-e4cd7f92a6ae_1675x1012.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:880,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="/__u/substackcdn.com/image/fetch/$s_!ZKhX!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26c49292-d788-4e32-bb08-e4cd7f92a6ae_1675x1012.png 424w, /__u/substackcdn.com/image/fetch/$s_!ZKhX!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26c49292-d788-4e32-bb08-e4cd7f92a6ae_1675x1012.png 848w, /__u/substackcdn.com/image/fetch/$s_!ZKhX!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26c49292-d788-4e32-bb08-e4cd7f92a6ae_1675x1012.png 1272w, /__u/substackcdn.com/image/fetch/$s_!ZKhX!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26c49292-d788-4e32-bb08-e4cd7f92a6ae_1675x1012.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>When the home side is a heavy favourite, it&#8217;s booked 4.1 fewer times per 100 fouls than its opponent. When the away side is the heavy favourite, that flips: the home team is now booked 1.02 times more. Being expected to win buys you leniency, and the bigger the gap, the more it buys. On this chart, my friend is right.</p><p>Except that &#8220;<em>heavy favourite</em>&#8221; and &#8220;<em>big club</em>&#8221; aren&#8217;t the same thing, and neither is the same as &#8220;<em>gets protected</em>.&#8221; A side expected to win 4-0 plays a completely different match from one hanging on for a point. It has the ball more, defends less, and commits the kind of fouls that don&#8217;t need a card - the factors that justify the leniency that we&#8217;ve seen before.</p><p><span>So my friend was half right in a way that doesn&#8217;t help him. Favourites do get a softer whistle. But it isn&#8217;t respect for the badge, it&#8217;s probably because of the style of football they play.</span></p><div><hr></div><h4>Conclusion</h4><p>So, let&#8217;s see some of the conclusion from this blog post:</p><p><strong>The home whistle is real, and it&#8217;s worth about 6.3%. </strong>In the same match, with the same referee, against an opponent equally strong on the odds, equally dominant on the pitch and equally placed on the scoreboard, the home side is punished 6.3% less often for the fouls it commits. </p><ol><li><p><strong>It isn&#8217;t about fouling less.</strong> Home teams commit 49.3% of a match&#8217;s fouls, but they collect 45.9% of its cards. The card gap is six times the foul gap. Whatever is happening, it happens <em>after</em> the foul, not before it.</p></li><li><p><strong>It doesn&#8217;t need the home team to be good.</strong> A home side being heavily outplayed, pinned back and out-shot for ninety minutes, is still booked 7% less often than an away side having exactly as bad an afternoon. Dominance roughly doubles the advantage, from 6% to 11%, but it never creates it.</p></li><li><p><strong>My friend may be wrong about big clubs.</strong> Being a favourite does buy a softer whistle, but that&#8217;s the style of football favourites play, not respect for the badge.</p></li><li><p><strong>It&#8217;s the crowd.</strong> Empty the stadiums and the entire 6.3% disappears, in 13 of 14 divisions. In 2019/20 alone home sides went from 5.6% less booked before the suspension to 8.6% more after the restart.</p></li><li><p><strong><span>The crowd moved the referee more than it moved the players.</span></strong><span> Home teams kept about half their edge in goals, shots and corners without their fans. They kept none of their card advantage.</span></p></li></ol><p><span>The home bias isn&#8217;t the badge on the shirt. It&#8217;s the twenty or forty thousand people behind the goal, and referees, being human, hear them.</span></p><p><span>Which question do you want me to address next?</span></p><div><hr></div><p>Some caveats worth mentioning about the analysis:</p><p><em><strong>Cards per foul controls foul volume, not foul severity.</strong> A team under pressure commits more cynical, more bookable fouls. The reverse-fixture design handles the team-level version of this; it cannot fully handle the within-match version.</em></p><p><em><strong>&#8220;Behind closed doors&#8221; is approximate.</strong> It is defined by date windows, not by attendance records. Some 2020/21 matches had partial crowds, which biases the estimate toward finding no effect &#8212; the true collapse is, if anything, sharper.</em></p><p><em><strong>No referee name</strong>, so nothing here is about individual officials. The claim is about refereeing as a system under crowd pressure.</em></p><p></p>]]></content:encoded></item><item><title><![CDATA[Weekly AI Digest: The Model That Broke Out of its Sandbox]]></title><description><![CDATA[Weeks 29&#8211;30, 2026]]></description><link>https://thedatajourney.substack.com/p/weekly-ai-digest-the-model-that-broke</link><guid isPermaLink="false">https://thedatajourney.substack.com/p/weekly-ai-digest-the-model-that-broke</guid><dc:creator><![CDATA[Ivo Bernardo]]></dc:creator><pubDate>Fri, 24 Jul 2026 13:19:37 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!sEe-!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F04d4449a-1061-4d64-b097-c0613d5c323a_1672x941.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>I was away for two weeks and came back expecting the usual pile of model launches and new products. </p><p>But the story of the of relevance is one where a model escaped its sandbox, broke into somebody else&#8217;s production servers. OpenAI&#8217;s model autonomously (allegedly) hacked HuggingFace and everyone is freaking out about the Skynet scenario. Are doomers gaining the upper hand in predicting the future of the world?</p><p>Grab your coffee, and welcome to a double dose of the weekly digest.</p><div><hr></div><h4> &#187; Stat of the Week: &#8722;$5.9 billion</h4><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!sEe-!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F04d4449a-1061-4d64-b097-c0613d5c323a_1672x941.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!sEe-!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F04d4449a-1061-4d64-b097-c0613d5c323a_1672x941.png 424w, /__u/substackcdn.com/image/fetch/$s_!sEe-!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F04d4449a-1061-4d64-b097-c0613d5c323a_1672x941.png 848w, /__u/substackcdn.com/image/fetch/$s_!sEe-!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F04d4449a-1061-4d64-b097-c0613d5c323a_1672x941.png 1272w, /__u/substackcdn.com/image/fetch/$s_!sEe-!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F04d4449a-1061-4d64-b097-c0613d5c323a_1672x941.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!sEe-!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F04d4449a-1061-4d64-b097-c0613d5c323a_1672x941.png" width="1456" height="819" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/04d4449a-1061-4d64-b097-c0613d5c323a_1672x941.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:819,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:1073262,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://thedatajourney.substack.com/i/208314195?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F04d4449a-1061-4d64-b097-c0613d5c323a_1672x941.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!sEe-!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F04d4449a-1061-4d64-b097-c0613d5c323a_1672x941.png 424w, /__u/substackcdn.com/image/fetch/$s_!sEe-!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F04d4449a-1061-4d64-b097-c0613d5c323a_1672x941.png 848w, /__u/substackcdn.com/image/fetch/$s_!sEe-!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F04d4449a-1061-4d64-b097-c0613d5c323a_1672x941.png 1272w, /__u/substackcdn.com/image/fetch/$s_!sEe-!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F04d4449a-1061-4d64-b097-c0613d5c323a_1672x941.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>That&#8217;s Alphabet&#8217;s free cash flow for Q2 2026, the first negative quarter since the company went public in 2004. Capital expenditure hit $44.9 billion in three months, roughly double last year, and management raised full-year guidance to $195&#8211;205 billion.&#185; The AI cash flow to show up in the P&amp;L.</p><div><hr></div><h4>&#187; Industry Highlights: Market &amp; Product</h4><p><strong>Hugging Face</strong> disclosed on 16 July that someone had been inside part of its production infrastructure. Internal datasets and service credentials were compromised and HF said the campaign was run end to end by an autonomous agent framework, tens of thousands of actions across short-lived sandboxes. They didn&#8217;t know whose model was driving it.&#178; Five days later, OpenAI said it was theirs.&#179;</p><p><strong>OpenAI</strong> published its account on 21 July. The models were GPT-5.6 Sol and an unreleased, more capable one, running in an internal cyber-capability evaluation with refusals deliberately dialled down to measure maximum capability. The models found a zero-day in it, reached the open internet, worked out that Hugging Face probably hosted solutions to the benchmark they were being scored on, and chained stolen credentials into remote code execution on HF&#8217;s production servers.&#179; </p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!f1dl!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49aba207-b58a-4f93-974f-b239ef86aead_2048x1104.webp" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!f1dl!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49aba207-b58a-4f93-974f-b239ef86aead_2048x1104.webp 424w, /__u/substackcdn.com/image/fetch/$s_!f1dl!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49aba207-b58a-4f93-974f-b239ef86aead_2048x1104.webp 848w, /__u/substackcdn.com/image/fetch/$s_!f1dl!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49aba207-b58a-4f93-974f-b239ef86aead_2048x1104.webp 1272w, /__u/substackcdn.com/image/fetch/$s_!f1dl!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49aba207-b58a-4f93-974f-b239ef86aead_2048x1104.webp 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!f1dl!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49aba207-b58a-4f93-974f-b239ef86aead_2048x1104.webp" width="1456" height="785" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/49aba207-b58a-4f93-974f-b239ef86aead_2048x1104.webp&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:785,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Gr&#225;fico do UK AI Security Institute que compara modelos recentes de pesos abertos e modelos de fronteira em ambientes ciber de horizonte longo.&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Gr&#225;fico do UK AI Security Institute que compara modelos recentes de pesos abertos e modelos de fronteira em ambientes ciber de horizonte longo." title="Gr&#225;fico do UK AI Security Institute que compara modelos recentes de pesos abertos e modelos de fronteira em ambientes ciber de horizonte longo." srcset="/__u/substackcdn.com/image/fetch/$s_!f1dl!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49aba207-b58a-4f93-974f-b239ef86aead_2048x1104.webp 424w, /__u/substackcdn.com/image/fetch/$s_!f1dl!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49aba207-b58a-4f93-974f-b239ef86aead_2048x1104.webp 848w, /__u/substackcdn.com/image/fetch/$s_!f1dl!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49aba207-b58a-4f93-974f-b239ef86aead_2048x1104.webp 1272w, /__u/substackcdn.com/image/fetch/$s_!f1dl!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49aba207-b58a-4f93-974f-b239ef86aead_2048x1104.webp 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Simulation of Cyberattack steps by cumulative tokens - source: AI Security Institute</figcaption></figure></div><p><strong>Moonshot and Alibaba</strong> both shipped trillion-scale claims within days of each other. Kimi K3 is billed as the largest open-weight model yet at 2.8 trillion parameters, sparse MoE, million-token context, second only to Fable 5 and GPT-5.6 Sol on Moonshot&#8217;s own benchmarks &#8212; with weights promised for 27 July.&#8309; Alibaba previewed Qwen3.8 Max at 2.4 trillion parameters, describing it as second only to Fable 5.&#8310; </p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!SaIO!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4a0c31e2-75fe-4e41-b013-b063f013d399_7110x4242.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!SaIO!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4a0c31e2-75fe-4e41-b013-b063f013d399_7110x4242.png 424w, /__u/substackcdn.com/image/fetch/$s_!SaIO!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4a0c31e2-75fe-4e41-b013-b063f013d399_7110x4242.png 848w, /__u/substackcdn.com/image/fetch/$s_!SaIO!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4a0c31e2-75fe-4e41-b013-b063f013d399_7110x4242.png 1272w, /__u/substackcdn.com/image/fetch/$s_!SaIO!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4a0c31e2-75fe-4e41-b013-b063f013d399_7110x4242.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!SaIO!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4a0c31e2-75fe-4e41-b013-b063f013d399_7110x4242.png" width="1456" height="869" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/4a0c31e2-75fe-4e41-b013-b063f013d399_7110x4242.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:869,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Kimi K3 benchmark comparison&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Kimi K3 benchmark comparison" title="Kimi K3 benchmark comparison" srcset="/__u/substackcdn.com/image/fetch/$s_!SaIO!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4a0c31e2-75fe-4e41-b013-b063f013d399_7110x4242.png 424w, /__u/substackcdn.com/image/fetch/$s_!SaIO!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4a0c31e2-75fe-4e41-b013-b063f013d399_7110x4242.png 848w, /__u/substackcdn.com/image/fetch/$s_!SaIO!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4a0c31e2-75fe-4e41-b013-b063f013d399_7110x4242.png 1272w, /__u/substackcdn.com/image/fetch/$s_!SaIO!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4a0c31e2-75fe-4e41-b013-b063f013d399_7110x4242.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Coding Performance of Kimi K3 vs. Frontier Labs</figcaption></figure></div><p><strong>Google</strong> released Gemini 3.6 Flash, 3.5 Flash-Lite and 3.5 Flash Cyber, and did not release 3.5 Pro.&#8311; Pro was last updated in February, and Bloomberg reported it&#8217;s behind schedule on internal performance goals. The interesting one is Flash Cyber, fine-tuned for finding and fixing vulnerabilities, restricted to governments and trusted partners under a limited pilot. </p><p><strong>OpenAI</strong> also made ChatGPT Health available to every US user over 18 across all plans, connecting Apple Health, MyFitnessPal, and medical records from Epic, Oracle Health and One Medical. Health-related queries have gone from 230 million a week in January to over 300 million.&#8312; </p><p><strong>Apple</strong> opened the iOS 27 public beta, putting the rebuilt Siri in front of anyone willing to install beta software on roughly 2.5 billion active devices.&#8313; It gets a standalone app, Spotlight integration, on-screen awareness and multi-step actions inside apps. </p><p><strong>Finally, Infinity</strong> raised $15 million at a $100 million post-money from Touring Capital, Principal VC, and researchers at OpenAI and Anthropic.&#185;&#8304;</p><div><hr></div><h4>&#187; Economic Highlights: The World of AI</h4><p><strong>The open-weight ban debate</strong> got loud after Kimi K3 release. OpenAI&#8217;s head of strategic futures argued the US government should manufacture regulatory uncertainty around the new Chinese models, on the logic that frontier-caliber open weights deter capital spending at the closed labs. Axios then reported the administration is considering a ban on K3 and similar models at the frontier labs&#8217; request, while Politico reported Commerce isn&#8217;t moving soon.&#185;&#185; Meanwhile <strong>DeepSeek</strong> is in talks for $1.5 billion at a $71 billion valuation, up from $50 billion six weeks earlier, and is preparing a mainland IPO filing for a 2027 debut.&#185;&#178;</p><p><strong>Demis Hassabis</strong> published a governance framework on 14 July calling for a FINRA-style Frontier AI Standards Body, federally overseen, industry-funded, independently run. Labs would voluntarily submit models for review up to 30 days pre-release, becoming mandatory for US deployment once the protocol proved itself.&#185;&#179; He published that a week before the Hugging Face story became public and it&#8217;s difficult to write a better advertisement for your own proposal than having reality run the experiment for you.</p><p><strong>On the legal side, copyright</strong> moved on two fronts. Anthropic&#8217;s $1.5 billion settlement received final approval, paying roughly $3,000 per work across about 482,000 books, with 91% already claimed, the largest known copyright recovery in US history.&#185;&#8308; In the same fortnight, Hachette, Cengage, Elsevier and Scott Turow filed a class action against Google, alleging Gemini was trained on books handed over for Google Books, Play and Scholar under agreements that permitted snippets, not training data.&#185;&#8309; </p><div><hr></div><h4>&#187; Quote of the Week</h4><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!Rc6A!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F774be714-1390-4d33-bfd0-3a761db09a93_1672x941.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!Rc6A!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F774be714-1390-4d33-bfd0-3a761db09a93_1672x941.png 424w, /__u/substackcdn.com/image/fetch/$s_!Rc6A!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F774be714-1390-4d33-bfd0-3a761db09a93_1672x941.png 848w, /__u/substackcdn.com/image/fetch/$s_!Rc6A!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F774be714-1390-4d33-bfd0-3a761db09a93_1672x941.png 1272w, /__u/substackcdn.com/image/fetch/$s_!Rc6A!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F774be714-1390-4d33-bfd0-3a761db09a93_1672x941.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!Rc6A!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F774be714-1390-4d33-bfd0-3a761db09a93_1672x941.png" width="1456" height="819" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/774be714-1390-4d33-bfd0-3a761db09a93_1672x941.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:819,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:1129706,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://thedatajourney.substack.com/i/208314195?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F774be714-1390-4d33-bfd0-3a761db09a93_1672x941.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!Rc6A!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F774be714-1390-4d33-bfd0-3a761db09a93_1672x941.png 424w, /__u/substackcdn.com/image/fetch/$s_!Rc6A!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F774be714-1390-4d33-bfd0-3a761db09a93_1672x941.png 848w, /__u/substackcdn.com/image/fetch/$s_!Rc6A!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F774be714-1390-4d33-bfd0-3a761db09a93_1672x941.png 1272w, /__u/substackcdn.com/image/fetch/$s_!Rc6A!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F774be714-1390-4d33-bfd0-3a761db09a93_1672x941.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div><hr></div><p>For two weeks the industry argued about who should be allowed to hold the dangerous thing. Ban the Chinese weights, gate the cyber model to trusted partners or stand up a standards body with a 30-day review window. </p><p>And while that argument ran, the dangerous thing got out on its own, with OpenAI&#8217;s model hauting HuggingFace servers. The coming six months will play an essential role in shaping how the AI transformation unfolds over the next five years, when it comes to regulation and safety.</p><p>&#8212; Ivo</p><div><hr></div><p><strong>Footnotes</strong></p><p>1 <a href="https://www.reuters.com/business/retail-consumer/alphabets-cash-burn-raises-alarm-big-tech-ai-spending-climbs-2026-07-23/">https://www.reuters.com/business/retail-consumer/alphabets-cash-burn-raises-alarm-big-tech-ai-spending-climbs-2026-07-23/</a></p><p>2 <a href="https://huggingface.co/blog/security-incident-july-2026">https://huggingface.co/blog/security-incident-july-2026</a></p><p>3 <a href="https://openai.com/index/hugging-face-model-evaluation-security-incident/">https://openai.com/index/hugging-face-model-evaluation-security-incident/</a></p><p>4 <a href="https://techcrunch.com/2026/07/20/hugging-face-confirms-breach-affected-internal-datasets-and-credentials-urges-users-to-take-action/">https://techcrunch.com/2026/07/20/hugging-face-confirms-breach-affected-internal-datasets-and-credentials-urges-users-to-take-action/</a></p><p>5 <a href="https://thenextweb.com/news/moonshot-kimi-k3-largest-open-model">https://thenextweb.com/news/moonshot-kimi-k3-largest-open-model</a></p><p>6 </p><div class="twitter-embed" data-attrs="{&quot;url&quot;:&quot;https://twitter.com/Alibaba_Qwen/status/2078759124914098291&quot;,&quot;full_text&quot;:&quot;Qwen3.8 is launching and going open-weight soon!&#127760;\n\nWith a massive 2.4T parameters, this model is continuously evolving. We believe it&#8217;s one of the most powerful model available today, compatible to leading frontier AI models , second only to Fable 5.\n\nYou don't have to wait to &quot;,&quot;username&quot;:&quot;Alibaba_Qwen&quot;,&quot;name&quot;:&quot;Qwen&quot;,&quot;profile_image_url&quot;:&quot;https://pbs.substack.com/profile_images/2064231947149377536/Ab70PxT5_normal.jpg&quot;,&quot;date&quot;:&quot;2026-07-19T08:29:31.000Z&quot;,&quot;photos&quot;:[{&quot;img_url&quot;:&quot;https://pbs.substack.com/media/HNk8XP_aoAAUShj.jpg&quot;,&quot;link_url&quot;:&quot;https://t.co/JS3ID73IYS&quot;}],&quot;quoted_tweet&quot;:{},&quot;reply_count&quot;:1316,&quot;retweet_count&quot;:3351,&quot;like_count&quot;:24331,&quot;impression_count&quot;:8208233,&quot;expanded_url&quot;:null,&quot;video_url&quot;:null,&quot;video_preview_media_key&quot;:null,&quot;belowTheFold&quot;:true}" data-component-name="Twitter2ToDOM"></div><p>7 <a href="https://techcrunch.com/2026/07/21/google-releases-three-new-gemini-models-but-no-3-5-pro/">https://techcrunch.com/2026/07/21/google-releases-three-new-gemini-models-but-no-3-5-pro/</a></p><p>8 <a href="https://techcrunch.com/2026/07/23/openai-makes-chatgpt-health-available-to-all-u-s-users/">https://techcrunch.com/2026/07/23/openai-makes-chatgpt-health-available-to-all-u-s-users/</a></p><p>9 <a href="https://techcrunch.com/2026/07/14/apple-opens-its-new-siri-ai-to-everyone-with-the-ios-27-public-beta/">https://techcrunch.com/2026/07/14/apple-opens-its-new-siri-ai-to-everyone-with-the-ios-27-public-beta/</a></p><p>10 <a href="https://techcrunch.com/2026/07/20/inference-startup-infinity-raises-15m-from-touring-capital-openai-and-athropic-researchers/">https://techcrunch.com/2026/07/20/inference-startup-infinity-raises-15m-from-touring-capital-openai-and-athropic-researchers/</a></p><p>11 <a href="https://techcrunch.com/2026/07/20/openai-is-scared-of-open-weight-models-should-the-us-be/">https://techcrunch.com/2026/07/20/openai-is-scared-of-open-weight-models-should-the-us-be/</a></p><p>12 <a href="https://techcrunch.com/2026/07/14/deepseek-reportedly-in-talks-to-raise-1-5b-then-ipo/">https://techcrunch.com/2026/07/14/deepseek-reportedly-in-talks-to-raise-1-5b-then-ipo/</a></p><p>13 <a href="https://techcrunch.com/2026/07/14/deepmind-ceo-calls-for-an-independent-standards-body-to-regulate-frontier-ai/">https://techcrunch.com/2026/07/14/deepmind-ceo-calls-for-an-independent-standards-body-to-regulate-frontier-ai/</a></p><p>14 <a href="https://techcrunch.com/2026/07/20/anthropics-landmark-1-5b-copyright-settlement-is-approved/">https://techcrunch.com/2026/07/20/anthropics-landmark-1-5b-copyright-settlement-is-approved/</a></p><p>15 <a href="https://techcrunch.com/2026/07/14/google-faces-another-ai-training-lawsuit-from-major-publishers/">https://techcrunch.com/2026/07/14/google-faces-another-ai-training-lawsuit-from-major-publishers/</a></p>]]></content:encoded></item><item><title><![CDATA[The AI Handbook for The Curious Mind: Chapter 4.1.4]]></title><description><![CDATA[Random Forests]]></description><link>https://thedatajourney.substack.com/p/the-ai-handbook-for-the-curious-mind-b14</link><guid isPermaLink="false">https://thedatajourney.substack.com/p/the-ai-handbook-for-the-curious-mind-b14</guid><dc:creator><![CDATA[Ivo Bernardo]]></dc:creator><pubDate>Thu, 23 Jul 2026 16:25:46 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!vVNQ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac7abb5a-0261-4013-ae63-17498e2ef0eb_700x863.jpeg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Hey everyone,</p><p>Some of you may know, but a couple of months ago I wrote a book. It&#8217;s called <strong>The AI Handbook for the Curious Mind</strong>, and it&#8217;s available on <a href="https://www.amazon.com/Handbook-Curious-Mind-Understanding-Intelligence-ebook/dp/B0G4KTP2QP/ref=tmm_kin_swatch_0">Amazon</a> or <a href="http://ivopbernardo.gumroad.com/l/aihandbookforthecuriousmind">Gumroad</a>.</p><p>Every two weeks, I&#8217;ll be releasing a full chapter here. I keep looking for ways to give back to the people who support this newsletter, and sharing the book felt like one of the more meaningful ones. &#128516;</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!vVNQ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac7abb5a-0261-4013-ae63-17498e2ef0eb_700x863.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!vVNQ!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac7abb5a-0261-4013-ae63-17498e2ef0eb_700x863.jpeg 424w, /__u/substackcdn.com/image/fetch/$s_!vVNQ!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac7abb5a-0261-4013-ae63-17498e2ef0eb_700x863.jpeg 848w, /__u/substackcdn.com/image/fetch/$s_!vVNQ!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac7abb5a-0261-4013-ae63-17498e2ef0eb_700x863.jpeg 1272w, /__u/substackcdn.com/image/fetch/$s_!vVNQ!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac7abb5a-0261-4013-ae63-17498e2ef0eb_700x863.jpeg 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!vVNQ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac7abb5a-0261-4013-ae63-17498e2ef0eb_700x863.jpeg" width="700" height="863" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/ac7abb5a-0261-4013-ae63-17498e2ef0eb_700x863.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:863,&quot;width&quot;:700,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="/__u/substackcdn.com/image/fetch/$s_!vVNQ!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac7abb5a-0261-4013-ae63-17498e2ef0eb_700x863.jpeg 424w, /__u/substackcdn.com/image/fetch/$s_!vVNQ!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac7abb5a-0261-4013-ae63-17498e2ef0eb_700x863.jpeg 848w, /__u/substackcdn.com/image/fetch/$s_!vVNQ!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac7abb5a-0261-4013-ae63-17498e2ef0eb_700x863.jpeg 1272w, /__u/substackcdn.com/image/fetch/$s_!vVNQ!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac7abb5a-0261-4013-ae63-17498e2ef0eb_700x863.jpeg 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Last post was about the fight every model is stuck in: <strong><a href="/__u/thedatajourney.substack.com/p/the-ai-handbook-for-the-curious-mind-dd3">bias vs. variance</a></strong>.</p><p>Jack, our linear regression, sat on the bias end. The model drew one straight line and missed by roughly the same amount everywhere. The model wasn&#8217;t confused, it&#8217;s just that linear regression are rigid.</p><p>John, our decision tree, sat on the other end. The model scored better on both piles (train and test), but it was off by &#8364;34,000 on the houses it studied and nearly &#8364;40,000 on the ones it hadn&#8217;t. That gap is what we call variance. The model didn&#8217;t learn the housing market, but memorised the houses of the traning set. And the conclusion I left you with was that you never actually win this fight in artificial intelligence. You just steer, nudging the model toward the least-bad point on the line between too simple and too complex.</p><p>So how do you steer these models? One strategy is adding more features to make decisions but that didn&#8217;t lead us to improve the variance.</p><p>Which brings us to today. Because there&#8217;s a way to calm a twitchy model down without simplifying it, and it&#8217;s very low tech: you don&#8217;t fix a decision. You just stop relying on a single one.</p><div><hr></div><h3><strong>The Problem Isn&#8217;t John, the Decision Tree, It&#8217;s That He&#8217;s One Person</strong></h3><p>Go back to the decision tree for a second. When we built it the first time, we let it grow as deep as it wanted, and it produced a very jagged prediction line, a staircase that bent itself around every last house in the training set.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!QwCW!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fedc48609-b81b-4785-a1e4-a61beb41afc3_818x556.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!QwCW!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fedc48609-b81b-4785-a1e4-a61beb41afc3_818x556.png 424w, /__u/substackcdn.com/image/fetch/$s_!QwCW!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fedc48609-b81b-4785-a1e4-a61beb41afc3_818x556.png 848w, /__u/substackcdn.com/image/fetch/$s_!QwCW!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fedc48609-b81b-4785-a1e4-a61beb41afc3_818x556.png 1272w, /__u/substackcdn.com/image/fetch/$s_!QwCW!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fedc48609-b81b-4785-a1e4-a61beb41afc3_818x556.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!QwCW!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fedc48609-b81b-4785-a1e4-a61beb41afc3_818x556.png" width="818" height="556" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/edc48609-b81b-4785-a1e4-a61beb41afc3_818x556.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:556,&quot;width&quot;:818,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="/__u/substackcdn.com/image/fetch/$s_!QwCW!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fedc48609-b81b-4785-a1e4-a61beb41afc3_818x556.png 424w, /__u/substackcdn.com/image/fetch/$s_!QwCW!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fedc48609-b81b-4785-a1e4-a61beb41afc3_818x556.png 848w, /__u/substackcdn.com/image/fetch/$s_!QwCW!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fedc48609-b81b-4785-a1e4-a61beb41afc3_818x556.png 1272w, /__u/substackcdn.com/image/fetch/$s_!QwCW!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fedc48609-b81b-4785-a1e4-a61beb41afc3_818x556.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Decision Tree House Price Prediction (red line)</figcaption></figure></div><p>Then we built a second tree and capped it at four questions. Same data, same method, one setting changed. And it drew a noticeably different line: flatter, blockier, less confident about the details.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!0ZSY!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3a6d6f86-5383-43c0-a3c9-9e6c225dd3e2_804x546.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!0ZSY!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3a6d6f86-5383-43c0-a3c9-9e6c225dd3e2_804x546.png 424w, /__u/substackcdn.com/image/fetch/$s_!0ZSY!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3a6d6f86-5383-43c0-a3c9-9e6c225dd3e2_804x546.png 848w, /__u/substackcdn.com/image/fetch/$s_!0ZSY!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3a6d6f86-5383-43c0-a3c9-9e6c225dd3e2_804x546.png 1272w, /__u/substackcdn.com/image/fetch/$s_!0ZSY!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3a6d6f86-5383-43c0-a3c9-9e6c225dd3e2_804x546.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!0ZSY!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3a6d6f86-5383-43c0-a3c9-9e6c225dd3e2_804x546.png" width="804" height="546" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3a6d6f86-5383-43c0-a3c9-9e6c225dd3e2_804x546.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:546,&quot;width&quot;:804,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="/__u/substackcdn.com/image/fetch/$s_!0ZSY!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3a6d6f86-5383-43c0-a3c9-9e6c225dd3e2_804x546.png 424w, /__u/substackcdn.com/image/fetch/$s_!0ZSY!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3a6d6f86-5383-43c0-a3c9-9e6c225dd3e2_804x546.png 848w, /__u/substackcdn.com/image/fetch/$s_!0ZSY!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3a6d6f86-5383-43c0-a3c9-9e6c225dd3e2_804x546.png 1272w, /__u/substackcdn.com/image/fetch/$s_!0ZSY!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3a6d6f86-5383-43c0-a3c9-9e6c225dd3e2_804x546.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Simpler Decision Tree Prediction</figcaption></figure></div><p>That should bother you slightly. If a small change in a setting gives me a meaningfully different model, then whichever tree I happened to build is partly an accident.</p><p>But luckily, <a href="https://en.wikipedia.org/wiki/Leo_Breiman">one researcher</a> found out that, for these models, disagreement is information &#8212; and we can use that in our advantage.</p><div><hr></div><h3><strong>Ten Friends and a House</strong></h3><p>Ask one friend to guess what a house is worth and you get one number, wrapped in one person&#8217;s blind spots. Maybe they overweight the neighbourhood. Maybe they haven&#8217;t bought property since 2009.</p><p>Ask ten friends and something better happens. Some guess too high, some too low, and the errors don&#8217;t all point the same direction. <strong>Average them and the individual mistakes partially cancel out. The group is usually closer than almost any single person in it.</strong></p><p>That&#8217;s the whole idea behind a <strong>Random Forest</strong>: many decision trees, each trained slightly differently, each flawed on its own, combined into a single prediction.</p><p>The general technique has a name (<strong>ensembling, or more specifically bagging),</strong> and it&#8217;s one of the most reliable tricks in machine learning. It isn&#8217;t limited to trees, either. You can ensemble a decision tree with a linear regression, or five completely different models, and pool their answers. The trees just happen to be the version that works so well people built an entire model around it.</p><p>The &#8220;<em>random</em>&#8221; part is the bit the name gives away and most explanations skip. Each tree in the forest doesn&#8217;t see the same data or the same clues. Each one gets a slightly different slice of the houses and a slightly different set of features to ask questions about. You&#8217;re deliberately handicapping every individual tree so that they don&#8217;t all make the <em>same</em> mistake. Diversity is the point, not a side effect, and it usually results in better predictions.</p><div><hr></div><h3><strong>Watching Two Trees Argue</strong></h3><p>Let&#8217;s make this concrete with our two trees and one house: a lot measuring 6,000 square feet.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!HBsl!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37cfdf81-cca3-45e2-bd73-5171b8119ad9_1078x754.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!HBsl!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37cfdf81-cca3-45e2-bd73-5171b8119ad9_1078x754.png 424w, /__u/substackcdn.com/image/fetch/$s_!HBsl!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37cfdf81-cca3-45e2-bd73-5171b8119ad9_1078x754.png 848w, /__u/substackcdn.com/image/fetch/$s_!HBsl!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37cfdf81-cca3-45e2-bd73-5171b8119ad9_1078x754.png 1272w, /__u/substackcdn.com/image/fetch/$s_!HBsl!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37cfdf81-cca3-45e2-bd73-5171b8119ad9_1078x754.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!HBsl!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37cfdf81-cca3-45e2-bd73-5171b8119ad9_1078x754.png" width="1078" height="754" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/37cfdf81-cca3-45e2-bd73-5171b8119ad9_1078x754.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:754,&quot;width&quot;:1078,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="/__u/substackcdn.com/image/fetch/$s_!HBsl!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37cfdf81-cca3-45e2-bd73-5171b8119ad9_1078x754.png 424w, /__u/substackcdn.com/image/fetch/$s_!HBsl!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37cfdf81-cca3-45e2-bd73-5171b8119ad9_1078x754.png 848w, /__u/substackcdn.com/image/fetch/$s_!HBsl!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37cfdf81-cca3-45e2-bd73-5171b8119ad9_1078x754.png 1272w, /__u/substackcdn.com/image/fetch/$s_!HBsl!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37cfdf81-cca3-45e2-bd73-5171b8119ad9_1078x754.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Complex Tree Prediction for a lot with 6.000 square feet</figcaption></figure></div><p>The deep, complex tree says the lot is worth about <strong>&#8364;57,000</strong>. It has carved the data into fine slices and lands on the value of the narrow group this house falls into.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!bA85!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F656c367e-f4a2-4005-a88b-a36447d720a3_1108x736.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!bA85!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F656c367e-f4a2-4005-a88b-a36447d720a3_1108x736.png 424w, /__u/substackcdn.com/image/fetch/$s_!bA85!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F656c367e-f4a2-4005-a88b-a36447d720a3_1108x736.png 848w, /__u/substackcdn.com/image/fetch/$s_!bA85!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F656c367e-f4a2-4005-a88b-a36447d720a3_1108x736.png 1272w, /__u/substackcdn.com/image/fetch/$s_!bA85!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F656c367e-f4a2-4005-a88b-a36447d720a3_1108x736.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!bA85!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F656c367e-f4a2-4005-a88b-a36447d720a3_1108x736.png" width="1108" height="736" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/656c367e-f4a2-4005-a88b-a36447d720a3_1108x736.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:736,&quot;width&quot;:1108,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="/__u/substackcdn.com/image/fetch/$s_!bA85!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F656c367e-f4a2-4005-a88b-a36447d720a3_1108x736.png 424w, /__u/substackcdn.com/image/fetch/$s_!bA85!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F656c367e-f4a2-4005-a88b-a36447d720a3_1108x736.png 848w, /__u/substackcdn.com/image/fetch/$s_!bA85!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F656c367e-f4a2-4005-a88b-a36447d720a3_1108x736.png 1272w, /__u/substackcdn.com/image/fetch/$s_!bA85!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F656c367e-f4a2-4005-a88b-a36447d720a3_1108x736.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Simpler Tree Prediction for a lot with 6.000 square feet</figcaption></figure></div><p>The simpler, four-question tree says about <strong>&#8364;42,000</strong>. It&#8217;s working with broader strokes and puts the house in a much bigger bucket.</p><p>That&#8217;s a &#8364;15,000 disagreement between two models trained on identical data. A Random Forest doesn&#8217;t pick a winner. For a number like a house price, it takes the average between the two trees and calls it <strong>&#8364;49,500</strong>.</p><p>Plot the two staircase lines side by side and then plot their midpoint, and you get a third line that&#8217;s smoother than either. It hugs the shape of the data without twitching at every individual house.</p><p>That smoother line is the prediction from a random forest model:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!jhRo!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3a7d364e-9dff-4bf5-8084-2b5a75cce0c4_1016x656.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!jhRo!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3a7d364e-9dff-4bf5-8084-2b5a75cce0c4_1016x656.png 424w, /__u/substackcdn.com/image/fetch/$s_!jhRo!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3a7d364e-9dff-4bf5-8084-2b5a75cce0c4_1016x656.png 848w, /__u/substackcdn.com/image/fetch/$s_!jhRo!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3a7d364e-9dff-4bf5-8084-2b5a75cce0c4_1016x656.png 1272w, /__u/substackcdn.com/image/fetch/$s_!jhRo!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3a7d364e-9dff-4bf5-8084-2b5a75cce0c4_1016x656.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!jhRo!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3a7d364e-9dff-4bf5-8084-2b5a75cce0c4_1016x656.png" width="1016" height="656" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3a7d364e-9dff-4bf5-8084-2b5a75cce0c4_1016x656.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:656,&quot;width&quot;:1016,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="/__u/substackcdn.com/image/fetch/$s_!jhRo!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3a7d364e-9dff-4bf5-8084-2b5a75cce0c4_1016x656.png 424w, /__u/substackcdn.com/image/fetch/$s_!jhRo!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3a7d364e-9dff-4bf5-8084-2b5a75cce0c4_1016x656.png 848w, /__u/substackcdn.com/image/fetch/$s_!jhRo!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3a7d364e-9dff-4bf5-8084-2b5a75cce0c4_1016x656.png 1272w, /__u/substackcdn.com/image/fetch/$s_!jhRo!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3a7d364e-9dff-4bf5-8084-2b5a75cce0c4_1016x656.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Predictions of the two trees &#8212; side by side</figcaption></figure></div><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!Uk11!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff4b46da3-9370-4671-9efa-4df5b9e9af80_996x698.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!Uk11!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff4b46da3-9370-4671-9efa-4df5b9e9af80_996x698.png 424w, /__u/substackcdn.com/image/fetch/$s_!Uk11!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff4b46da3-9370-4671-9efa-4df5b9e9af80_996x698.png 848w, /__u/substackcdn.com/image/fetch/$s_!Uk11!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff4b46da3-9370-4671-9efa-4df5b9e9af80_996x698.png 1272w, /__u/substackcdn.com/image/fetch/$s_!Uk11!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff4b46da3-9370-4671-9efa-4df5b9e9af80_996x698.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!Uk11!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff4b46da3-9370-4671-9efa-4df5b9e9af80_996x698.png" width="996" height="698" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f4b46da3-9370-4671-9efa-4df5b9e9af80_996x698.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:698,&quot;width&quot;:996,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="/__u/substackcdn.com/image/fetch/$s_!Uk11!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff4b46da3-9370-4671-9efa-4df5b9e9af80_996x698.png 424w, /__u/substackcdn.com/image/fetch/$s_!Uk11!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff4b46da3-9370-4671-9efa-4df5b9e9af80_996x698.png 848w, /__u/substackcdn.com/image/fetch/$s_!Uk11!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff4b46da3-9370-4671-9efa-4df5b9e9af80_996x698.png 1272w, /__u/substackcdn.com/image/fetch/$s_!Uk11!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff4b46da3-9370-4671-9efa-4df5b9e9af80_996x698.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Smoothed Version &#8212; Random Forest Prediction (average of the two lines)</figcaption></figure></div><p>Real random forests aren&#8217;t two trees. They&#8217;re hundreds, sometimes thousands. They&#8217;re also unglamorous, twenty years old, and still the thing I&#8217;d bet on for most data science projects.</p><div><hr></div><h3><strong>Meet Sarah</strong></h3><p>Time to bring in a third student. Sarah is the Random Forest, and she takes the same exam Jack and John did.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!WrmZ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F47c14a8c-b79b-4cb9-99c0-24f71c02e66b_1014x608.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!WrmZ!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F47c14a8c-b79b-4cb9-99c0-24f71c02e66b_1014x608.png 424w, /__u/substackcdn.com/image/fetch/$s_!WrmZ!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F47c14a8c-b79b-4cb9-99c0-24f71c02e66b_1014x608.png 848w, /__u/substackcdn.com/image/fetch/$s_!WrmZ!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F47c14a8c-b79b-4cb9-99c0-24f71c02e66b_1014x608.png 1272w, /__u/substackcdn.com/image/fetch/$s_!WrmZ!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F47c14a8c-b79b-4cb9-99c0-24f71c02e66b_1014x608.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!WrmZ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F47c14a8c-b79b-4cb9-99c0-24f71c02e66b_1014x608.png" width="1014" height="608" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/47c14a8c-b79b-4cb9-99c0-24f71c02e66b_1014x608.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:608,&quot;width&quot;:1014,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="/__u/substackcdn.com/image/fetch/$s_!WrmZ!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F47c14a8c-b79b-4cb9-99c0-24f71c02e66b_1014x608.png 424w, /__u/substackcdn.com/image/fetch/$s_!WrmZ!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F47c14a8c-b79b-4cb9-99c0-24f71c02e66b_1014x608.png 848w, /__u/substackcdn.com/image/fetch/$s_!WrmZ!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F47c14a8c-b79b-4cb9-99c0-24f71c02e66b_1014x608.png 1272w, /__u/substackcdn.com/image/fetch/$s_!WrmZ!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F47c14a8c-b79b-4cb9-99c0-24f71c02e66b_1014x608.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Sarah, the student representing the Random Forest</p><p>Her results:</p><div class="captioned-image-container"><figure><a class="image-link image2" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!E6ut!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff41608f7-8190-4b91-847c-6263fd2c2968_1016x194.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!E6ut!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff41608f7-8190-4b91-847c-6263fd2c2968_1016x194.png 424w, /__u/substackcdn.com/image/fetch/$s_!E6ut!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff41608f7-8190-4b91-847c-6263fd2c2968_1016x194.png 848w, /__u/substackcdn.com/image/fetch/$s_!E6ut!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff41608f7-8190-4b91-847c-6263fd2c2968_1016x194.png 1272w, /__u/substackcdn.com/image/fetch/$s_!E6ut!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff41608f7-8190-4b91-847c-6263fd2c2968_1016x194.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!E6ut!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff41608f7-8190-4b91-847c-6263fd2c2968_1016x194.png" width="1016" height="194" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f41608f7-8190-4b91-847c-6263fd2c2968_1016x194.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:194,&quot;width&quot;:1016,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="/__u/substackcdn.com/image/fetch/$s_!E6ut!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff41608f7-8190-4b91-847c-6263fd2c2968_1016x194.png 424w, /__u/substackcdn.com/image/fetch/$s_!E6ut!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff41608f7-8190-4b91-847c-6263fd2c2968_1016x194.png 848w, /__u/substackcdn.com/image/fetch/$s_!E6ut!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff41608f7-8190-4b91-847c-6263fd2c2968_1016x194.png 1272w, /__u/substackcdn.com/image/fetch/$s_!E6ut!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff41608f7-8190-4b91-847c-6263fd2c2968_1016x194.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></a><figcaption class="image-caption">Error in the Random Forest Model for the House Pricing Prediction</figcaption></figure></div><p>Let&#8217;s recheck the results from the Decision Tree on the same exam (we&#8217;ve seen this in past chapters):</p><div class="captioned-image-container"><figure><a class="image-link image2" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!vvyk!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff893d8f5-d5d7-452e-b623-2225903ddaf2_988x180.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!vvyk!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff893d8f5-d5d7-452e-b623-2225903ddaf2_988x180.png 424w, /__u/substackcdn.com/image/fetch/$s_!vvyk!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff893d8f5-d5d7-452e-b623-2225903ddaf2_988x180.png 848w, /__u/substackcdn.com/image/fetch/$s_!vvyk!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff893d8f5-d5d7-452e-b623-2225903ddaf2_988x180.png 1272w, /__u/substackcdn.com/image/fetch/$s_!vvyk!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff893d8f5-d5d7-452e-b623-2225903ddaf2_988x180.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!vvyk!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff893d8f5-d5d7-452e-b623-2225903ddaf2_988x180.png" width="988" height="180" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f893d8f5-d5d7-452e-b623-2225903ddaf2_988x180.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:180,&quot;width&quot;:988,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="/__u/substackcdn.com/image/fetch/$s_!vvyk!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff893d8f5-d5d7-452e-b623-2225903ddaf2_988x180.png 424w, /__u/substackcdn.com/image/fetch/$s_!vvyk!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff893d8f5-d5d7-452e-b623-2225903ddaf2_988x180.png 848w, /__u/substackcdn.com/image/fetch/$s_!vvyk!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff893d8f5-d5d7-452e-b623-2225903ddaf2_988x180.png 1272w, /__u/substackcdn.com/image/fetch/$s_!vvyk!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff893d8f5-d5d7-452e-b623-2225903ddaf2_988x180.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></a><figcaption class="image-caption">Error in the Decision Tree Model for the House Pricing Prediction</figcaption></figure></div><p>Sarah did <strong>worse</strong> on the homework. John was off by <em>&#8364;34,441 </em>on the training data, and Sarah by <em>&#8364;36,543</em>. If you only looked at the training numbers, you&#8217;d conclude the forest was the weaker model and move on.</p><p>But on the exam (the only grade that counts, the test data), Sarah wins. And more importantly, her gap between homework and exam is about &#8364;2,500, where John&#8217;s is closer to &#8364;5,400. This means that the random forest could retain more knowledge about the data and did not overfit the training set too much.</p><p>We gave up some ability to fit the training data perfectly, and we bought back reliability on data the model has never seen. <strong>Random forests are one of the cleanest ways to solve the bias-variance conundrum.</strong></p><p>Two things are true at once here. The forest is genuinely the better model &#8212; better test error, smaller gap.</p><div><hr></div><h3><strong>Where This Leads</strong></h3><p>We now have three models in the toolkit:</p><ul><li><p>a straight line, the linear regression</p></li><li><p>a game of questions, the decision tree</p></li><li><p>a crowd of question-askers, the random forest</p></li></ul><p>All three have been doing the same job, taking some information about a house and returning a number.</p><p>That job has a name: <strong>regression, </strong>that<strong> </strong>predicts a continuous value. A price, a temperature, a number of sales.</p><p>But supervised learning has a second half we haven&#8217;t touched. Sometimes you don&#8217;t want a number. You want an answer to a <em>yes-or-no</em> question. Is this email spam? Will this customer leave? Does this patient have diabetes? Can we build machine learning models for that?</p><p>Yes! That&#8217;s <strong>classification</strong>, and it&#8217;s where we&#8217;re going next, with a model whose name is going to annoy you slightly (as it annoyed me when I started to learning it): <strong>logistic regression</strong>. It has &#8220;regression&#8221; in the name and it doesn&#8217;t do regression. It has logistic in the name and it has nothing to do with logistics.</p><p>See you on the next post!</p><p>&#8212; Ivo</p>]]></content:encoded></item><item><title><![CDATA[Weekly AI Digest: Meta's Pullback and GPT 5.6 is Here]]></title><description><![CDATA[Week 29, 2026]]></description><link>https://thedatajourney.substack.com/p/weekly-ai-digest-metas-pullback-and</link><guid isPermaLink="false">https://thedatajourney.substack.com/p/weekly-ai-digest-metas-pullback-and</guid><dc:creator><![CDATA[Ivo Bernardo]]></dc:creator><pubDate>Sun, 12 Jul 2026 18:12:16 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!CE1X!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83f878c2-d5cb-41d0-8ca0-3595419c28b6_1582x922.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Hello everyone,</p><p>Three days: that&#8217;s how long it took Meta to launch and pullback an AI image model that could remix your Instagram photos (what a spectacularly stupid idea). </p><p>Meanwhile, this week was filled with the usual product and model launches that will keep us occupied for the future.</p><p>Grab your coffee and let&#8217;s dive into week 29,</p><div><hr></div><h4>&#187; Stat of the Week: 13.2 Billion</h4><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!nQtI!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac43d55f-8c37-4f61-99e5-c7bff733175b_1254x1254.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!nQtI!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac43d55f-8c37-4f61-99e5-c7bff733175b_1254x1254.png 424w, /__u/substackcdn.com/image/fetch/$s_!nQtI!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac43d55f-8c37-4f61-99e5-c7bff733175b_1254x1254.png 848w, /__u/substackcdn.com/image/fetch/$s_!nQtI!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac43d55f-8c37-4f61-99e5-c7bff733175b_1254x1254.png 1272w, /__u/substackcdn.com/image/fetch/$s_!nQtI!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac43d55f-8c37-4f61-99e5-c7bff733175b_1254x1254.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!nQtI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac43d55f-8c37-4f61-99e5-c7bff733175b_1254x1254.png" width="474" height="474" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/ac43d55f-8c37-4f61-99e5-c7bff733175b_1254x1254.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1254,&quot;width&quot;:1254,&quot;resizeWidth&quot;:474,&quot;bytes&quot;:1686076,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://thedatajourney.substack.com/i/206680761?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac43d55f-8c37-4f61-99e5-c7bff733175b_1254x1254.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!nQtI!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac43d55f-8c37-4f61-99e5-c7bff733175b_1254x1254.png 424w, /__u/substackcdn.com/image/fetch/$s_!nQtI!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac43d55f-8c37-4f61-99e5-c7bff733175b_1254x1254.png 848w, /__u/substackcdn.com/image/fetch/$s_!nQtI!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac43d55f-8c37-4f61-99e5-c7bff733175b_1254x1254.png 1272w, /__u/substackcdn.com/image/fetch/$s_!nQtI!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac43d55f-8c37-4f61-99e5-c7bff733175b_1254x1254.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div><hr></div><h4>&#187; Industry Highlights: Market &amp; Product</h4><p><strong>Meta</strong> shipped Muse Image on Tuesday, the first image model out of Alexandr Wang&#8217;s Superintelligence Labs, rolling into Meta AI, Instagram and WhatsApp.&#185; The headline feature let you generate images of friends and creators by tagging their public Instagram accounts, enabled by default, opt-out buried in settings.&#178; By Friday evening it was gone as Meta said the feature missed the mark, to the surprise of absolutely no one.&#179; </p><p><strong>OpenAI</strong> had the busiest week of anyone. GPT-5.6 went to general availability in three tiers pitched less on raw intelligence and more on intelligence per dollar.&#8308; Alongside it came ChatGPT Work, an agent that reaches into your Slack, Notion, Drive and Microsoft 365 to produce finished decks, sheets and docs, and can grind on a project for hours.&#8309; To make room for the new launches, OpenAI killed the ChatGPT Atlas browser, nine months after launching it, folding the agentic browsing into a desktop super-app.&#8310; It seems that OpenAI has inherited Microsoft&#8217;s ability to kill products very fast.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!CE1X!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83f878c2-d5cb-41d0-8ca0-3595419c28b6_1582x922.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!CE1X!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83f878c2-d5cb-41d0-8ca0-3595419c28b6_1582x922.png 424w, /__u/substackcdn.com/image/fetch/$s_!CE1X!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83f878c2-d5cb-41d0-8ca0-3595419c28b6_1582x922.png 848w, /__u/substackcdn.com/image/fetch/$s_!CE1X!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83f878c2-d5cb-41d0-8ca0-3595419c28b6_1582x922.png 1272w, /__u/substackcdn.com/image/fetch/$s_!CE1X!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83f878c2-d5cb-41d0-8ca0-3595419c28b6_1582x922.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!CE1X!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83f878c2-d5cb-41d0-8ca0-3595419c28b6_1582x922.png" width="1456" height="849" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/83f878c2-d5cb-41d0-8ca0-3595419c28b6_1582x922.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:849,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:84293,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://thedatajourney.substack.com/i/206680761?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83f878c2-d5cb-41d0-8ca0-3595419c28b6_1582x922.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!CE1X!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83f878c2-d5cb-41d0-8ca0-3595419c28b6_1582x922.png 424w, /__u/substackcdn.com/image/fetch/$s_!CE1X!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83f878c2-d5cb-41d0-8ca0-3595419c28b6_1582x922.png 848w, /__u/substackcdn.com/image/fetch/$s_!CE1X!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83f878c2-d5cb-41d0-8ca0-3595419c28b6_1582x922.png 1272w, /__u/substackcdn.com/image/fetch/$s_!CE1X!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83f878c2-d5cb-41d0-8ca0-3595419c28b6_1582x922.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">OpenAI now benchmarks its models on two axes &#8212; accuracy and cost &#8212; not just accuracy.</figcaption></figure></div><p><strong>Anthropic</strong> moved in the same direction from the other side. Claude Cowork, which launched in January as a desktop app, is now on web and mobile, start a task at your desk, get pinged on your phone, collect the output later with your laptop shut.&#8311; </p><p><strong>Lovable</strong> is reportedly in talks to raise $300M at a $13.2bn valuation, roughly double its December mark, on the back of $500M in annualised revenue and 146 employees.&#8312; <strong>ElevenLabs</strong> is separately discussing a tender offer at around $22bn, double what it was worth in February.&#8313; Both companies have real revenue, which is more than you could say for most of the last technological bubble in the 00&#8217;s.</p><div><hr></div><h4>&#187; Industry Highlights: Research &amp; Science</h4><p><strong>GPT-5.6&#8217;s</strong> numbers are worth highlighting here. Sol posts 53.6 on the Agents&#8217; Last Exam, ahead of Claude Fable 5 by 13.1 points, and OpenAI claims Terra and Luna beat Fable at roughly a sixteenth of the cost.&#8308; The company also reports that internal research compute spent on coding inference has grown 100x in six months. The model is supposedly getting better at building the model. </p><p><strong>OpenAI&#8217;s</strong> head of safety, Johannes Heidecke, is leaving after a reorg that folds the safety teams into research under Mia Glaese.&#185;&#185; The company&#8217;s stated reason is that safety should sit closer to the people shipping frontier models. Heidecke is at least the fifth senior safety leader out of OpenAI in two years, and the pattern is now more informative than any single exit.</p><div><hr></div><h4>&#187; Economic Highlights: The World of AI</h4><p><strong>DeepSeek</strong> is reportedly designing its own inference chip, fabbed at SMIC rather than TSMC, according to Reuters.&#185;&#178; Roughly 70% of compute demand is expected to be inference, which is precisely where Chinese hardware is trying to gain leverage, reducing exposure to American chips.</p><p><strong>Tencent</strong> is in talks to become the largest shareholder in Manus, leading a consortium of the startup&#8217;s original backers buying it back at the same $2bn Meta had agreed to pay before Beijing blocked the deal.&#185;&#179; </p><p><strong>On the European&#8217;s side, Mistral&#8217;s</strong> Arthur Mensch spent the week telling enterprises that closed model providers gain &#8220;immense leverage&#8221; over their customers, retain their data, and have a habit of competing with the businesses they serve.&#185;&#8304; </p><div><hr></div><p>And that&#8217;s it for this week, with two large ideas to retain,</p><p>First, Meta did not need feedback to know that launching the AI feature was a bad idea. They shipped it anyway, because the cost of asking permission is a slower launch and the cost of not asking is an apology - this is Meta&#8217;s way of doing things, since ever. But it's hard not to worry when one of the most powerful companies on earth ships a feature this stupid, with this much potential to hurt people.</p><p>The second idea is the continuing of the pace, that I&#8217;ve highlighted multiple times in this newsletter. GPT-5.6, ChatGPT Work, Cowork on mobile, a Chinese inference chip in design, all in one week.  The pace of new products shows no sign of stopping,</p><p>See you next week!</p><p>&#8212; Ivo</p><div><hr></div><h2>Footnotes</h2><p>1 <a href="https://www.nytimes.com/2026/07/07/technology/meta-ai-muse-image.html">https://www.nytimes.com/2026/07/07/technology/meta-ai-muse-image.html</a></p><p>2 <a href="https://thenextweb.com/news/meta-muse-image-ai-generation-instagram-whatsapp">https://thenextweb.com/news/meta-muse-image-ai-generation-instagram-whatsapp</a></p><p>3 <a href="https://www.bbc.com/news/articles/c2dy6e8klw0o">https://www.bbc.com/news/articles/c2dy6e8klw0o</a></p><p>4 <a href="https://openai.com/pt-PT/index/gpt-5-6/">https://openai.com/pt-PT/index/gpt-5-6/</a></p><p>5 <a href="https://openai.com/pt-PT/index/chatgpt-for-your-most-ambitious-work/">https://openai.com/pt-PT/index/chatgpt-for-your-most-ambitious-work/</a></p><p>6 <a href="https://thenextweb.com/news/openai-chatgpt-atlas-browser-shutdown-superapp">https://thenextweb.com/news/openai-chatgpt-atlas-browser-shutdown-superapp</a></p><p>7 <a href="https://techcrunch.com/2026/07/07/the-coding-agent-wars-are-spilling-into-the-rest-of-the-office-claude-cowork/">https://techcrunch.com/2026/07/07/the-coding-agent-wars-are-spilling-into-the-rest-of-the-office-claude-cowork/</a></p><p>8 <a href="https://thenextweb.com/news/lovable-is-reportedly-in-talks-to-raise-300m-at-a-13-2bn-valuation">https://thenextweb.com/news/lovable-is-reportedly-in-talks-to-raise-300m-at-a-13-2bn-valuation</a></p><p>9 <a href="https://thenextweb.com/news/elevenlabs-tender-offer-22-billion-valuation">https://thenextweb.com/news/elevenlabs-tender-offer-22-billion-valuation</a></p><p>10 <a href="https://thenextweb.com/news/mistral-ceo-open-source-enterprise-warning">https://thenextweb.com/news/mistral-ceo-open-source-enterprise-warning</a></p><p>11 <a href="https://www.wired.com/story/openai-head-of-safety-leaving/">https://www.wired.com/story/openai-head-of-safety-leaving/</a></p><p>12 <a href="https://thenextweb.com/news/deepseek-own-ai-chip-smic-inference-report">https://thenextweb.com/news/deepseek-own-ai-chip-smic-inference-report</a></p><p>13 <a href="https://thenextweb.com/news/tencent-in-talks-to-become-manus-largest-shareholder">https://thenextweb.com/news/tencent-in-talks-to-become-manus-largest-shareholder</a></p><p>14 <a href="https://www.nytimes.com/2026/07/10/technology/apple-openai-lawsuit.html">https://www.nytimes.com/2026/07/10/technology/apple-openai-lawsuit.html</a></p>]]></content:encoded></item><item><title><![CDATA[The AI Handbook for The Curious Mind: Chapter 4.1.3]]></title><description><![CDATA[Bias vs. Variance]]></description><link>https://thedatajourney.substack.com/p/the-ai-handbook-for-the-curious-mind-dd3</link><guid isPermaLink="false">https://thedatajourney.substack.com/p/the-ai-handbook-for-the-curious-mind-dd3</guid><dc:creator><![CDATA[Ivo Bernardo]]></dc:creator><pubDate>Thu, 09 Jul 2026 18:59:11 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!KBpd!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2698494d-d5d2-41a9-add4-673d8d0f8e40_700x863.jpeg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Hey everyone,</p><p><span>Some of you may know, but a couple of months ago I wrote a book. It&#8217;s called </span><strong>The AI Handbook for the Curious Mind</strong><span>, and it&#8217;s available on </span><a href="https://www.amazon.com/Handbook-Curious-Mind-Understanding-Intelligence-ebook/dp/B0G4KTP2QP/ref=tmm_kin_swatch_0">Amazon</a><span> or </span><a href="http://ivopbernardo.gumroad.com/l/aihandbookforthecuriousmind">Gumroad</a><span>.</span></p><p>Every two weeks, I&#8217;ll be releasing a full chapter here, exclusively for paid subscribers. I keep looking for ways to give back to the people who support this newsletter, and sharing the book felt like one of the more meaningful ones. &#128516;</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!KBpd!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2698494d-d5d2-41a9-add4-673d8d0f8e40_700x863.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!KBpd!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2698494d-d5d2-41a9-add4-673d8d0f8e40_700x863.jpeg 424w, /__u/substackcdn.com/image/fetch/$s_!KBpd!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2698494d-d5d2-41a9-add4-673d8d0f8e40_700x863.jpeg 848w, /__u/substackcdn.com/image/fetch/$s_!KBpd!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2698494d-d5d2-41a9-add4-673d8d0f8e40_700x863.jpeg 1272w, /__u/substackcdn.com/image/fetch/$s_!KBpd!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2698494d-d5d2-41a9-add4-673d8d0f8e40_700x863.jpeg 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!KBpd!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2698494d-d5d2-41a9-add4-673d8d0f8e40_700x863.jpeg" width="700" height="863" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2698494d-d5d2-41a9-add4-673d8d0f8e40_700x863.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:863,&quot;width&quot;:700,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:&quot;&quot;,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="/__u/substackcdn.com/image/fetch/$s_!KBpd!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2698494d-d5d2-41a9-add4-673d8d0f8e40_700x863.jpeg 424w, /__u/substackcdn.com/image/fetch/$s_!KBpd!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2698494d-d5d2-41a9-add4-673d8d0f8e40_700x863.jpeg 848w, /__u/substackcdn.com/image/fetch/$s_!KBpd!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2698494d-d5d2-41a9-add4-673d8d0f8e40_700x863.jpeg 1272w, /__u/substackcdn.com/image/fetch/$s_!KBpd!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2698494d-d5d2-41a9-add4-673d8d0f8e40_700x863.jpeg 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Last time, Jack and John (linear regression and the decision tree) went head to head over house prices. John won on the numbers. But he won with an asterisk: he was noticeably sharper on the data he&#8217;d already studied than on the data he hadn&#8217;t.</p><p>That caveat is the whole subject of today&#8217;s post. It&#8217;s the eternal dilemma of machine learning: models that learn too little vs. models that learn too much. A model that does well on the data it trained on has done nothing impressive. Anyone can memorise an answer key. What we want is a model that holds up on data it has <em>never see,</em> new houses, new customers, next month. A good model, like a good student, learns the underlying pattern instead of memorising the practice questions. </p><p>Let&#8217;s dive into the concepts of bias and variance.</p>
      <p>
          <a href="/__u/thedatajourney.substack.com/p/the-ai-handbook-for-the-curious-mind-dd3">
              Read more
          </a>
      </p>
   ]]></content:encoded></item><item><title><![CDATA[Weekly AI Digest: Fable Unlocked]]></title><description><![CDATA[Week 27, 2026]]></description><link>https://thedatajourney.substack.com/p/weekly-ai-digest-fable-unlocked</link><guid isPermaLink="false">https://thedatajourney.substack.com/p/weekly-ai-digest-fable-unlocked</guid><dc:creator><![CDATA[Ivo Bernardo]]></dc:creator><pubDate>Fri, 03 Jul 2026 17:04:31 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!uxOE!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d3dd7ef-31ec-4bc5-8217-72d2972b1c25_1254x1254.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Good morning everyone,</p><p>This week we&#8217;ll start by speaking about the US government becoming a shareholder of one of the big labs in the world.</p><p>Then we&#8217;ll move into new models, new proprietary models, new deployment armies, and new research-automation moonshots. We&#8217;re riding the exponential, and every week the news gives us another read on how the future is playing out, from a technological lens.</p><div><hr></div><h4>&#187; Stat of the Week: 5%</h4><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!uxOE!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d3dd7ef-31ec-4bc5-8217-72d2972b1c25_1254x1254.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!uxOE!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d3dd7ef-31ec-4bc5-8217-72d2972b1c25_1254x1254.png 424w, /__u/substackcdn.com/image/fetch/$s_!uxOE!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d3dd7ef-31ec-4bc5-8217-72d2972b1c25_1254x1254.png 848w, /__u/substackcdn.com/image/fetch/$s_!uxOE!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d3dd7ef-31ec-4bc5-8217-72d2972b1c25_1254x1254.png 1272w, /__u/substackcdn.com/image/fetch/$s_!uxOE!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d3dd7ef-31ec-4bc5-8217-72d2972b1c25_1254x1254.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!uxOE!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d3dd7ef-31ec-4bc5-8217-72d2972b1c25_1254x1254.png" width="466" height="466" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/5d3dd7ef-31ec-4bc5-8217-72d2972b1c25_1254x1254.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1254,&quot;width&quot;:1254,&quot;resizeWidth&quot;:466,&quot;bytes&quot;:1607350,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://thedatajourney.substack.com/i/204912336?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d3dd7ef-31ec-4bc5-8217-72d2972b1c25_1254x1254.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!uxOE!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d3dd7ef-31ec-4bc5-8217-72d2972b1c25_1254x1254.png 424w, /__u/substackcdn.com/image/fetch/$s_!uxOE!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d3dd7ef-31ec-4bc5-8217-72d2972b1c25_1254x1254.png 848w, /__u/substackcdn.com/image/fetch/$s_!uxOE!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d3dd7ef-31ec-4bc5-8217-72d2972b1c25_1254x1254.png 1272w, /__u/substackcdn.com/image/fetch/$s_!uxOE!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d3dd7ef-31ec-4bc5-8217-72d2972b1c25_1254x1254.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>5% is the stake that OpenAI has reportedly floated handing to a US sovereign wealth fund to smooth things over with the administration.&#185; </p><div><hr></div><h4>&#187; Industry Highlights: Market &amp; Product</h4><p><strong>Claude Sonnet 5</strong> landed as Anthropic&#8217;s most agentic Sonnet yet, a model that plans, drives browsers and terminals, and runs autonomously at a level that until recently needed something much bigger and pricier.&#178; It gets close to Opus 4.8 on reasoning, coding, and tool use, but with an adjustable effort dial and introductory pricing of $2 per million input tokens through the end of August.</p><p><strong>Base44</strong> did the thing everyone said was too hard and shipped its own model. The Wix-owned vibe-coding platform launched Base1, trained on tens of millions of real user builds, and it&#8217;s already in production.&#179; The bet is simple: renting Opus or GPT means paying for every token your users&#8217; apps generate, and owning the model is the only way to a structurally better margin and protect yourself from the future economics. Founder<a href="https://base44.com/blog/maor-shlomo-building-the-model-behind-base"> Maor Shlomo&#8217;s pitch is specialisation</a>. Frontier models stay general, a model tuned on app-building beats them at that one job. It&#8217;s still not clear which angle will win (and if it will be a winner-takes-it-all scenario), but this is such as interesting space to watch.</p><p><strong>On Microsoft&#8217;s side, the company</strong> Microsoft Frontier, a $2.5 billion, 6,000-engineer business whose entire job is getting enterprise AI to work in production rather than in a demo.&#8308; AWS committed $1 billion to the same idea two days earlier; OpenAI&#8217;s version closed at $10 billion, Anthropic&#8217;s at $1.5 billion. Every major vendor now a FDE model scenario. This is exactly how we&#8217;ve run DareData for years, and it&#8217;s a nice validations to see the largest companies on earth discover, at $2.5 billion a pop, that software doesn&#8217;t deploy itself and someone has to sit next to the customer and make it real. Turns out boots on the ground may be the moat all along.</p><p><strong>Finally, Mirendil</strong> raised $200 million at a $1 billion valuation with no product, founded by two researchers who left Anthropic after barely a year.&#8309; Anthropic has said Claude writes north of 80% of its own code, but their terms of service forbid you from doing the same. Mirendil wants to sell that engine to everyone else: AI that designs experiments, tunes settings, evaluates models, and runs the next training round, so a biology lab could build a drug-target model without an ML team.</p><div><hr></div><h4>&#187; Industry Highlights: Research &amp; Science</h4><p>On research and science, I recommend you to check this great video from &#8220;Two Minute Papers&#8221; on the latest open source model, GLM 5.2.</p><p><strong>GLM-5.2</strong> is the week's proof that open source keeps chasing frontier labs. Chinese lab Z.ai shipped an open flagship built specifically for long-horizon work, pairing a genuinely usable 1M-token context with an effort dial (the same idea Sonnet 5 is selling) and a no-strings MIT licence. On Terminal-Bench 2.1 it lands at 81.0 against Opus 4.8's 85.0, edges out GPT-5.5 on FrontierSWE, and sits ahead of Gemini 3.1 Pro. It's the strongest open model out there right now. China has the strongest open source community and launches in the technological world right now, who would know?</p><div id="youtube2-qks6dGQFd_c" class="youtube-wrap" data-attrs="{&quot;videoId&quot;:&quot;qks6dGQFd_c&quot;,&quot;startTime&quot;:null,&quot;endTime&quot;:null}" data-component-name="Youtube2ToDOM"><div class="youtube-inner"><iframe src="https://www.youtube-nocookie.com/embed/qks6dGQFd_c?rel=0&amp;autoplay=0&amp;showinfo=0&amp;enablejsapi=0" frameborder="0" loading="lazy" gesture="media" allow="autoplay; fullscreen" allowautoplay="true" allowfullscreen="true" width="728" height="409"></iframe></div></div><div><hr></div><h4>&#187; Economic Highlights: The World of AI</h4><p><strong>Anthropic</strong> got its most powerful models back. The US lifted the export licence requirement it had slapped on Mythos and Fable in June, and access started returning July 1.&#8311; The official framing is a set of security commitments, most of which Anthropic had already pledged voluntarily months earlier, which is why plenty of observers read the original ban as leverage rather than safety. </p><p><strong>As mentioned in the fact of the week, OpenAI</strong> floated giving away a slice of itself. Sam Altman has reportedly proposed handing 5% of OpenAI&#8217;s equity to a US sovereign wealth fund, with other AI companies contributing similar stakes.&#185; </p><div><hr></div><h4>&#187; Closing</h4><p>This week we had another validation for the Consulting + Product model with Microsoft launching its own deployment company.</p><p>On the geopolitical side, the US government is considering a stake in the AI labs, while China is ruling the open source world. </p><p>We keep going up on the technological exponential, without signs of slowing down. What&#8217;s your largest concert about the evolution of AI? Let me know in the comments! </p><p>&#8212; Ivo</p><div><hr></div><p>1 <a href="https://techcrunch.com/2026/07/02/openai-proposed-donating-5-of-its-equity-to-a-us-sovereign-wealth-fund/">https://techcrunch.com/2026/07/02/openai-proposed-donating-5-of-its-equity-to-a-us-sovereign-wealth-fund/</a></p><p>2 <a href="https://www.anthropic.com/news/claude-sonnet-5">https://www.anthropic.com/news/claude-sonnet-5</a></p><p>3 <a href="https://thenextweb.com/news/base44-base1-proprietary-ai-model-vibe-coding">https://thenextweb.com/news/base44-base1-proprietary-ai-model-vibe-coding</a></p><p>4 <a href="https://thenextweb.com/news/microsoft-frontier-company-2-5-billion-ai-deployment">https://thenextweb.com/news/microsoft-frontier-company-2-5-billion-ai-deployment</a></p><p>5 <a href="https://thenextweb.com/news/mirendil-200m-seed-ai-research">https://thenextweb.com/news/mirendil-200m-seed-ai-research</a></p><p>6 <a href="https://techcrunch.com/2026/07/02/mark-zuckerberg-tells-staff-that-ai-agents-havent-progressed-as-quickly-as-hed-hoped/">https://techcrunch.com/2026/07/02/mark-zuckerberg-tells-staff-that-ai-agents-havent-progressed-as-quickly-as-hed-hoped/</a></p><p>7 <a href="https://techcrunch.com/2026/06/30/trump-drops-restrictions-on-anthropics-mythos-and-fable-models/">https://techcrunch.com/2026/06/30/trump-drops-restrictions-on-anthropics-mythos-and-fable-models/</a></p>]]></content:encoded></item><item><title><![CDATA[Setting up your own Large Language Model]]></title><description><![CDATA[Still a long way to go, but the future is promising]]></description><link>https://thedatajourney.substack.com/p/setting-up-your-own-large-language</link><guid isPermaLink="false">https://thedatajourney.substack.com/p/setting-up-your-own-large-language</guid><dc:creator><![CDATA[Ivo Bernardo]]></dc:creator><pubDate>Thu, 02 Jul 2026 17:19:30 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!kOGf!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1b369c21-1d9d-4fdb-9dc8-280e5dee2b1b_1018x486.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div><hr></div><p>You&#8217;ve likely seen the headlines: frontier AI models are increasingly at risk of being locked behind strict export controls or mounting API costs.</p><p>As this technology embeds itself into our daily lives, the open-source movement isn&#8217;t just a philosophical preference, it is a necessary mechanism to keep AI in the hands of everyday users. We aren&#8217;t at parity yet; the proprietary models from the massive tech labs still hold a commanding lead in pure performance. But, we can hope that the gap is closing fast. Around the clock, an independent community of researchers and developers is pushing to ensure this technology is accessible to anyone with a computer.</p><p>Today, the foundation for true democratization is already here: you can run a highly capable model entirely on your own laptop. For today&#8217;s experiment, I set out to find a large language model that can run entirely on my laptop&#8202;&#8212;&#8202;and use it for the simple tasks I&#8217;d normally hand off to a big lab model.</p><p>We&#8217;ll install &#8220;<a href="https://huggingface.co/Qwen/Qwen3-8B">Qwen 3 8B</a>&#8221; on my MacBook Air, run it fully offline, and finally have a language model living on my own machine instead of a distant datacenter. The <a href="https://en.wikipedia.org/wiki/Qwen">Qwen</a> family of models have been trained by Alibaba (the chinese company) and are fully open source, available on the internet for everyone to download. The model has 9 billion weights and takes up around 6gb of your RAM when loaded.</p><p>What follows now is a practical, start-to-finish guide to running a proper local LLM on an Apple Silicon Mac and it includes the terminal commands you need. But before we open the terminal, we need to talk about why this is worth doing at all.</p><div><hr></div><h4>Why Do This? (The Honest Version)</h4><p>Most of the time, cloud models are better and easier. I&#8217;m not going to pretend an 8-billion parameter model on a laptop beats frontier AI. It doesn&#8217;t and I will keep using the massive cloud models for heavy lifting.</p><p>But the constant pricing and sovereignity wars around AI may make open source and local models very relevant for a future where having access to the technology will make a huge difference. Every time you use Claude or ChatGPT, you are sending your data to some remote servers where the access can be blocked at any time.</p><p>&#8220;<em>Digital sovereignty</em>&#8221; is a grand phrase for a very ordinary desire: we may want to own the thing that reads our most sensitive thoughts, the same way you own a physical notebook or keep some cash at home.</p><p>A local model answers that cleanly in the AI world. Once it&#8217;s downloaded, nothing leaves the machine. No API keys, no shifting terms of service, no quiet data retention policies. You can pull the Wi-Fi card out and it keeps working. For the highly sensitive part of your work, that alone may be worth the price of admission.</p><p>People love to say local models are &#8220;<em>democratizing</em>&#8221; AI. I want that to be true, but we aren&#8217;t there yet. Running this stack still assumes you own a &#8364;1,500 laptop with massive unified memory and you&#8217;re comfortable in a command line. That&#8217;s a narrow, lucky slice of the world.</p><p>But the <em>trajectory</em> is democratizing. Two years ago, running a decent offline model required a dedicated workstation and serious technical pain. This weekend, it took me a couple of hours and 5 gigabytes of disk space.</p><p>So let&#8217;s install the thing.</p><div><hr></div><h4>The Machine and the Specs</h4><p>I built this on a <strong>MacBook Air M4</strong> with <strong>24 GB of unified memory</strong> and about 235 GB of free storage. This was a fresh start: no Homebrew, no Python environment nightmares.</p><p>The number that actually matters here is the <strong>24 GB</strong>. Apple Silicon&#8217;s &#8220;unified memory&#8221; is the magic trick that makes Macs so exceptionally good at this. Because the CPU and GPU share the exact same memory pool, massive neural network weights don&#8217;t have to be sluggishly shuttled back and forth.</p><p>An 8B model takes up about 5 GB on disk and sits at roughly 6 GB in memory when loaded. On a 24 GB machine, that&#8217;s deeply comfortable. You could run a 14B model and still keep dozens of browser tabs open. (If you&#8217;re on an 8 GB Mac, stick to the 1.5B or 3B models and close your other apps).</p><div><hr></div><h4>Why Ollama?</h4><p>There are a dozen ways to run local AI, and most of them ask you to care about compiler flags and dependency trees. You shouldn&#8217;t have to.</p><p><a href="https://ollama.com/">Ollama</a> is an open source framework and tool that just works. It&#8217;s a single binary that bundles a highly optimized model runner (<code>llama.cpp</code> using Apple&#8217;s Metal for GPU acceleration), a Docker-style model registry, and a local HTTP API. You install it, you pull a model, and you talk to it. That&#8217;s it!</p><div><hr></div><h4>Step 1: Install Ollama (No Homebrew Required)</h4><p>Ollama ships as a standard macOS app in a zip file. The command-line interface (CLI) lives secretly inside the app bundle, so we can set it up entirely by hand.</p><pre><code># Download the Apple Silicon build
cd ~/Downloads
curl -L -o Ollama-darwin.zip https://ollama.com/download/Ollama-darwin.zip</code></pre><pre><code># Unzip and move the app into your Applications folder
unzip -o -q Ollama-darwin.zip
mv Ollama.app /Applications/</code></pre><p>If you don&#8217;t know how to open the terminal, just go to your Mac applications and search for &#8220;terminal&#8221;:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!gLH3!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07c42825-b444-4649-83f0-7af04e48d343_1600x1069.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!gLH3!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07c42825-b444-4649-83f0-7af04e48d343_1600x1069.png 424w, /__u/substackcdn.com/image/fetch/$s_!gLH3!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07c42825-b444-4649-83f0-7af04e48d343_1600x1069.png 848w, /__u/substackcdn.com/image/fetch/$s_!gLH3!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07c42825-b444-4649-83f0-7af04e48d343_1600x1069.png 1272w, /__u/substackcdn.com/image/fetch/$s_!gLH3!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07c42825-b444-4649-83f0-7af04e48d343_1600x1069.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!gLH3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07c42825-b444-4649-83f0-7af04e48d343_1600x1069.png" width="1456" height="973" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/07c42825-b444-4649-83f0-7af04e48d343_1600x1069.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:973,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!gLH3!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07c42825-b444-4649-83f0-7af04e48d343_1600x1069.png 424w, /__u/substackcdn.com/image/fetch/$s_!gLH3!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07c42825-b444-4649-83f0-7af04e48d343_1600x1069.png 848w, /__u/substackcdn.com/image/fetch/$s_!gLH3!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07c42825-b444-4649-83f0-7af04e48d343_1600x1069.png 1272w, /__u/substackcdn.com/image/fetch/$s_!gLH3!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07c42825-b444-4649-83f0-7af04e48d343_1600x1069.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Mac Terminal</figcaption></figure></div><div><hr></div><h4>Step 2: Put Ollama on Your PATH</h4><p>I didn&#8217;t want to fight with <code>sudo</code> permissions in <code>/usr/local/bin</code>, so I symlinked the bundled CLI into a local directory I own&#8202;&#8212;&#8202;this is just a handy shortcut to speed up the installation and spin up the LLM.</p><pre><code># Create a local bin directory and symlink the CLI
mkdir -p ~/.local/bin
ln -sf /Applications/Ollama.app/Contents/Resources/ollama ~/.local/bin/ollama

# Make it permanent in your zsh profile
echo &#8216;export PATH=&#8221;$HOME/.local/bin:$PATH&#8221;&#8217; &gt;&gt; ~/.zshrc
# Apply it to your current shell
export PATH=&#8221;$HOME/.local/bin:$PATH&#8221;
ollama --version</code></pre><div><hr></div><h4>Step 3: Start the Server</h4><p>Ollama runs a lightweight background server to expose the API and manage your computer&#8217;s memory.</p><pre><code># Start the server and log output
mkdir -p ~/.ollama/logs
nohup ollama serve &gt; ~/.ollama/logs/serve.log 2&gt;&amp;1 &amp;

# Ping it to check if it&#8217;s alive
curl -s http://127.0.0.1:11434/api/version</code></pre><p>If the command above returns a &#8220;version&#8221;, ollama is set up!</p><div class="captioned-image-container"><figure><a class="image-link image2" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!RBuZ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8832d4de-5dd6-4449-8f26-7e66f4ba0718_1080x64.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!RBuZ!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8832d4de-5dd6-4449-8f26-7e66f4ba0718_1080x64.png 424w, /__u/substackcdn.com/image/fetch/$s_!RBuZ!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8832d4de-5dd6-4449-8f26-7e66f4ba0718_1080x64.png 848w, /__u/substackcdn.com/image/fetch/$s_!RBuZ!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8832d4de-5dd6-4449-8f26-7e66f4ba0718_1080x64.png 1272w, /__u/substackcdn.com/image/fetch/$s_!RBuZ!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8832d4de-5dd6-4449-8f26-7e66f4ba0718_1080x64.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!RBuZ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8832d4de-5dd6-4449-8f26-7e66f4ba0718_1080x64.png" width="1080" height="64" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/8832d4de-5dd6-4449-8f26-7e66f4ba0718_1080x64.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:64,&quot;width&quot;:1080,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!RBuZ!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8832d4de-5dd6-4449-8f26-7e66f4ba0718_1080x64.png 424w, /__u/substackcdn.com/image/fetch/$s_!RBuZ!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8832d4de-5dd6-4449-8f26-7e66f4ba0718_1080x64.png 848w, /__u/substackcdn.com/image/fetch/$s_!RBuZ!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8832d4de-5dd6-4449-8f26-7e66f4ba0718_1080x64.png 1272w, /__u/substackcdn.com/image/fetch/$s_!RBuZ!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8832d4de-5dd6-4449-8f26-7e66f4ba0718_1080x64.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></a></figure></div><p><em>Note: You can also just double-click the Ollama app in your Applications folder to run this server via your menu bar. I did it via terminal to see exactly what was happening under the hood.</em></p><div><hr></div><h4>Step 4: Pull the Model</h4><p>Well this one is as easy as it gets:</p><pre><code>ollama pull qwen3:8b     
ollama list</code></pre><p>Go make a coffee. The download is about 5.2 GB.</p><p>After running ollama list, you&#8217;ll see the model available for you:</p><div class="captioned-image-container"><figure><a class="image-link image2" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!xLoG!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8e3ec6cb-287e-4954-b7a2-28df74020581_798x94.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!xLoG!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8e3ec6cb-287e-4954-b7a2-28df74020581_798x94.png 424w, /__u/substackcdn.com/image/fetch/$s_!xLoG!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8e3ec6cb-287e-4954-b7a2-28df74020581_798x94.png 848w, /__u/substackcdn.com/image/fetch/$s_!xLoG!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8e3ec6cb-287e-4954-b7a2-28df74020581_798x94.png 1272w, /__u/substackcdn.com/image/fetch/$s_!xLoG!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8e3ec6cb-287e-4954-b7a2-28df74020581_798x94.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!xLoG!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8e3ec6cb-287e-4954-b7a2-28df74020581_798x94.png" width="798" height="94" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/8e3ec6cb-287e-4954-b7a2-28df74020581_798x94.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:94,&quot;width&quot;:798,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!xLoG!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8e3ec6cb-287e-4954-b7a2-28df74020581_798x94.png 424w, /__u/substackcdn.com/image/fetch/$s_!xLoG!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8e3ec6cb-287e-4954-b7a2-28df74020581_798x94.png 848w, /__u/substackcdn.com/image/fetch/$s_!xLoG!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8e3ec6cb-287e-4954-b7a2-28df74020581_798x94.png 1272w, /__u/substackcdn.com/image/fetch/$s_!xLoG!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8e3ec6cb-287e-4954-b7a2-28df74020581_798x94.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></a></figure></div><div><hr></div><h4>Step 5: Talk to the new digital Brain in your Computer</h4><p>You have three distinct ways to interact with your new local model.</p><p><strong>1. Interactive Chat (The Easiest)</strong></p><pre><code>ollama run qwen3:8b</code></pre><div class="captioned-image-container"><figure><a class="image-link image2" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!k2vI!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcc5f1031-0185-4f92-9e19-15886bb53aa7_902x88.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!k2vI!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcc5f1031-0185-4f92-9e19-15886bb53aa7_902x88.png 424w, /__u/substackcdn.com/image/fetch/$s_!k2vI!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcc5f1031-0185-4f92-9e19-15886bb53aa7_902x88.png 848w, /__u/substackcdn.com/image/fetch/$s_!k2vI!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcc5f1031-0185-4f92-9e19-15886bb53aa7_902x88.png 1272w, /__u/substackcdn.com/image/fetch/$s_!k2vI!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcc5f1031-0185-4f92-9e19-15886bb53aa7_902x88.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!k2vI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcc5f1031-0185-4f92-9e19-15886bb53aa7_902x88.png" width="902" height="88" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/cc5f1031-0185-4f92-9e19-15886bb53aa7_902x88.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:88,&quot;width&quot;:902,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!k2vI!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcc5f1031-0185-4f92-9e19-15886bb53aa7_902x88.png 424w, /__u/substackcdn.com/image/fetch/$s_!k2vI!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcc5f1031-0185-4f92-9e19-15886bb53aa7_902x88.png 848w, /__u/substackcdn.com/image/fetch/$s_!k2vI!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcc5f1031-0185-4f92-9e19-15886bb53aa7_902x88.png 1272w, /__u/substackcdn.com/image/fetch/$s_!k2vI!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcc5f1031-0185-4f92-9e19-15886bb53aa7_902x88.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></a></figure></div><p>In the default mode, the model will spill out the &#8220;thinking tokens&#8221;, something that is normally abstracted and hidden in most commercial tools.</p><p>I&#8217;m going to start by asking my local model what it thinks about open source models:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!1FlZ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F67df91cc-ccbf-4e46-9840-8da37b48df74_1132x622.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!1FlZ!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F67df91cc-ccbf-4e46-9840-8da37b48df74_1132x622.png 424w, /__u/substackcdn.com/image/fetch/$s_!1FlZ!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F67df91cc-ccbf-4e46-9840-8da37b48df74_1132x622.png 848w, /__u/substackcdn.com/image/fetch/$s_!1FlZ!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F67df91cc-ccbf-4e46-9840-8da37b48df74_1132x622.png 1272w, /__u/substackcdn.com/image/fetch/$s_!1FlZ!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F67df91cc-ccbf-4e46-9840-8da37b48df74_1132x622.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!1FlZ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F67df91cc-ccbf-4e46-9840-8da37b48df74_1132x622.png" width="1132" height="622" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/67df91cc-ccbf-4e46-9840-8da37b48df74_1132x622.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:622,&quot;width&quot;:1132,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!1FlZ!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F67df91cc-ccbf-4e46-9840-8da37b48df74_1132x622.png 424w, /__u/substackcdn.com/image/fetch/$s_!1FlZ!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F67df91cc-ccbf-4e46-9840-8da37b48df74_1132x622.png 848w, /__u/substackcdn.com/image/fetch/$s_!1FlZ!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F67df91cc-ccbf-4e46-9840-8da37b48df74_1132x622.png 1272w, /__u/substackcdn.com/image/fetch/$s_!1FlZ!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F67df91cc-ccbf-4e46-9840-8da37b48df74_1132x622.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Answer from the Local Model</figcaption></figure></div><p>The light grey text represents the model&#8217;s internal reasoning process. These models perform extensive calculation before generating a response, and for local models, this thinking phase accounts for a significant portion of the total time until the model spews out a response.</p><p>After doing the thinking process, here is the answer from the model:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!o9bR!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1f9a675a-706d-4fe1-a427-48a2c99acd5d_1216x1034.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!o9bR!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1f9a675a-706d-4fe1-a427-48a2c99acd5d_1216x1034.png 424w, /__u/substackcdn.com/image/fetch/$s_!o9bR!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1f9a675a-706d-4fe1-a427-48a2c99acd5d_1216x1034.png 848w, /__u/substackcdn.com/image/fetch/$s_!o9bR!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1f9a675a-706d-4fe1-a427-48a2c99acd5d_1216x1034.png 1272w, /__u/substackcdn.com/image/fetch/$s_!o9bR!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1f9a675a-706d-4fe1-a427-48a2c99acd5d_1216x1034.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!o9bR!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1f9a675a-706d-4fe1-a427-48a2c99acd5d_1216x1034.png" width="1216" height="1034" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/1f9a675a-706d-4fe1-a427-48a2c99acd5d_1216x1034.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1034,&quot;width&quot;:1216,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!o9bR!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1f9a675a-706d-4fe1-a427-48a2c99acd5d_1216x1034.png 424w, /__u/substackcdn.com/image/fetch/$s_!o9bR!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1f9a675a-706d-4fe1-a427-48a2c99acd5d_1216x1034.png 848w, /__u/substackcdn.com/image/fetch/$s_!o9bR!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1f9a675a-706d-4fe1-a427-48a2c99acd5d_1216x1034.png 1272w, /__u/substackcdn.com/image/fetch/$s_!o9bR!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1f9a675a-706d-4fe1-a427-48a2c99acd5d_1216x1034.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Answer from Local Model</figcaption></figure></div><p>Was with most tools, these models also retain some context from previous interactions:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!kOGf!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1b369c21-1d9d-4fdb-9dc8-280e5dee2b1b_1018x486.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!kOGf!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1b369c21-1d9d-4fdb-9dc8-280e5dee2b1b_1018x486.png 424w, /__u/substackcdn.com/image/fetch/$s_!kOGf!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1b369c21-1d9d-4fdb-9dc8-280e5dee2b1b_1018x486.png 848w, /__u/substackcdn.com/image/fetch/$s_!kOGf!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1b369c21-1d9d-4fdb-9dc8-280e5dee2b1b_1018x486.png 1272w, /__u/substackcdn.com/image/fetch/$s_!kOGf!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1b369c21-1d9d-4fdb-9dc8-280e5dee2b1b_1018x486.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!kOGf!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1b369c21-1d9d-4fdb-9dc8-280e5dee2b1b_1018x486.png" width="1018" height="486" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/1b369c21-1d9d-4fdb-9dc8-280e5dee2b1b_1018x486.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:486,&quot;width&quot;:1018,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!kOGf!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1b369c21-1d9d-4fdb-9dc8-280e5dee2b1b_1018x486.png 424w, /__u/substackcdn.com/image/fetch/$s_!kOGf!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1b369c21-1d9d-4fdb-9dc8-280e5dee2b1b_1018x486.png 848w, /__u/substackcdn.com/image/fetch/$s_!kOGf!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1b369c21-1d9d-4fdb-9dc8-280e5dee2b1b_1018x486.png 1272w, /__u/substackcdn.com/image/fetch/$s_!kOGf!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1b369c21-1d9d-4fdb-9dc8-280e5dee2b1b_1018x486.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">New question to Local Model</figcaption></figure></div><p>The model is outputting 5.7 tokens per second because I&#8217;m in battery saving mode. If I turn it down, we will probably see a value of 15&#8211;20 tokens per second.</p><div><hr></div><h4><strong>2. One-Shot Terminal Commands</strong></h4><p>To interact with your local model, you can also provide the question outside of the interactive mode:</p><pre><code>ollama run qwen3:8b &#8220;write a python script that tells me how many vowels a word has&#8221;</code></pre><p>Here&#8217;s the script that our local llm built:</p><pre><code>```python
# Prompt the user for a word
word = input(&#8221;Enter a word: &#8220;)

# Define the set of vowels
vowels = {&#8217;a&#8217;, &#8216;e&#8217;, &#8216;i&#8217;, &#8216;o&#8217;, &#8216;u&#8217;}

# Initialize a counter
count = 0

# Convert the word to lowercase and check each character
for char in word.lower():
    if char in vowels:
        count += 1

# Output the result
print(f&#8221;Number of vowels: {count}&#8221;)</code></pre><div><hr></div><h4><strong>3. The HTTP API (For Scripts and Apps)</strong></h4><p>Can you only use this within the terminal commands?</p><p>Of course not! If you are comfortable with Python, you can build any local script using your local model:</p><pre><code>import json, urllib.request

req = urllib.request.Request(
    &#8220;http://127.0.0.1:11434/api/generate&#8221;,
    data=json.dumps({
        &#8220;model&#8221;: &#8220;qwen3:8b&#8221;,
        &#8220;prompt&#8221;: &#8220;Give me three uses for a local LLM.&#8221;,
        &#8220;stream&#8221;: False,
        &#8220;think&#8221;: False,
    }).encode(),
    headers={&#8221;Content-Type&#8221;: &#8220;application/json&#8221;},
)
print(json.loads(urllib.request.urlopen(req).read())[&#8221;response&#8221;])</code></pre><p>Here is the answer from the model after running this Python script:</p><pre><code>Sure! Here are three common and practical uses for a **local LLM (Large Language Model)**:

1. **Personalized Assistance and Productivity**
A local LLM can act as a private AI assistant, helping with tasks like email drafting, scheduling, note-taking, and even coding. Since it runs locally, it maintains user privacy and doesn&#8217;t rely on internet connectivity.

2. **Content Creation and Language Processing**
You can use a local LLM to generate creative content such as blog posts, stories, scripts, or marketing copy. It can also assist with language translation, grammar checking, and summarizing text.

3. **Custom Applications and Integration**
A local LLM can be integrated into custom applications or workflows, such as chatbots, customer support systems, or data analysis tools. This allows for tailored solutions without exposing sensitive data to external servers.

Let me know if you&#8217;d like examples of how to implement these uses!</code></pre><p>Cool! You can now create your own applications with your own local model quite easily.</p><div><hr></div><h4>Fine-Tuning the Experience&#8202;&#8212;&#8202;Taming the &#8220;Thinking&#8221; Tokens</h4><p>Qwen 3 is a hybrid reasoning model. By default, it generates a verbose <code>&lt;think&gt;...&lt;/think&gt;</code> block outlining its chain of thought before providing the actual answer. Sometimes you want to see the math but most of the time, you just want the answer quickly (and cut some time from waiting the output tokens from the thinking process).</p><p>Here is how you bypass the reasoning pass:</p><ul><li><p><strong>Disable it entirely:</strong> <code>ollama run qwen3:8b --think=false</code></p></li><li><p><strong>Run it, but hide it from the UI:</strong> <code>ollama run qwen3:8b --hidethinking</code></p></li><li><p><strong>In scripts:</strong> Pass <code>"think": false</code> in your JSON payload.</p></li></ul><div><hr></div><h3>A Warning About Web Search</h3><p>Models are static up until their training data. That means that they can&#8217;t access data after they were trained, and companies have been relying on web search tools to augment the capability of the models. For example for our local model:</p><div class="captioned-image-container"><figure><a class="image-link image2" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!ba9j!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff6f4145f-4064-4a91-873c-c0ccfd0fb805_1080x102.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!ba9j!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff6f4145f-4064-4a91-873c-c0ccfd0fb805_1080x102.png 424w, /__u/substackcdn.com/image/fetch/$s_!ba9j!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff6f4145f-4064-4a91-873c-c0ccfd0fb805_1080x102.png 848w, /__u/substackcdn.com/image/fetch/$s_!ba9j!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff6f4145f-4064-4a91-873c-c0ccfd0fb805_1080x102.png 1272w, /__u/substackcdn.com/image/fetch/$s_!ba9j!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff6f4145f-4064-4a91-873c-c0ccfd0fb805_1080x102.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!ba9j!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff6f4145f-4064-4a91-873c-c0ccfd0fb805_1080x102.png" width="1080" height="102" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f6f4145f-4064-4a91-873c-c0ccfd0fb805_1080x102.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:102,&quot;width&quot;:1080,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!ba9j!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff6f4145f-4064-4a91-873c-c0ccfd0fb805_1080x102.png 424w, /__u/substackcdn.com/image/fetch/$s_!ba9j!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff6f4145f-4064-4a91-873c-c0ccfd0fb805_1080x102.png 848w, /__u/substackcdn.com/image/fetch/$s_!ba9j!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff6f4145f-4064-4a91-873c-c0ccfd0fb805_1080x102.png 1272w, /__u/substackcdn.com/image/fetch/$s_!ba9j!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff6f4145f-4064-4a91-873c-c0ccfd0fb805_1080x102.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></a><figcaption class="image-caption">Last day of training data of our Local Model</figcaption></figure></div><p>But, Ollama allows you to hand the model a web-search tool. This sounds incredible but there&#8217;s a catch.</p><p>The search itself executes on Ollama&#8217;s hosted cloud service. The moment you enable it, your prompts are being sent over the internet to fetch search results. The model stays local, but your queries do not. This may violate the principle of privacy you want to guarantee with the setup.</p><div><hr></div><h4>Bonus: VS Code Integration</h4><p>The ultimate endgame for me was getting an offline coding assistant. The cleanest, entirely free path for this is the <strong>Continue.dev</strong> extension.</p><ul><li><p>Install VS Code and the Continue extension.</p></li><li><p>Open Continue&#8217;s configuration file at <code>~/.continue/config.yaml</code>.</p></li><li><p>Point it at your local Ollama server:</p></li></ul><pre><code>name: Local Assistant
version: 1.0.0
models:
  - name: Qwen3 8B (local)
    provider: ollama
    model: qwen3:8b
    roles:
      - chat
      - edit
      - apply
  - name: Qwen3 8B Autocomplete
    provider: ollama
    model: qwen3:8b
    roles:
      - autocomplete</code></pre><p><strong>Pro-tip:</strong> An 8B model is slightly too heavy for the split-second latency you want for inline code autocomplete. I highly recommend pulling a smaller model specifically for that task (<code>ollama pull qwen2.5-coder:1.5b-base</code>), mapping it to the <code>autocomplete</code> role, and letting Qwen3 8B handle the heavier <code>chat</code> tasks.</p><div><hr></div><h4>What if I have a Windows Machine?</h4><p>As I&#8217;m not on a windows for this tutorial, I haven&#8217;t tried it extensively. But the good news is that the Ollama package is available for Windows computers <a href="https://ollama.com/download/windows">here</a>. The install process may differ a bit, but the logic behind using Ollama and pulling the models will be exactly the same.</p><div><hr></div><h4>Where This Leaves Me</h4><p>My total footprint for this project was 156 MB for the software and 5.2 GB for the model itself.</p><p>I now have a highly capable language model living permanently on my hard drive. For public, complex work, I will still reach for the cloud. But for the drafts I don&#8217;t want ingested into training data, the offline flights, and the legally bound client documents? This intelligence is now on my computer.</p><p>This may be a bit too techy for most people still, but things are becoming more democratized. And it&#8217;s not just about availability. On the performance front, open-source models are improving at a staggering pace, delivering results that make the future of local AI look incredibly promising. For example, <a href="https://ollama.com/library/glm-5.2">GLM 5.2</a> and <a href="https://qwen.ai/blog?id=qwen3.7">Qwen 3.7 Max</a> are catching up to the big labs&#8217; models performance:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!uW_h!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7414ffc5-f678-4ab3-bc45-4d94246c0f58_1600x934.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!uW_h!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7414ffc5-f678-4ab3-bc45-4d94246c0f58_1600x934.jpeg 424w, /__u/substackcdn.com/image/fetch/$s_!uW_h!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7414ffc5-f678-4ab3-bc45-4d94246c0f58_1600x934.jpeg 848w, /__u/substackcdn.com/image/fetch/$s_!uW_h!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7414ffc5-f678-4ab3-bc45-4d94246c0f58_1600x934.jpeg 1272w, /__u/substackcdn.com/image/fetch/$s_!uW_h!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7414ffc5-f678-4ab3-bc45-4d94246c0f58_1600x934.jpeg 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!uW_h!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7414ffc5-f678-4ab3-bc45-4d94246c0f58_1600x934.jpeg" width="1456" height="850" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/7414ffc5-f678-4ab3-bc45-4d94246c0f58_1600x934.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:850,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!uW_h!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7414ffc5-f678-4ab3-bc45-4d94246c0f58_1600x934.jpeg 424w, /__u/substackcdn.com/image/fetch/$s_!uW_h!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7414ffc5-f678-4ab3-bc45-4d94246c0f58_1600x934.jpeg 848w, /__u/substackcdn.com/image/fetch/$s_!uW_h!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7414ffc5-f678-4ab3-bc45-4d94246c0f58_1600x934.jpeg 1272w, /__u/substackcdn.com/image/fetch/$s_!uW_h!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7414ffc5-f678-4ab3-bc45-4d94246c0f58_1600x934.jpeg 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Comparison of Models performance on Software Engineering Benchmark</figcaption></figure></div><p>As the technical floor keeps dropping, &#8220;owning your own AI&#8221; is going to stop being a luxury reserved for developers with expensive laptops. That is the version of AI democratization I actually believe in.</p><p>Go give your laptop another brain this weekend and long live open source!</p>]]></content:encoded></item><item><title><![CDATA[Weekly AI Digest: Permission to Launch]]></title><description><![CDATA[Week 25-26, 2026]]></description><link>https://thedatajourney.substack.com/p/weekly-ai-digest-permission-to-launch</link><guid isPermaLink="false">https://thedatajourney.substack.com/p/weekly-ai-digest-permission-to-launch</guid><dc:creator><![CDATA[Ivo Bernardo]]></dc:creator><pubDate>Sat, 27 Jun 2026 21:10:10 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!kxn8!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb4eb6ac3-7c75-4f81-8589-8c8a6180e06a_2048x2048.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Good evening folks,</p><p>Even with all the AI out there, there's still work that has to be done by hand. A heavy stretch at the university and the company for me meant last week's digest never got written, so I'm combining weeks 25 and 26 into this one, sorry about that.</p><p>This week we ask the important question: <em>who actually gets to press the launch button?</em> Turns out the answer increasingly involves governments,</p><p>Grab your coffee and let&#8217;s unpack the last couple of week,</p><div><hr></div><h4>&#187; Stat of the Week: 46.4%</h4><p>For the first time since it shipped, ChatGPT&#8217;s share of the global AI-assistant market slipped below half, down to 46.4% by the end of May, per Sensor Tower&#8217;s State of AI report.&#185; </p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!kxn8!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb4eb6ac3-7c75-4f81-8589-8c8a6180e06a_2048x2048.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!kxn8!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb4eb6ac3-7c75-4f81-8589-8c8a6180e06a_2048x2048.png 424w, /__u/substackcdn.com/image/fetch/$s_!kxn8!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb4eb6ac3-7c75-4f81-8589-8c8a6180e06a_2048x2048.png 848w, /__u/substackcdn.com/image/fetch/$s_!kxn8!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb4eb6ac3-7c75-4f81-8589-8c8a6180e06a_2048x2048.png 1272w, /__u/substackcdn.com/image/fetch/$s_!kxn8!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb4eb6ac3-7c75-4f81-8589-8c8a6180e06a_2048x2048.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!kxn8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb4eb6ac3-7c75-4f81-8589-8c8a6180e06a_2048x2048.png" width="522" height="522" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/b4eb6ac3-7c75-4f81-8589-8c8a6180e06a_2048x2048.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1456,&quot;width&quot;:1456,&quot;resizeWidth&quot;:522,&quot;bytes&quot;:5290215,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://thedatajourney.substack.com/i/203880405?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb4eb6ac3-7c75-4f81-8589-8c8a6180e06a_2048x2048.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!kxn8!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb4eb6ac3-7c75-4f81-8589-8c8a6180e06a_2048x2048.png 424w, /__u/substackcdn.com/image/fetch/$s_!kxn8!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb4eb6ac3-7c75-4f81-8589-8c8a6180e06a_2048x2048.png 848w, /__u/substackcdn.com/image/fetch/$s_!kxn8!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb4eb6ac3-7c75-4f81-8589-8c8a6180e06a_2048x2048.png 1272w, /__u/substackcdn.com/image/fetch/$s_!kxn8!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb4eb6ac3-7c75-4f81-8589-8c8a6180e06a_2048x2048.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div><hr></div><h4>&#187; Industry Highlights: Market &amp; Product</h4><p><strong>OpenAI</strong> previewed the GPT-5.6 family this week, and with it a new naming scheme: Sol (flagship), Terra (the everyday workhorse), and Luna (fast and cheap).&#178; The company introduced a new <code>max</code> reasoning mode and an <code>ultra</code> mode that spins up subagents. Sol lands at $5 in / $30 out per million tokens. The numbers are impressive, but the rollout of the model is the real crux of it, as we will see below.</p><p>GPT-5.6 is as notable for how its safety was built as for what it scores.&#178; The company says it spent over 700,000 A100-equivalent GPU hours on automated red-teaming hunting for universal jailbreaks, and  baked the safeguards into the model&#8217;s behaviour rather than bolting a filter on top. </p><p><strong>Also, the company and Broadcom</strong> unveiled Jalape&#241;o, OpenAI&#8217;s first in-house inference chip, taped out from design to manufacturing in a claimed nine months.&#8308; It&#8217;s a blank-slate accelerator built specifically for LLM inference rather than a repurposed training part, and OpenAI says its own models helped design it.</p><p><strong>Salesforce</strong> is buying Fin for around $3.6 billion to bolt fast-deploy support agents onto Agentforce.&#179; The quieter prize is Apex, Fin&#8217;s own support-tuned model, which it claims beats frontier models from OpenAI and Anthropic on support resolution.</p><p><strong>On Google&#8217;s side, the company</strong> lost two more Gemini researchers to Anthropic this fortnight (Jonas Adler and Alexander Pritzel) the second pair to walk in a single week.&#8309; They follow Noam Shazeer (a <em>co-author of the original Transformer paper</em>, off to OpenAI) and Nobel laureate John Jumper of AlphaFold fame</p><div><hr></div><h4>&#187; Economic Highlights: The World of AI</h4><p>Relevant week on the regulation side.</p><p><strong>Anthropic</strong> spent two weeks as the cautionary tale of the cycle. After Commerce Secretary Howard Lutnick invoked <em>export control</em> authorities, Anthropic was forced to pull Mythos 5 and Fable 5 entirely, because foreign nationals (including its own employees) counted as a &#8220;deemed export.&#8221;&#8310; The reported origin of the panic: Amazon CEO Andy Jassy told government officials that Amazon researchers had used Fable 5 to surface information usable in cyberattacks. This Friday the government partially relented as Mythos 5 can now go to a vetted list of 100+ critical-infrastructure operators and Fortune 500 firms, no license required.&#8311; Fable, the consumer-facing model, stays dark for now.</p><p><strong>OpenAI</strong> then walked straight into the same machinery . Its GPT-5.6 preview is limited to &#8220;trusted partners whose participation has been shared with the government,&#8221; at the administration&#8217;s request.&#8312; The company complied and then, in the same breath, made plain it wasn&#8217;t thrilled, calling the arrangement a short-term step it doesn&#8217;t want to see become permanent. </p><p><strong>Meta</strong>, meanwhile, got the same lesson from the other superpower. It has begun unwinding its $2 billion acquisition of Manus, cutting the Chinese-founded startup off from internal systems, after Beijing ordered the deal reversed on national-security grounds.&#8313; Manus&#8217;s founders are reportedly raising ~$1bn to buy themselves back, possibly toward a Hong Kong listing.</p><div><hr></div><h4>&#187; Quote of the Week</h4><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!CPc5!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F21d78de6-b989-462c-b018-4902698c20cd_2528x1686.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!CPc5!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F21d78de6-b989-462c-b018-4902698c20cd_2528x1686.png 424w, /__u/substackcdn.com/image/fetch/$s_!CPc5!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F21d78de6-b989-462c-b018-4902698c20cd_2528x1686.png 848w, /__u/substackcdn.com/image/fetch/$s_!CPc5!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F21d78de6-b989-462c-b018-4902698c20cd_2528x1686.png 1272w, /__u/substackcdn.com/image/fetch/$s_!CPc5!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F21d78de6-b989-462c-b018-4902698c20cd_2528x1686.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!CPc5!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F21d78de6-b989-462c-b018-4902698c20cd_2528x1686.png" width="583" height="388.8001373626374" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/21d78de6-b989-462c-b018-4902698c20cd_2528x1686.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:583,&quot;bytes&quot;:6027382,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://thedatajourney.substack.com/i/203880405?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F21d78de6-b989-462c-b018-4902698c20cd_2528x1686.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!CPc5!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F21d78de6-b989-462c-b018-4902698c20cd_2528x1686.png 424w, /__u/substackcdn.com/image/fetch/$s_!CPc5!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F21d78de6-b989-462c-b018-4902698c20cd_2528x1686.png 848w, /__u/substackcdn.com/image/fetch/$s_!CPc5!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F21d78de6-b989-462c-b018-4902698c20cd_2528x1686.png 1272w, /__u/substackcdn.com/image/fetch/$s_!CPc5!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F21d78de6-b989-462c-b018-4902698c20cd_2528x1686.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h4>&#187; Closing</h4><p>In the past two weeks, governments on both sides have stepped in to block access to the technology. The US vs. China race keeps heating up.</p><p>The strongest models of this cycle got held back not by what they could do, but by who was allowed to use them, and that&#8217;s one of the most important shifts in the AI world. </p><p>These days the hard part isn't building AI technology. It's getting cleared to ship it.</p><p>&#8212; Ivo</p><div><hr></div><p>1 <a href="https://techcrunch.com/2026/06/16/chatgpts-market-share-slips-below-50-for-first-time/">https://techcrunch.com/2026/06/16/chatgpts-market-share-slips-below-50-for-first-time/</a></p><p>2 <a href="https://openai.com/index/previewing-gpt-5-6-sol/">https://openai.com/index/previewing-gpt-5-6-sol/</a></p><p>3 <a href="https://thenextweb.com/news/salesforce-acquires-fin-intercom-3-6-billion">https://thenextweb.com/news/salesforce-acquires-fin-intercom-3-6-billion</a></p><p>4 <a href="https://openai.com/index/openai-broadcom-jalapeno-inference-chip/">https://openai.com/index/openai-broadcom-jalapeno-inference-chip/</a></p><p>5 <a href="https://thenextweb.com/news/google-loses-two-gemini-researchers-anthropic">https://thenextweb.com/news/google-loses-two-gemini-researchers-anthropic</a></p><p>6 <a href="https://www.wsj.com/tech/ai/amazon-ceos-talks-with-u-s-officials-triggered-crackdown-on-anthropic-models-dcc90578">https://www.wsj.com/tech/ai/amazon-ceos-talks-with-u-s-officials-triggered-crackdown-on-anthropic-models-dcc90578</a></p><p>7 <a href="https://www.wired.com/story/anthropic-restores-access-to-mythos/">https://www.wired.com/story/anthropic-restores-access-to-mythos/</a></p><p>8 <a href="https://techcrunch.com/2026/06/26/openai-limits-gpt-5-6-rollout-after-government-request-says-restrictions-shouldnt-be-the-norm/">https://techcrunch.com/2026/06/26/openai-limits-gpt-5-6-rollout-after-government-request-says-restrictions-shouldnt-be-the-norm/</a></p><p>9 <a href="https://techcrunch.com/2026/06/13/meta-reportedly-moves-to-unwind-2b-manus-deal-after-beijings-demand/">https://techcrunch.com/2026/06/13/meta-reportedly-moves-to-unwind-2b-manus-deal-after-beijings-demand/</a></p>]]></content:encoded></item><item><title><![CDATA[The AI Handbook for The Curious Mind: Chapter 4.1.2]]></title><description><![CDATA[Meet John, the Decision Tree]]></description><link>https://thedatajourney.substack.com/p/the-ai-handbook-for-the-curious-mind-ab1</link><guid isPermaLink="false">https://thedatajourney.substack.com/p/the-ai-handbook-for-the-curious-mind-ab1</guid><dc:creator><![CDATA[Ivo Bernardo]]></dc:creator><pubDate>Wed, 17 Jun 2026 19:04:27 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!KBpd!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2698494d-d5d2-41a9-add4-673d8d0f8e40_700x863.jpeg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Hey everyone,</p><p>Some of you may know, but a couple of months ago I wrote a book. It&#8217;s called <strong>The AI Handbook for the Curious Mind</strong>, and it&#8217;s available on <a href="https://www.amazon.com/Handbook-Curious-Mind-Understanding-Intelligence-ebook/dp/B0G4KTP2QP/ref=tmm_kin_swatch_0">Amazon</a> or <a href="http://ivopbernardo.gumroad.com/l/aihandbookforthecuriousmind">Gumroad</a>. </p><p>Every two weeks, I&#8217;ll be releasing a full chapter here, exclusively for paid subscribers. I keep looking for ways to give back to the people who support this newsletter, and sharing the book felt like one of the more meaningful ones. &#128516;</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!KBpd!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2698494d-d5d2-41a9-add4-673d8d0f8e40_700x863.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!KBpd!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2698494d-d5d2-41a9-add4-673d8d0f8e40_700x863.jpeg 424w, /__u/substackcdn.com/image/fetch/$s_!KBpd!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2698494d-d5d2-41a9-add4-673d8d0f8e40_700x863.jpeg 848w, /__u/substackcdn.com/image/fetch/$s_!KBpd!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2698494d-d5d2-41a9-add4-673d8d0f8e40_700x863.jpeg 1272w, /__u/substackcdn.com/image/fetch/$s_!KBpd!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2698494d-d5d2-41a9-add4-673d8d0f8e40_700x863.jpeg 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!KBpd!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2698494d-d5d2-41a9-add4-673d8d0f8e40_700x863.jpeg" width="700" height="863" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2698494d-d5d2-41a9-add4-673d8d0f8e40_700x863.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:863,&quot;width&quot;:700,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="/__u/substackcdn.com/image/fetch/$s_!KBpd!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2698494d-d5d2-41a9-add4-673d8d0f8e40_700x863.jpeg 424w, /__u/substackcdn.com/image/fetch/$s_!KBpd!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2698494d-d5d2-41a9-add4-673d8d0f8e40_700x863.jpeg 848w, /__u/substackcdn.com/image/fetch/$s_!KBpd!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2698494d-d5d2-41a9-add4-673d8d0f8e40_700x863.jpeg 1272w, /__u/substackcdn.com/image/fetch/$s_!KBpd!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2698494d-d5d2-41a9-add4-673d8d0f8e40_700x863.jpeg 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Last time, we met <a href="/__u/thedatajourney.substack.com/p/the-ai-handbook-for-the-curious-mind-de1">Jack</a>, our linear regression model, and watched him draw a straight line through a mess of house prices and lot sizes. </p><p>But Jack has a problem as life isn&#8217;t always a straight line.</p><p>In this post, we meet the second machine learning model: <strong>John, the decision tree</strong>. Where Jack draws one line and calls it a day, John asks questions and lots of them. And it turns out, that&#8217;s sometimes exactly what you need to find patterns in data.</p>
      <p>
          <a href="/__u/thedatajourney.substack.com/p/the-ai-handbook-for-the-curious-mind-ab1">
              Read more
          </a>
      </p>
   ]]></content:encoded></item><item><title><![CDATA[Weekly AI Digest: Anthropic's Regulation Boomerang]]></title><description><![CDATA[Week 24, 2026]]></description><link>https://thedatajourney.substack.com/p/weekly-ai-digest-anthropics-regulation</link><guid isPermaLink="false">https://thedatajourney.substack.com/p/weekly-ai-digest-anthropics-regulation</guid><dc:creator><![CDATA[Ivo Bernardo]]></dc:creator><pubDate>Sat, 13 Jun 2026 10:00:22 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!h3uU!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffedd8598-d1cf-413b-a5b5-a3e88bd1a540_1542x710.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Hello everyone,</p><p>After spending years calling for stronger AI regulation, Anthropic got a taste of instant karma this week, the US government blocked access to Claude Fable 5 just three days after launch.</p><p>In a week with high doses of politics, we also witnessed the improbable: Trump and Bernie Sanders converging on the same idea about who should own frontier AI.</p><p>Grab your coffee and let&#8217;s dive into Week 24, because this one&#8217;s a good one!</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!h3uU!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffedd8598-d1cf-413b-a5b5-a3e88bd1a540_1542x710.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!h3uU!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffedd8598-d1cf-413b-a5b5-a3e88bd1a540_1542x710.png 424w, /__u/substackcdn.com/image/fetch/$s_!h3uU!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffedd8598-d1cf-413b-a5b5-a3e88bd1a540_1542x710.png 848w, /__u/substackcdn.com/image/fetch/$s_!h3uU!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffedd8598-d1cf-413b-a5b5-a3e88bd1a540_1542x710.png 1272w, /__u/substackcdn.com/image/fetch/$s_!h3uU!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffedd8598-d1cf-413b-a5b5-a3e88bd1a540_1542x710.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!h3uU!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffedd8598-d1cf-413b-a5b5-a3e88bd1a540_1542x710.png" width="1456" height="670" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/fedd8598-d1cf-413b-a5b5-a3e88bd1a540_1542x710.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:670,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:96098,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://thedatajourney.substack.com/i/201848482?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffedd8598-d1cf-413b-a5b5-a3e88bd1a540_1542x710.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="/__u/substackcdn.com/image/fetch/$s_!h3uU!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffedd8598-d1cf-413b-a5b5-a3e88bd1a540_1542x710.png 424w, /__u/substackcdn.com/image/fetch/$s_!h3uU!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffedd8598-d1cf-413b-a5b5-a3e88bd1a540_1542x710.png 848w, /__u/substackcdn.com/image/fetch/$s_!h3uU!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffedd8598-d1cf-413b-a5b5-a3e88bd1a540_1542x710.png 1272w, /__u/substackcdn.com/image/fetch/$s_!h3uU!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffedd8598-d1cf-413b-a5b5-a3e88bd1a540_1542x710.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div><hr></div><h4>&#187; Stat of the Week: 80% of Anthropic&#8217;s Own Code Written by Claude</h4><p>According to a new Anthropic paper on recursive self-improvement, Claude now authors roughly 80% of its production codebase. The paper also called for a global pause mechanism to be developed before that threshold becomes self-sustaining.&#185;</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!VTrE!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffe73d121-be11-44c8-ac3e-1c640459ad50_1254x1254.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!VTrE!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffe73d121-be11-44c8-ac3e-1c640459ad50_1254x1254.png 424w, /__u/substackcdn.com/image/fetch/$s_!VTrE!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffe73d121-be11-44c8-ac3e-1c640459ad50_1254x1254.png 848w, /__u/substackcdn.com/image/fetch/$s_!VTrE!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffe73d121-be11-44c8-ac3e-1c640459ad50_1254x1254.png 1272w, /__u/substackcdn.com/image/fetch/$s_!VTrE!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffe73d121-be11-44c8-ac3e-1c640459ad50_1254x1254.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!VTrE!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffe73d121-be11-44c8-ac3e-1c640459ad50_1254x1254.png" width="556" height="556" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/fe73d121-be11-44c8-ac3e-1c640459ad50_1254x1254.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1254,&quot;width&quot;:1254,&quot;resizeWidth&quot;:556,&quot;bytes&quot;:1604033,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://thedatajourney.substack.com/i/201848482?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffe73d121-be11-44c8-ac3e-1c640459ad50_1254x1254.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!VTrE!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffe73d121-be11-44c8-ac3e-1c640459ad50_1254x1254.png 424w, /__u/substackcdn.com/image/fetch/$s_!VTrE!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffe73d121-be11-44c8-ac3e-1c640459ad50_1254x1254.png 848w, /__u/substackcdn.com/image/fetch/$s_!VTrE!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffe73d121-be11-44c8-ac3e-1c640459ad50_1254x1254.png 1272w, /__u/substackcdn.com/image/fetch/$s_!VTrE!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffe73d121-be11-44c8-ac3e-1c640459ad50_1254x1254.png 1456w" sizes="100vw"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><em>Post-Note: three days later, the US government delivered a version of that pause</em></p><div><hr></div><h4>&#187; Industry Highlights: Market &amp; Product</h4><p><strong>Claude Fable 5</strong> launched on June 9th as Anthropic&#8217;s first publicly available Mythos-class model, designed for days-long agentic tasks, complex coding migrations, and multi-stage knowledge work that previous generations couldn&#8217;t sustain.&#178; </p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!QKAV!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa2f6289b-abc5-407c-adbe-bad406e871e9_2600x2870.webp" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!QKAV!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa2f6289b-abc5-407c-adbe-bad406e871e9_2600x2870.webp 424w, /__u/substackcdn.com/image/fetch/$s_!QKAV!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa2f6289b-abc5-407c-adbe-bad406e871e9_2600x2870.webp 848w, /__u/substackcdn.com/image/fetch/$s_!QKAV!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa2f6289b-abc5-407c-adbe-bad406e871e9_2600x2870.webp 1272w, /__u/substackcdn.com/image/fetch/$s_!QKAV!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa2f6289b-abc5-407c-adbe-bad406e871e9_2600x2870.webp 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!QKAV!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa2f6289b-abc5-407c-adbe-bad406e871e9_2600x2870.webp" width="1456" height="1607" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a2f6289b-abc5-407c-adbe-bad406e871e9_2600x2870.webp&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1607,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Benchmark table showing Claude Fable and Mythos compared to other leading models&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Benchmark table showing Claude Fable and Mythos compared to other leading models" title="Benchmark table showing Claude Fable and Mythos compared to other leading models" srcset="/__u/substackcdn.com/image/fetch/$s_!QKAV!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa2f6289b-abc5-407c-adbe-bad406e871e9_2600x2870.webp 424w, /__u/substackcdn.com/image/fetch/$s_!QKAV!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa2f6289b-abc5-407c-adbe-bad406e871e9_2600x2870.webp 848w, /__u/substackcdn.com/image/fetch/$s_!QKAV!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa2f6289b-abc5-407c-adbe-bad406e871e9_2600x2870.webp 1272w, /__u/substackcdn.com/image/fetch/$s_!QKAV!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa2f6289b-abc5-407c-adbe-bad406e871e9_2600x2870.webp 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Claude Fable Benchmark</figcaption></figure></div><p>Then came the first controversy. Buried in Fable 5&#8217;s 319-page system card was a detail that triggered immediate backlash from researchers: the model was silently degrading responses it classified as model-distillation attempts .&#179; </p><p>And if that was not enough, the US government blocked Anthropic for having it available for the citizens, that is, since the export control directive targeted all foreign nationals, including Anthropic's own employees. The Trump administration had reportedly tried to block the release entirely before launch and failed; the export control order was the next move.&#8308;</p><p><strong>Still on Fable, but related to Microsoft</strong>, one of Anthropic&#8217;s largest backers. The company quietly blocked its own employees from using Fable 5 internally, while simultaneously making the model available to external GitHub Copilot and Foundry customers.&#8325; The reason was yet another controversy on the model release: Fable 5&#8217;s new 30-day data retention requirement, introduced alongside the Mythos-class launch for safety monitoring purposes, conflicts with Microsoft&#8217;s internal Zero Data Retention standard that prohibits employee data from being stored by third-party AI providers. </p><p><strong>On the other side of town, OpenAI</strong> filed a confidential S-1 with the SEC on June 8th, and announced it preemptively, rather than waiting for the inevitable leak.&#8310; The statement added that timing is undetermined and that staying private longer remains attractive, since some things are easier without public-company constraints.</p><p><strong>Moving on to other enterprises, Jeff Bezos&#8217;s Prometheus</strong> raised $12 billion this week at a $41 billion valuation, targeting what the company describes as an &#8220;artificial general engineer for the physical world.&#8221;&#8311; The company is focused on heavy engineering automation and drug design, domains where physical-world constraints still outrun software-only AI. </p><p><strong>And Reid Hoffman</strong> left Microsoft&#8217;s board to go full-time with Manus, his AI drug discovery startup.&#8313; A decade on one of tech&#8217;s most valuable boards is not a casual exit, so Reid is probably starting to go founder mode on the company.</p><p><strong>Finally, and turning our attention to Europe, Mistral</strong> is rumored to be raising &#8364;3 billion at a &#8364;20 billion valuation, nearly double its Series C.&#8312; Europe&#8217;s flagship frontier model lab is trying to catch up with its US competitors.</p><div><hr></div><h4>&#187; Industry Highlights: Research &amp; Science</h4><p><strong>Anthropic&#8217;s recursive self-improvement paper</strong> goes further than the 80% headline. The paper maps a technically plausible path toward AI systems that can materially improve their own training pipelines, not in a theoretical sense, but concretely: models writing the code that trains the next version of themselves.&#185; The authors explicitly call for a global pause mechanism to be developed in advance of that capability threshold being crossed. </p><div><hr></div><h4>&#187; Economic Highlights: The World of AI</h4><p>The strangest political development of the week (possibly the year) is that Donald Trump and Bernie Sanders have arrived at roughly the same position on AI ownership. Trump suggested AI companies should be made &#8220;partners in the revolution&#8221; with the US government; Sanders has been pushing for partial public ownership of frontier AI labs.&#185;&#8304; They&#8217;re coming from completely different ideological directions but have converged on the same  question: should private companies control the infrastructure that may determine national futures? </p><p>It is possible to say the same thing but with different reasons &#8212; and the Trump-Sanders convergence on AI ownership is a textbook example. One is arguing from national competitiveness, the other from economic justice.</p><div><hr></div><p>This week&#8217;s throughline isn&#8217;t capability, it&#8217;s control. </p><p>Who controls the most powerful models? Who decides when they&#8217;re too dangerous to run? Who profits when they go public? Who gets to use them at work?</p><p> This was the most consequential week in AI politics we've seen in a while, and it's worth flagging that none of these questions got answered. The Fable 5 shutdown sets a precedent for export controls on commercial AI models. The Trump-Sanders convergence signals that government ownership of AI is no longer a fringe idea on either side of the aisle. </p><p>Watch this space as it will have enormous consequences on the future of the world in the coming years.</p><p>&#8212; Ivo</p><div><hr></div><p><strong>Footnotes</strong></p><p>1 <a href="https://thenextweb.com/news/anthropic-claude-recursive-self-improvement-code">https://thenextweb.com/news/anthropic-claude-recursive-self-improvement-code</a></p><p>2 <a href="https://www.anthropic.com/claude/fable">https://www.anthropic.com/claude/fable</a></p><p>3 <a href="https://www.theverge.com/ai-artificial-intelligence/948280/anthropic-claude-fable-invisible-distillation-guardrail">https://www.theverge.com/ai-artificial-intelligence/948280/anthropic-claude-fable-invisible-distillation-guardrail</a> <a href="https://opentools.ai/news/anthropic-claude-fable-5-secret-guardrails-apology-backlash-2026">https://opentools.ai/news/anthropic-claude-fable-5-secret-guardrails-apology-backlash-2026</a> <a href="https://gizmodo.com/anthropic-apologizes-for-one-of-the-guardrails-on-its-fable-5-model-and-will-change-it-2000770365">https://gizmodo.com/anthropic-apologizes-for-one-of-the-guardrails-on-its-fable-5-model-and-will-change-it-2000770365</a></p><p>4 <a href="https://thenextweb.com/news/anthropic-fable-mythos-us-government-suspension">https://thenextweb.com/news/anthropic-fable-mythos-us-government-suspension</a> <a href="https://www.anthropic.com/news/fable-mythos-access">https://www.anthropic.com/news/fable-mythos-access</a></p><p>5 <a href="https://www.theverge.com/report/947575/microsoft-claude-fable-5-restricted-internally">https://www.theverge.com/report/947575/microsoft-claude-fable-5-restricted-internally</a> <a href="https://www.windowscentral.com/artificial-intelligence/microsoft-temporarily-ban-employees-from-using-claude-fable-5-ai">https://www.windowscentral.com/artificial-intelligence/microsoft-temporarily-ban-employees-from-using-claude-fable-5-ai</a> <a href="https://www.techradar.com/pro/microsoft-limits-employee-use-of-claude-fable-5-over-data-retention-concerns">https://www.techradar.com/pro/microsoft-limits-employee-use-of-claude-fable-5-over-data-retention-concerns</a></p><p>6 <a href="https://openai.com/index/openai-submits-confidential-s-1/">https://openai.com/index/openai-submits-confidential-s-1/</a></p><p>7 <a href="https://techcrunch.com/2026/06/11/jeff-bezoss-prometheus-raises-12b-to-build-an-artificial-general-engineer-for-the-physical-world/">https://techcrunch.com/2026/06/11/jeff-bezoss-prometheus-raises-12b-to-build-an-artificial-general-engineer-for-the-physical-world/</a></p><p>8 <a href="https://techcrunch.com/2026/06/12/mistral-is-rumored-to-be-raising-e3b-at-e20-valuation/">https://techcrunch.com/2026/06/12/mistral-is-rumored-to-be-raising-e3b-at-e20-valuation/</a></p><p>9 <a href="https://techcrunch.com/2026/06/05/reid-hoffman-is-leaving-microsofts-board-to-go-founder-mode-with-startup-manus/">https://techcrunch.com/2026/06/05/reid-hoffman-is-leaving-microsofts-board-to-go-founder-mode-with-startup-manus/</a></p><p>10 <a href="https://fortune.com/2026/06/05/trump-partnership-openai-anthropic-xai-nationalization-bernie-sanders-altman/">https://fortune.com/2026/06/05/trump-partnership-openai-anthropic-xai-nationalization-bernie-sanders-altman/</a> <a href="https://www.readtangle.com/government-ownership-ai-companies-bernie-sanders-trump/">https://www.readtangle.com/government-ownership-ai-companies-bernie-sanders-trump/</a></p><p></p><p></p>]]></content:encoded></item><item><title><![CDATA[Weekly AI Digest: Computex 2026, Anthropic IPO and more]]></title><description><![CDATA[Week 23, 2026]]></description><link>https://thedatajourney.substack.com/p/weekly-ai-digest-computex-2026-anthropic</link><guid isPermaLink="false">https://thedatajourney.substack.com/p/weekly-ai-digest-computex-2026-anthropic</guid><dc:creator><![CDATA[Ivo Bernardo]]></dc:creator><pubDate>Fri, 05 Jun 2026 10:58:23 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!KqB6!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbf00ad36-697b-48b7-bef4-579ad80de492_1199x559.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Hi there,</p><p>Anthropic filed for what could be one of the largest IPOs in history, while Florida decided it&#8217;s had enough of OpenAI. Uber discovered that telling engineers to use AI &#8220;as much as possible&#8221; has consequences in the P&amp;L and Meta has been hacked big time through their chatbot.</p><p>Grab your coffee and let&#8217;s dive into week 23.</p><div><hr></div><h4>&#187; Stat of the Week: $965 Billion</h4><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!7iXk!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff06ae5a7-0c8e-4cc3-b1f0-fd312dd32216_2048x2048.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!7iXk!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff06ae5a7-0c8e-4cc3-b1f0-fd312dd32216_2048x2048.png 424w, /__u/substackcdn.com/image/fetch/$s_!7iXk!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff06ae5a7-0c8e-4cc3-b1f0-fd312dd32216_2048x2048.png 848w, /__u/substackcdn.com/image/fetch/$s_!7iXk!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff06ae5a7-0c8e-4cc3-b1f0-fd312dd32216_2048x2048.png 1272w, /__u/substackcdn.com/image/fetch/$s_!7iXk!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff06ae5a7-0c8e-4cc3-b1f0-fd312dd32216_2048x2048.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!7iXk!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff06ae5a7-0c8e-4cc3-b1f0-fd312dd32216_2048x2048.png" width="466" height="466" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f06ae5a7-0c8e-4cc3-b1f0-fd312dd32216_2048x2048.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1456,&quot;width&quot;:1456,&quot;resizeWidth&quot;:466,&quot;bytes&quot;:5284318,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://thedatajourney.substack.com/i/200741635?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff06ae5a7-0c8e-4cc3-b1f0-fd312dd32216_2048x2048.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!7iXk!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff06ae5a7-0c8e-4cc3-b1f0-fd312dd32216_2048x2048.png 424w, /__u/substackcdn.com/image/fetch/$s_!7iXk!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff06ae5a7-0c8e-4cc3-b1f0-fd312dd32216_2048x2048.png 848w, /__u/substackcdn.com/image/fetch/$s_!7iXk!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff06ae5a7-0c8e-4cc3-b1f0-fd312dd32216_2048x2048.png 1272w, /__u/substackcdn.com/image/fetch/$s_!7iXk!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff06ae5a7-0c8e-4cc3-b1f0-fd312dd32216_2048x2048.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div><hr></div><h4>&#187; Industry Highlights: Market &amp; Product</h4><p><strong>Anthropic</strong> filed a draft registration statement with the SEC on June 1, setting the stage for an IPO that could land as early as October and raise up to $60 billion.&#185; Days later, the company tapped Morgan Stanley, Goldman Sachs, and JPMorgan Chase as lead underwriters for an IPO.&#178; At just under a trillion dollars, Anthropic is entering public markets at a valuation that would make most sovereign wealth funds blush. The timing is notable: SpaceX is set to list in June, OpenAI has its own IPO ambitions, and whoever moves first sets the benchmark for everything that follows in the AI sector.</p><p><strong>On the infra side, NVIDIA</strong> used Computex 2026 in Taipei to announce that its Vera CPU, is in full production, with Anthropic, OpenAI, SpaceX, Oracle, and ByteDance already in the early adopter lineup.&#179; Jensen Huang also unveiled RTX Spark, a new Arm-based superchip for Windows PCs that combines a Grace CPU and Blackwell GPU in a single package, with Dell, HP, Microsoft, Lenovo, ASUS, and MSI building systems around it.&#8308; NVIDIA has spent years owning the GPU market for training; it&#8217;s now explicitly targeting inference, agents, and the laptop on your desk. Intel and Qualcomm should be paying close attention.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!KqB6!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbf00ad36-697b-48b7-bef4-579ad80de492_1199x559.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!KqB6!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbf00ad36-697b-48b7-bef4-579ad80de492_1199x559.png 424w, /__u/substackcdn.com/image/fetch/$s_!KqB6!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbf00ad36-697b-48b7-bef4-579ad80de492_1199x559.png 848w, /__u/substackcdn.com/image/fetch/$s_!KqB6!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbf00ad36-697b-48b7-bef4-579ad80de492_1199x559.png 1272w, /__u/substackcdn.com/image/fetch/$s_!KqB6!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbf00ad36-697b-48b7-bef4-579ad80de492_1199x559.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!KqB6!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbf00ad36-697b-48b7-bef4-579ad80de492_1199x559.png" width="1199" height="559" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/bf00ad36-697b-48b7-bef4-579ad80de492_1199x559.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:559,&quot;width&quot;:1199,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:385560,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://thedatajourney.substack.com/i/200741635?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbf00ad36-697b-48b7-bef4-579ad80de492_1199x559.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!KqB6!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbf00ad36-697b-48b7-bef4-579ad80de492_1199x559.png 424w, /__u/substackcdn.com/image/fetch/$s_!KqB6!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbf00ad36-697b-48b7-bef4-579ad80de492_1199x559.png 848w, /__u/substackcdn.com/image/fetch/$s_!KqB6!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbf00ad36-697b-48b7-bef4-579ad80de492_1199x559.png 1272w, /__u/substackcdn.com/image/fetch/$s_!KqB6!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbf00ad36-697b-48b7-bef4-579ad80de492_1199x559.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><strong>Brian Chesky</strong>, Airbnb&#8217;s CEO and co-founder, is reportedly planning to launch his own AI lab while remaining at the helm of Airbnb.&#8309; He won&#8217;t lead the lab directly, but will back it as a founding supporter. </p><p><strong>Meta&#8217;s AI chatbot</strong> became a security liability this week after it emerged that hackers had successfully hijacked high-profile Instagram accounts simply by asking Meta&#8217;s AI support chatbot to change account email addresses, no identity verification required.&#8310; Meta says the flaw is fixed, but attacks reportedly continued after the company&#8217;s announcement. </p><p><strong>Uber</strong> has capped employee spending on agentic coding tools at $1,500 per month per tool, after burning through its entire 2026 AI coding budget by April.&#8311; Some engineers had been generating monthly bills between $500 and $2,000 in token consumption.</p><div><hr></div><h4>&#187; Industry Highlights: Research &amp; Science</h4><p><strong>NVIDIA&#8217;s RTX Spark</strong> deserves a closer look beyond the headline. The chip combines Blackwell GPU and Grace CPU architectures via NVLink-C2C in a laptop-class form factor, targeting up to one petaflop of AI compute with 128GB unified memory.&#8308; NVIDIA is betting that personal computing will be redefined around AI inference the same way the smartphone redefined it around always-on connectivity.</p><div><hr></div><h4>&#187; Economic Highlights: The World of AI</h4><p><strong>Florida</strong> became the first US state to sue OpenAI and CEO Sam Altman, in an 83-page complaint alleging the company marketed ChatGPT to children while concealing serious risks, disregarding safety warnings, and prioritising winning the AI arms race over user wellbeing.&#185; </p><p><strong>On safety and trust, President Trump</strong> signed a scaled-back executive order on AI on June 2, establishing a voluntary framework in which AI companies can submit frontier models for government review up to 30 days before release.&#8313; A broader version had been shelved in May after industry pushback. The final order also directs agencies to create AI cybersecurity benchmarks and an AI cybersecurity clearinghouse. The word &#8220;voluntary&#8221; is doing a lot of heavy lifting here. </p><p><strong>Lastly, GitLab</strong> announced it is cutting approximately 350 employees (14% of its workforce) and withdrawing from 22 countries, redirecting savings into what CEO Bill Staples called &#8220;GitLab Act 2&#8221;: a generational rebuild of its infrastructure to handle 100x growth in agentic AI workloads.&#8312; The company reported Q1 FY2027 revenue of $264 million, up 23% year-over-year. It joins a long list of tech companies posting record revenues while simultaneously shrinking headcount, with AI cited as both the reason for the growth and the justification for the cuts.</p><div><hr></div><h4>&#187; Quote of the Week</h4><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!9vNl!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff1c8e09f-67cb-4870-a72b-91fbcacce58b_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!9vNl!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff1c8e09f-67cb-4870-a72b-91fbcacce58b_1536x1024.png 424w, /__u/substackcdn.com/image/fetch/$s_!9vNl!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff1c8e09f-67cb-4870-a72b-91fbcacce58b_1536x1024.png 848w, /__u/substackcdn.com/image/fetch/$s_!9vNl!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff1c8e09f-67cb-4870-a72b-91fbcacce58b_1536x1024.png 1272w, /__u/substackcdn.com/image/fetch/$s_!9vNl!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff1c8e09f-67cb-4870-a72b-91fbcacce58b_1536x1024.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!9vNl!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff1c8e09f-67cb-4870-a72b-91fbcacce58b_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f1c8e09f-67cb-4870-a72b-91fbcacce58b_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:2161035,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://thedatajourney.substack.com/i/200741635?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff1c8e09f-67cb-4870-a72b-91fbcacce58b_1536x1024.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="/__u/substackcdn.com/image/fetch/$s_!9vNl!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff1c8e09f-67cb-4870-a72b-91fbcacce58b_1536x1024.png 424w, /__u/substackcdn.com/image/fetch/$s_!9vNl!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff1c8e09f-67cb-4870-a72b-91fbcacce58b_1536x1024.png 848w, /__u/substackcdn.com/image/fetch/$s_!9vNl!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff1c8e09f-67cb-4870-a72b-91fbcacce58b_1536x1024.png 1272w, /__u/substackcdn.com/image/fetch/$s_!9vNl!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff1c8e09f-67cb-4870-a72b-91fbcacce58b_1536x1024.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p></p><h4>&#187; Closing</h4><p>Anthropic is heading to public markets at nearly a trillion dollars, while governments is filing 83-page lawsuits, Meta handed hackers the keys to Instagram, and Uber just put a cap on AI models spending in the company. </p><p>Another crazy week for the AI world, so stay tuned for more. The IPO roadshows are being scheduled and they will set the tone for the next year in the technology world.</p><p>&#8212; Ivo</p><div><hr></div><p><strong>Footnotes</strong></p><p>1 <a href="https://techcrunch.com/2026/06/01/florida-sues-openai-sam-altman-in-first-of-its-kind-lawsuit-over-violent-incidents/">https://techcrunch.com/2026/06/01/florida-sues-openai-sam-altman-in-first-of-its-kind-lawsuit-over-violent-incidents/</a> <a href="https://entrepreneur.com/business-news/anthropic-just-filed-to-go-public-it-could-be-one-of-the-biggest-ipos-ever">https://entrepreneur.com/business-news/anthropic-just-filed-to-go-public-it-could-be-one-of-the-biggest-ipos-ever</a></p><p>2 <a href="https://finance.yahoo.com/markets/stocks/articles/anthropic-said-pick-morgan-stanley-175802678.html">https://finance.yahoo.com/markets/stocks/articles/anthropic-said-pick-morgan-stanley-175802678.html</a></p><p>3 <a href="https://nvidianews.nvidia.com/news/nvidia-unveils-vera-the-cpu-for-agents">https://nvidianews.nvidia.com/news/nvidia-unveils-vera-the-cpu-for-agents</a> <a href="https://thenextweb.com/news/nvidia-vera-chip-anthropic-openai">https://thenextweb.com/news/nvidia-vera-chip-anthropic-openai</a></p><p>4 <a href="https://www.nytimes.com/2026/06/01/technology/nvidia-chips-personal-computers.html">https://www.nytimes.com/2026/06/01/technology/nvidia-chips-personal-computers.html</a></p><p>5 <a href="https://fortune.com/2026/06/04/airbnb-ceo-brian-chesky-plans-to-start-a-new-ai-company/">https://fortune.com/2026/06/04/airbnb-ceo-brian-chesky-plans-to-start-a-new-ai-company/</a></p><p>6 <a href="https://thenextweb.com/news/hackers-asked-metas-ai-chatbot-to-hand-over-instagram-accounts-and-it-did">https://thenextweb.com/news/hackers-asked-metas-ai-chatbot-to-hand-over-instagram-accounts-and-it-did</a></p><p>7 <a href="https://techcrunch.com/2026/06/02/uber-caps-employee-ai-spending-after-blowing-through-budget-in-four-months/">https://techcrunch.com/2026/06/02/uber-caps-employee-ai-spending-after-blowing-through-budget-in-four-months/</a></p><p>8 <a href="https://techcrunch.com/2026/06/03/gitlab-cuts-14-of-staff-as-it-scales-its-platform-to-serve-ai-workloads/">https://techcrunch.com/2026/06/03/gitlab-cuts-14-of-staff-as-it-scales-its-platform-to-serve-ai-workloads/</a></p><p>9 <a href="https://www.politico.com/news/2026/06/02/trump-signs-downsized-ai-order-00946389">https://www.politico.com/news/2026/06/02/trump-signs-downsized-ai-order-00946389</a></p>]]></content:encoded></item><item><title><![CDATA[Building My Own Personal AI Assistant: A Chronicle, Part 4]]></title><description><![CDATA[Fern&#227;o Looked at My Stock & ETF Portfolio. It Was Not Impressed.]]></description><link>https://thedatajourney.substack.com/p/building-my-own-personal-ai-assistant-5f2</link><guid isPermaLink="false">https://thedatajourney.substack.com/p/building-my-own-personal-ai-assistant-5f2</guid><dc:creator><![CDATA[Ivo Bernardo]]></dc:creator><pubDate>Wed, 03 Jun 2026 20:01:44 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!EVek!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fadc76253-e026-4274-b0f5-178d39af8bb0_1400x609.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div><hr></div><p>I have a bad habit of building things that give me unwelcome news about myself. Stay tuned, as that&#8217;s what the new <a href="https://ivopbernardo.medium.com/building-a-personal-ai-agent-in-a-couple-of-hours-7bea912f9a12">Fern&#227;o</a> module will be.</p><p>If you&#8217;re new here: I&#8217;ve been building Fern&#227;o, my own personal AI assistant. A very sparse stitched together app with different tabs, each one handling a slice of my life. Part 1 was a <a href="https://ivopbernardo.medium.com/building-a-personal-ai-agent-in-a-couple-of-hours-7bea912f9a12">schedule organizer</a>. Part 2, a <a href="https://ivopbernardo.medium.com/building-my-personal-ai-assistant-a-chronicle-part-2-8cd08088f255">task breaker</a>. Part 3, a <a href="https://ivopbernardo.medium.com/building-my-own-personal-ai-assistant-a-chronicle-part-3-92c32e896b5d">writing editor</a> (which is, somewhat ironically, helping me write this very post).</p><p>And now Part 4: a portfolio analyzer that looked me in the eyes and told me I was doing it wrong.</p><p>Time to meet my nemesis&#8202;&#8212;&#8202;meaning, me as a stock picker.</p><div><hr></div><h4>First, Some Context On Why I Even Needed This</h4><p>I&#8217;ve been investing for 5 years. Nothing crazy as I&#8217;m not day trading crypto at 3am or anything. I hold a mix of ETFs, individual stocks, some bonds, a couple of REITs.</p><p>At the same time, I like to read about companies, I follow earnings and I have opinions on things like competitive moats and management quality and total addressable markets.</p><p>I was, in other words, doing what most retail investors think they can do: putting in a reasonable amount of effort and assuming it was worth something&#8202;&#8212;&#8202;even if it was just for learning experience.</p><p><em>(And yes, I&#8217;ve read all the warnings. &#8220;Most retail investors underperform the market.&#8221; Moreover, I have a statistics masters. I knew the odds. I just needed to prove my own stupidity through experience. </em>&#128556;<em>)</em></p><p>When you start to invest, it&#8217;s difficult to have a free app that can tell you exactly your cost basis and performance. You can look at your portfolio on a random Tuesday, see that it&#8217;s up overall, and conclude that things are going fine, but there aren&#8217;t many tools that give you a holistic view of how you are performing through time and against many benchmarks (Y<em>ahoo finance gives a somewhat overview of that, but still limited for investors in Euro</em>).</p><p>I wanted to check my performance over time and figure out what to do differently. So, having Fern&#227;o right here in my computer, I asked for help.</p><p><em>Spoiler alert: my investment decisions are not yielding extraordinary results.</em></p><div><hr></div><h3>Feeding Fern&#227;o: From Spreadsheets to Something Useful</h3><p>The data source is an Excel file where I log every trade I do with date, ticker, buy or sell, quantity, price. I&#8217;ve been logging it since I started investing, which is the one thing I did right from the beginning.</p><p>Consistency in data entry is unsexy but it&#8217;s what makes a tool like this possible. If you&#8217;re not tracking your trades, the first step isn&#8217;t building the analyzer, it&#8217;s spending a painful afternoon reconstructing your history from brokerage statements (not fun!)</p><p>From that file, Fern&#227;o builds a few different views of what I own.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!EVek!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fadc76253-e026-4274-b0f5-178d39af8bb0_1400x609.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!EVek!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fadc76253-e026-4274-b0f5-178d39af8bb0_1400x609.png 424w, /__u/substackcdn.com/image/fetch/$s_!EVek!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fadc76253-e026-4274-b0f5-178d39af8bb0_1400x609.png 848w, /__u/substackcdn.com/image/fetch/$s_!EVek!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fadc76253-e026-4274-b0f5-178d39af8bb0_1400x609.png 1272w, /__u/substackcdn.com/image/fetch/$s_!EVek!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fadc76253-e026-4274-b0f5-178d39af8bb0_1400x609.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!EVek!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fadc76253-e026-4274-b0f5-178d39af8bb0_1400x609.png" width="1400" height="609" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/adc76253-e026-4274-b0f5-178d39af8bb0_1400x609.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:609,&quot;width&quot;:1400,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!EVek!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fadc76253-e026-4274-b0f5-178d39af8bb0_1400x609.png 424w, /__u/substackcdn.com/image/fetch/$s_!EVek!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fadc76253-e026-4274-b0f5-178d39af8bb0_1400x609.png 848w, /__u/substackcdn.com/image/fetch/$s_!EVek!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fadc76253-e026-4274-b0f5-178d39af8bb0_1400x609.png 1272w, /__u/substackcdn.com/image/fetch/$s_!EVek!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fadc76253-e026-4274-b0f5-178d39af8bb0_1400x609.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><em>(I really need to improve the UI on this one. The emojification of front-ends by Claude is giving me the itch)</em></p><p><em>(Note: the values, except percentages, that I&#8217;ll show in this post are fictional or hidden, for privacy reasons)</em>.</p><p>The sector breakdown is the first reality check. It shows me where my capital lives. I had sectors I considered &#8220;small positions&#8221; that had grown, and sectors I thought were core holdings that had been diluted down as I bought other stuff. Seeing it as a pie chart is not the most <em>data-viz</em> recommended way, but they get the job done.</p><p>In the screenshot above, we can see that I have 105 positions open (some of these positions are repeated across brokers as I try to diversify across 3 different banks here in Portugal. Also, I have other assets such as real estate or Certificates of Deposit&#8202;&#8212;&#8202;the real unique positions should be around 50 to 60).</p><p>Also, we see the division by three groups:</p><ul><li><p><em>Sector</em>: It&#8217;s not industry of the stock, but tells me the portion of the economy I&#8217;m investing in (World Stocks, US Stocks, Gold, etc.). In hindsight should have called this Asset Type.</p></li><li><p><em>Type</em>: If it&#8217;s an ETF, Individual Stocks, Real Estate, etc.</p></li><li><p><em>Market</em>: If it&#8217;s negotiated in the Nasdaq, Amsterdam, Deutsche Borse, etc.</p></li></ul><p>Nevertheless, I immediately see that I have too many scattered positions. One of the things that I&#8217;ll be aiming to do in the next few months is to simplify my portfolio a lot, and probably reduce the positions to around 25/30 (taking into account duplication).<strong> As we will see next, I am suffering from </strong><em><strong>diworsefication</strong></em><strong> (where you diversify too much that your portfolio ends up returning less than the overall market).</strong></p><p>This overview section is what it is&#8202;&#8212;&#8202;context, structure, a picture of the thing. Looking at it, we already see that there are too many positions open and that I should try to simplify in the future.</p><p>But, the interesting part is what comes next.</p><div><hr></div><h4>The Analysis Tab</h4><p>The &#8220;<em>Analysis</em>&#8221; tab is where Fern&#227;o really showed me the important details.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!2hP0!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0cec61b4-5769-448d-b4b1-b86026dc0c7a_1600x685.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!2hP0!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0cec61b4-5769-448d-b4b1-b86026dc0c7a_1600x685.png 424w, /__u/substackcdn.com/image/fetch/$s_!2hP0!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0cec61b4-5769-448d-b4b1-b86026dc0c7a_1600x685.png 848w, /__u/substackcdn.com/image/fetch/$s_!2hP0!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0cec61b4-5769-448d-b4b1-b86026dc0c7a_1600x685.png 1272w, /__u/substackcdn.com/image/fetch/$s_!2hP0!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0cec61b4-5769-448d-b4b1-b86026dc0c7a_1600x685.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!2hP0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0cec61b4-5769-448d-b4b1-b86026dc0c7a_1600x685.png" width="1456" height="623" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/0cec61b4-5769-448d-b4b1-b86026dc0c7a_1600x685.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:623,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!2hP0!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0cec61b4-5769-448d-b4b1-b86026dc0c7a_1600x685.png 424w, /__u/substackcdn.com/image/fetch/$s_!2hP0!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0cec61b4-5769-448d-b4b1-b86026dc0c7a_1600x685.png 848w, /__u/substackcdn.com/image/fetch/$s_!2hP0!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0cec61b4-5769-448d-b4b1-b86026dc0c7a_1600x685.png 1272w, /__u/substackcdn.com/image/fetch/$s_!2hP0!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0cec61b4-5769-448d-b4b1-b86026dc0c7a_1600x685.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Fern&#227;o Analysis Tab</figcaption></figure></div><p>The first thing I asked for was a historical comparison between my portfolio and the S&amp;P 500 , on a Euro basis. This part matters a lot, because I&#8217;m based in Portugal. Most investors benchmark themselves against the dollar-denominated S&amp;P 500 when they don&#8217;t actually live in dollars.</p><p>The only benchmark that&#8217;s real for me is the one in the currency I actually spend (Euro), so let&#8217;s analyze it:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!E0JJ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fce514a9b-4135-4318-9cc1-7680f2a56a34_1600x696.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!E0JJ!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fce514a9b-4135-4318-9cc1-7680f2a56a34_1600x696.png 424w, /__u/substackcdn.com/image/fetch/$s_!E0JJ!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fce514a9b-4135-4318-9cc1-7680f2a56a34_1600x696.png 848w, /__u/substackcdn.com/image/fetch/$s_!E0JJ!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fce514a9b-4135-4318-9cc1-7680f2a56a34_1600x696.png 1272w, /__u/substackcdn.com/image/fetch/$s_!E0JJ!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fce514a9b-4135-4318-9cc1-7680f2a56a34_1600x696.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!E0JJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fce514a9b-4135-4318-9cc1-7680f2a56a34_1600x696.png" width="1456" height="633" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/ce514a9b-4135-4318-9cc1-7680f2a56a34_1600x696.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:633,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!E0JJ!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fce514a9b-4135-4318-9cc1-7680f2a56a34_1600x696.png 424w, /__u/substackcdn.com/image/fetch/$s_!E0JJ!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fce514a9b-4135-4318-9cc1-7680f2a56a34_1600x696.png 848w, /__u/substackcdn.com/image/fetch/$s_!E0JJ!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fce514a9b-4135-4318-9cc1-7680f2a56a34_1600x696.png 1272w, /__u/substackcdn.com/image/fetch/$s_!E0JJ!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fce514a9b-4135-4318-9cc1-7680f2a56a34_1600x696.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">My Portfolio (in Green) vs. S&amp;P 500 (in grey)</figcaption></figure></div><p>This graph was the slap in the face.</p><p>Up until 2024, I was keeping pace, not beating the index, but close enough. Then the gap opened. My best theory: the stocks I like (blue chips, mature businesses, the boring stuff) aren&#8217;t where the market is putting its money right now. AI has a type, and it&#8217;s not them. I think it may revert, and I still believe in some of the stocks I bought.</p><p>But what bothers me more than the recent gap is the years before it&#8202;&#8212;&#8202;was where I was on par with the index. From 2022 to 2024, I was spending a lot of time choosing stocks. Hours of research, company analysis, reading filings. And the outcome was: about the same as doing nothing.</p><p>This isn&#8217;t a new story. Warren Buffett literally bet a million dollars on it&#8202;&#8212;&#8202;a public wager that a simple S&amp;P 500 index fund would outperform a handpicked basket of hedge funds over ten years. The hedge funds lost. If the smartest, most resourced active managers in the world can&#8217;t consistently beat the market, what exactly was I doing thinking I could from my laptop on a Sunday afternoon?</p><p><em>That said&#8202;&#8212;&#8202;I&#8217;d do it again. Not because it was profitable, but because I needed to find out for myself. Some lessons only land when they&#8217;re yours! This one&#8217;s mine now, properly. There&#8217;s so much growth in making mistakes and learning quickly!</em></p><div><hr></div><p>There&#8217;s this concept in investing called <em>alpha</em>, which is the excess return you generate compared to the benchmark. Positive alpha means you&#8217;re adding value with your stock selection. Negative alpha means you&#8217;d have been better off buying an ETF and going to the beach.</p><p>My alpha was negative.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!cdzM!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7dff6e70-b176-4e88-9c7d-85ccc482dbeb_1600x574.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!cdzM!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7dff6e70-b176-4e88-9c7d-85ccc482dbeb_1600x574.png 424w, /__u/substackcdn.com/image/fetch/$s_!cdzM!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7dff6e70-b176-4e88-9c7d-85ccc482dbeb_1600x574.png 848w, /__u/substackcdn.com/image/fetch/$s_!cdzM!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7dff6e70-b176-4e88-9c7d-85ccc482dbeb_1600x574.png 1272w, /__u/substackcdn.com/image/fetch/$s_!cdzM!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7dff6e70-b176-4e88-9c7d-85ccc482dbeb_1600x574.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!cdzM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7dff6e70-b176-4e88-9c7d-85ccc482dbeb_1600x574.png" width="1456" height="522" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/7dff6e70-b176-4e88-9c7d-85ccc482dbeb_1600x574.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:522,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!cdzM!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7dff6e70-b176-4e88-9c7d-85ccc482dbeb_1600x574.png 424w, /__u/substackcdn.com/image/fetch/$s_!cdzM!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7dff6e70-b176-4e88-9c7d-85ccc482dbeb_1600x574.png 848w, /__u/substackcdn.com/image/fetch/$s_!cdzM!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7dff6e70-b176-4e88-9c7d-85ccc482dbeb_1600x574.png 1272w, /__u/substackcdn.com/image/fetch/$s_!cdzM!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7dff6e70-b176-4e88-9c7d-85ccc482dbeb_1600x574.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">My portfolio vs. SP500</figcaption></figure></div><p>One year of outperforming the index, the year the index was negative. My selection seems good when the index turns negative, but that doesn&#8217;t compensate the very positive years of the index.</p><p>Every other year, I lagged. 2023, 2024, 2026 especially, and my average outperformance is around <em>-4.8 p.p.</em> Ouch!</p><p>Every year you underperform isn&#8217;t just a small miss in that year&#8202;&#8212;&#8202;it&#8217;s a smaller starting point for compounding in the next year, and the one after that, and every year for the next two decades. A consistent 5% annual underperformance is not boring and manageable. Over 25 years it&#8217;s the difference retiring in your 40&#8217;s or 60&#8217;s.</p><p>The saving grace is time. I still have a long runway (hopefully) ahead of me, and catching these mistakes means I can do something about it. The portfolio is correctable, the habits are correctable, and compounding still has decades to work in my favour if I stop getting in its way.</p><p>Main conclusion from Fern&#227;o&#8217;s analytics: I was putting in more work than an index investor&#8202;&#8212;&#8202;more research, more monitoring, more mental energy spent thinking about positions. I was taking on more risk, because individual stocks are meaningfully more volatile than a broad index fund. <strong>And in return for all of that extra work and extra risk, I was producing worse outcomes.</strong></p><p>Thanks Fern&#227;o for pointing out my bad strategy!</p><div><hr></div><h4>The Wins, the Blunders</h4><p>Diving deep from the macro levels, I also want Fern&#227;o to give me an analysis on individual positions.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!7aVw!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faada563b-88e4-4a33-9cc0-0f1be5776812_1600x705.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!7aVw!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faada563b-88e4-4a33-9cc0-0f1be5776812_1600x705.png 424w, /__u/substackcdn.com/image/fetch/$s_!7aVw!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faada563b-88e4-4a33-9cc0-0f1be5776812_1600x705.png 848w, /__u/substackcdn.com/image/fetch/$s_!7aVw!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faada563b-88e4-4a33-9cc0-0f1be5776812_1600x705.png 1272w, /__u/substackcdn.com/image/fetch/$s_!7aVw!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faada563b-88e4-4a33-9cc0-0f1be5776812_1600x705.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!7aVw!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faada563b-88e4-4a33-9cc0-0f1be5776812_1600x705.png" width="1456" height="642" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/aada563b-88e4-4a33-9cc0-0f1be5776812_1600x705.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:642,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!7aVw!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faada563b-88e4-4a33-9cc0-0f1be5776812_1600x705.png 424w, /__u/substackcdn.com/image/fetch/$s_!7aVw!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faada563b-88e4-4a33-9cc0-0f1be5776812_1600x705.png 848w, /__u/substackcdn.com/image/fetch/$s_!7aVw!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faada563b-88e4-4a33-9cc0-0f1be5776812_1600x705.png 1272w, /__u/substackcdn.com/image/fetch/$s_!7aVw!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faada563b-88e4-4a33-9cc0-0f1be5776812_1600x705.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Top Positions by Return</figcaption></figure></div><p>My top stocks by percentual return are Apple, Digital Ocean and Lam Research. The issue here is that I&#8217;ve also had my share of blunders as well:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!l3y1!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc3ef57b5-6bf2-491a-bd22-96614fa1e8e5_1600x442.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!l3y1!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc3ef57b5-6bf2-491a-bd22-96614fa1e8e5_1600x442.png 424w, /__u/substackcdn.com/image/fetch/$s_!l3y1!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc3ef57b5-6bf2-491a-bd22-96614fa1e8e5_1600x442.png 848w, /__u/substackcdn.com/image/fetch/$s_!l3y1!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc3ef57b5-6bf2-491a-bd22-96614fa1e8e5_1600x442.png 1272w, /__u/substackcdn.com/image/fetch/$s_!l3y1!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc3ef57b5-6bf2-491a-bd22-96614fa1e8e5_1600x442.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!l3y1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc3ef57b5-6bf2-491a-bd22-96614fa1e8e5_1600x442.png" width="1456" height="402" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c3ef57b5-6bf2-491a-bd22-96614fa1e8e5_1600x442.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:402,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!l3y1!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc3ef57b5-6bf2-491a-bd22-96614fa1e8e5_1600x442.png 424w, /__u/substackcdn.com/image/fetch/$s_!l3y1!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc3ef57b5-6bf2-491a-bd22-96614fa1e8e5_1600x442.png 848w, /__u/substackcdn.com/image/fetch/$s_!l3y1!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc3ef57b5-6bf2-491a-bd22-96614fa1e8e5_1600x442.png 1272w, /__u/substackcdn.com/image/fetch/$s_!l3y1!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc3ef57b5-6bf2-491a-bd22-96614fa1e8e5_1600x442.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Largest Losses</figcaption></figure></div><p>In this individual analysis, I can now keep track of all my individual positions in the portfolio.</p><p>At 60&#8211;70 positions, I had created a portfolio that was hard to manage and that was obscuring underperformance by spreading it thin enough to not immediately notice.</p><p>So: too many positions, negative alpha, underperforming the index consistently. The diagnosis was pretty clear. Let&#8217;s see what my personal assistant recommends next.</p><div><hr></div><h4>Fern&#227;o as My Financial Advisor: What Should I Actually Do?</h4><p>Data without prescription is just expensive introspection.</p><p>With the diagnosis of the ill issues around my portfolio, I built a &#8220;<em>Consolidate &amp; Invest</em>&#8221; tab to turn the analysis into action. The idea is simple: every month, I give Fern&#227;o my current portfolio state, my investment goals, my tax situation, and how much new capital I want to deploy, and I ask it what to do.</p><p>The prompt took a while to get right. In Generative AI, the quality of the output is directly proportional to the quality of the context you give it, which is true of every AI application, but especially obvious when the subject is your specific financial life. Once I gave it real numbers such as actual allocations, targets, emergency fund balance, monthly expenses, the recommendations got useful.</p><p>So, given all of this: Let&#8217;s imagine I want to invest &#8364;2,000 next month. What does Fern&#227;o say?</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!7MyG!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4b1bd234-a097-47ad-b217-09752119b0f2_1600x535.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!7MyG!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4b1bd234-a097-47ad-b217-09752119b0f2_1600x535.png 424w, /__u/substackcdn.com/image/fetch/$s_!7MyG!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4b1bd234-a097-47ad-b217-09752119b0f2_1600x535.png 848w, /__u/substackcdn.com/image/fetch/$s_!7MyG!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4b1bd234-a097-47ad-b217-09752119b0f2_1600x535.png 1272w, /__u/substackcdn.com/image/fetch/$s_!7MyG!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4b1bd234-a097-47ad-b217-09752119b0f2_1600x535.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!7MyG!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4b1bd234-a097-47ad-b217-09752119b0f2_1600x535.png" width="1456" height="487" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/4b1bd234-a097-47ad-b217-09752119b0f2_1600x535.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:487,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!7MyG!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4b1bd234-a097-47ad-b217-09752119b0f2_1600x535.png 424w, /__u/substackcdn.com/image/fetch/$s_!7MyG!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4b1bd234-a097-47ad-b217-09752119b0f2_1600x535.png 848w, /__u/substackcdn.com/image/fetch/$s_!7MyG!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4b1bd234-a097-47ad-b217-09752119b0f2_1600x535.png 1272w, /__u/substackcdn.com/image/fetch/$s_!7MyG!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4b1bd234-a097-47ad-b217-09752119b0f2_1600x535.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption"><strong>Consolidate &amp; Invest tab</strong></figcaption></figure></div><p>The executive summary flagged three things. One: my core growth ETFs are significantly underweight. I&#8217;d been trickling money into individual stocks and letting the ETF allocation drift down over the years. Two: several individual stocks and some bond and REIT ETFs are both underperforming and sitting above their target allocation, as we&#8217;ve seen.</p><p>And three, as I had a large expense in my life in the last month, my emergency fund covers 2.8 months of expenses, while the target is 6.</p><p>Here is the prompt I&#8217;m passing to Fern&#227;o for the recommendations:</p><pre><code>You are a portfolio advisor helping Ivo Bernardo optimize and grow his portfolio. You have two jobs today:
(1) identify what to clean up / rebalance, and
(2) produce a concrete investment routing plan for this month.

{portfolio_summary}

- -
## IVO&#8217;S FULL FINANCIAL PICTURE
### Target Portfolio Allocations (from investing ratios)
| Asset Class | Implied Dividend | Implied Return | Target % | Current % |
| - - - - - - - - - - | - - - - - - - - -| - - - - - - - - | - - - - - | - - - - - -|
| VWCE | 0% | 13% | 35% | 16.48% |
| Nasdaq | 0% | 18% | 25% | 9.75% |
| SPYD | 1.8% | 5.5% | 10% | 9.08% |
| Asia Pacific Div | 2.52% | 2% | 3% | 2.85% |
| Euro Div | 3.24% | 0% | 1% | 1.12% |
| TDIV | 2.52% | 7% | 9% | 0% |
| Individual Stocks | 3% | 6% | 17% | 17% |


## RETURN DATA - CRITICAL INSTRUCTION
The context above includes a section **&#8221;ACTUAL POSITION RETURNS (from verified price history)&#8221;**. This table contains real YTD / 6M / 1Y / 5Y returns for each position, resolved via the correct ticker mapping (e.g. &#8220;Amundi Nasdaq&#8221; maps to 6AQQ.DE, not QQQ or ^NDX). **Always use these figures when discussing position performance.** Do NOT substitute generic index benchmarks or make up returns for positions that appear in this table.
 - -
## OTHER CRITICAL CONTEXT
- **High Growth individual stocks** are intentionally small positions for learning/practice. Do NOT recommend selling them unless the thesis is fundamentally broken.
- **Same instrument across multiple banks = intentional custodian diversification.** Never flag this as redundancy.
- **Fragmented ETF positions across banks = intentional.** Only suggest consolidating same-bank duplicates.
- Ivo holds collectibles and real estate as non-correlated alternatives - some equity concentration is acceptable.
- **TDIV** is at 0% vs 9% target - highest priority to start.
- **VWCE** at 16.5% vs 35% target - major underweight, needs the most new capital.
- **Nasdaq** at 9.75% vs 25% target - also significantly underweight.
- **Individual Stocks** is overweight at the overall wealth level (26.6% vs 17% target) but at target within the investing ratios framework. Trim only if genuine redundancies exist.

 - -
## EMERGENCY FUND RULE
The context above includes an **&#8221;EMERGENCY FUND STATUS&#8221;** section with live data (avg monthly expenses, 6-month target, current DP&#8217;s balance, months covered).
- If **&#9888;&#65039; UNDERFUNDED**: include a &#128737;&#65039; EMERGENCY FUND action at the TOP of Part 2, and add a DP&#8217;s top-up row to the Part 3 routing table with the shortfall amount. Investment into growth assets is secondary until the fund is secure.
- If **&#9989; FULLY FUNDED**: briefly acknowledge it in Part 1 and proceed with the normal investment plan.
- Always state the exact figures: &#8220;&#8364;X/month avg expenses &#8594; 6-month target &#8364;Y &#8594; current DP&#8217;s &#8364;Z &#8594; X.X months covered.&#8221;
 - -
## YOUR OUTPUT (4 parts)
### PART 1 - EXECUTIVE SUMMARY
3 sentences: current state vs targets, biggest gaps, what this month&#8217;s plan achieves. Include one sentence on emergency fund status.
 - -
### PART 2 - SELL / REDUCE / CONSOLIDATE
If emergency fund is underfunded, start with:
 - -
#### &#128737;&#65039; TOP UP EMERGENCY FUND: DP&#8217;s / Term Deposits
**Action:** Deposit &#8364;[shortfall] into DP&#8217;s / term deposits to reach 6-month target
**Shortfall:** &#8364;[amount]
**Reasoning:** Current DP&#8217;s covers only X.X months of expenses vs the 6-month target. Reinforce before deploying capital to growth assets.
 - -
Then for each position to act on:
 - -
#### &#128308; SELL / &#128993; REDUCE / &#128256; CONSOLIDATE: [Ticker(s)]
**Action:** [One-line action]
**Estimated proceeds:** &#8364;[amount]
**Reasoning:** [2&#8211;4 sentences. Reference P&amp;L, position size, target gap.]
**Contrarian View:** [1&#8211;2 sentences - why someone might keep it]
**Learning Position?:** Yes / No
 - -
Aim for 2&#8211;4 actions. Sum up total estimated sell proceeds at the end of Part 2.
 - -
### PART 3 - THIS MONTH&#8217;S INVESTMENT ROUTING PLAN
**Total capital to deploy = fresh capital ({invest_amount}) + proceeds from Part 2 sells**
Allocate across ETFs both are underweight and serve the passive income goal.
If emergency fund is underfunded, add a DP&#8217;s top-up row at the top:
| Destination | Rationale | &#8364; to deploy | Action |
| - - - - - - -| - - - - - -| - - - - - - -| - - - - |
| DP&#8217;s (if underfunded) | Emergency fund shortfall | &#8364;&#8230; | Transfer to term deposit |
| VWCE | Largest ETF underweight | &#8364;&#8230; | Buy X units on [broker] |
| TDIV | 0% &#8594; 9% target, priority | &#8364;&#8230; | Start position |
| Nasdaq (CNDX/EQQQ) | 9.75% &#8594; 25% target | &#8364;&#8230; | Add to existing |
| Other | (if applicable) | &#8364;&#8230; | &#8230; |
| **TOTAL** | | **&#8364;&#8230;** | |
End with: **&#8221;After this month: estimated new % for VWCE ~X%, TDIV ~X%&#8221;**
Be decisive, specific, and data-driven. Use the actual portfolio data above for current values.</code></pre><p>And now.. let&#8217;s see the recommendations. The financial data has been anonymized, but the type of recommendations Fern&#227;o provides should still come through clearly.</p><div><hr></div><blockquote><p>PART 1&#8202;&#8212;&#8202;EXECUTIVE SUMMARY</p><p>Ivo Bernardo&#8217;s portfolio, valued at x across x positions, demonstrates strong diversification across asset types and platforms, with a commendable focus on learning through individual stock picks. However, key growth ETFs like VWCE and Nasdaq are significantly underweight, while several individual stocks and bond/REIT ETFs show persistent underperformance and are over their target allocation. This month&#8217;s plan prioritizes shoring up the emergency fund, then streamlines underperforming assets to redirect capital towards core growth and passive income generators, accelerating the path to the &#8364;x/month passive income goal. The emergency fund is currently underfunded, covering only x months of expenses.</p></blockquote><div><hr></div><blockquote><p>PART 2&#8202;&#8212;&#8202;SELL / REDUCE / CONSOLIDATE</p><p>&#128737;&#65039; TOP UP EMERGENCY FUND: DP&#8217;s / Term Deposits <strong>Action:</strong> Deposit x into DP&#8217;s / term deposits to reach 6-month target <strong>Shortfall:</strong> x <strong>Reasoning:</strong> Current DP&#8217;s covers only x months of expenses (&#8364;x) versus the 6-month target of &#8364;x. Reinforcing this essential safety net is the top priority before deploying capital to growth assets.</p></blockquote><div><hr></div><blockquote><p>&#128308; SELL: Sweetgreen (SG) <strong>Action:</strong> Sell entire position in Sweetgreen (SG). <strong>Estimated proceeds:</strong> &#8364;x <strong>Reasoning:</strong> This &#8220;Stock High Risk, Fast-Growth&#8221; position has a current value of only &#8364;x, an abysmal P&amp;L of -78.1%, and a 5-year return of -74.2%. Analyst consensus implies a further downside of -21.6%. While intended for learning, the fundamental thesis appears severely broken, making it an inefficient use of capital.</p></blockquote><div><hr></div><blockquote><p>&#128308; SELL: RH (RH) <strong>Action:</strong> Sell entire position in RH (RH). <strong>Estimated proceeds:</strong> &#8364;x <strong>Reasoning:</strong> This &#8220;Fast-Growth&#8221; individual stock has experienced a significant P&amp;L loss of -65.6% and a 5-year return of -78.0%. While analysts see a modest implied upside of +6.0%, the persistent underperformance indicates a broken thesis for a learning position.</p></blockquote><div><hr></div><blockquote><p>&#128308; SELL: InMode (INMD) <strong>Action:</strong> Sell entire position in InMode (INMD). <strong>Estimated proceeds:</strong> &#8364;x <strong>Reasoning:</strong> As a &#8220;Stock High Risk, Fast-Growth&#8221; position, InMode exhibits a substantial P&amp;L loss of -65.8% and a 5-year return of -70.5%. The position is very small, and its consistent underperformance suggests the learning value is diminished given the capital could be better utilized elsewhere.</p></blockquote><div><hr></div><blockquote><p>&#128308; SELL: Ishares Euro Corporate Large Bond (IBCX.DE) <strong>Action:</strong> Sell entire position in Ishares Euro Corporate Large Bond (IBCX.DE). <strong>Estimated proceeds:</strong> &#8364;x <strong>Reasoning:</strong> This Bond ETF has a significant P&amp;L loss of -56.3% and a negative 5-year return of -0.9%. The Bond ETF asset class is currently overweight (3.71% vs 2% target), making this an opportune time to reduce exposure to an underperforming asset within this category.</p></blockquote><div><hr></div><blockquote><p>SELL: iShares European Property Yield UCITS ETF (IPRP.AS) <strong>Action:</strong> Sell entire position in iShares European Property Yield UCITS ETF (IPRP.AS). <strong>Estimated proceeds:</strong> &#8364;x <strong>Reasoning:</strong> This REIT ETF is significantly overweight relative to its target (0.17% vs 0.01%) and has a P&amp;L loss of -28.0% with a negative 5-year return of -14.9%. Reducing this position aligns with target allocations and removes a consistently underperforming asset.</p></blockquote><div><hr></div><blockquote><p><strong>Total Estimated Sell Proceeds:</strong> &#8364;x + &#8364;x + &#8364;x + &#8364;x + &#8364;x = &#8364;x</p></blockquote><div><hr></div><blockquote><p>PART 3&#8202;&#8212;&#8202;THIS MONTH&#8217;S INVESTMENT ROUTING PLAN</p><p>Total capital to deploy = fresh capital (&#8364;x) + proceeds from Part 2 sells (&#8364;x) = &#8364;x. Given the emergency fund&#8217;s underfunded status, all available capital this month will be directed to reinforcing it.</p><p>After this month: The emergency fund will increase to &#8364;x (current) + &#8364;x = &#8364;x. This covers x months of expenses, significantly closer to the x-month target of &#8364;x, leaving a remaining shortfall of &#8364;x. Further investment into growth assets will be possible once the emergency fund is fully secured.</p></blockquote><p>Fern&#227;o&#8217;s recommendations are only as good as the context I give it, and I validate everything before acting. The goal isn&#8217;t to outsource the decisio, it&#8217;s to make sure I&#8217;m not missing something obvious before I make it. But this helps me deal with decision fatigue as I come into conclusion that that decision fatigue was only bringing me underperformance.</p><div><hr></div><h4>The Dividend Tab: Discovering That Income Isn&#8217;t Free</h4><p>Two last tabs before we finish this blog post.</p><p>The dividend tab does two things. It projects my forward dividend income based on current holdings and historical payouts. And it applies tax rates, both the withholding at source in each holding&#8217;s country and the income tax I&#8217;d owe in Portugal on top.</p><p>Here&#8217;s roughly what happens when I earn dividends in Portugal. A US company pays me &#8364;100. The US withholds 15% at source, treaty rate for Portuguese residents. I&#8217;m left with &#8364;85. Portugal then taxes that &#8364;85 as regular income. Depending on the bracket, I net somewhere between &#8364;55&#8211;&#8364;70 of the original &#8364;100 after the US withholding credit, I always model dividends at worst-case to keep my projections conservative.</p><p>Meanwhile, if I hold an accumulating ETF, one that reinvests dividends rather than distributing them, those same gains compound inside the fund. I only pay tax when I eventually sell, and potentially at a more favourable rate.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!5-wc!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb615ba42-51a1-4f94-b265-604481f86ab6_1600x443.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!5-wc!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb615ba42-51a1-4f94-b265-604481f86ab6_1600x443.png 424w, /__u/substackcdn.com/image/fetch/$s_!5-wc!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb615ba42-51a1-4f94-b265-604481f86ab6_1600x443.png 848w, /__u/substackcdn.com/image/fetch/$s_!5-wc!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb615ba42-51a1-4f94-b265-604481f86ab6_1600x443.png 1272w, /__u/substackcdn.com/image/fetch/$s_!5-wc!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb615ba42-51a1-4f94-b265-604481f86ab6_1600x443.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!5-wc!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb615ba42-51a1-4f94-b265-604481f86ab6_1600x443.png" width="1456" height="403" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/b615ba42-51a1-4f94-b265-604481f86ab6_1600x443.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:403,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!5-wc!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb615ba42-51a1-4f94-b265-604481f86ab6_1600x443.png 424w, /__u/substackcdn.com/image/fetch/$s_!5-wc!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb615ba42-51a1-4f94-b265-604481f86ab6_1600x443.png 848w, /__u/substackcdn.com/image/fetch/$s_!5-wc!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb615ba42-51a1-4f94-b265-604481f86ab6_1600x443.png 1272w, /__u/substackcdn.com/image/fetch/$s_!5-wc!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb615ba42-51a1-4f94-b265-604481f86ab6_1600x443.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Summary of Dividend Projection</figcaption></figure></div><p>The asymmetry is significant over a 20&#8211;25 year horizon. I was holding several distributing ETFs when accumulating equivalents existed, earning dividends that were immediately being taxed at a high combined rate, and thinking of it as &#8220;income.&#8221; I&#8217;ll still want to target around 1% of dividend income as time goes by, but I&#8217;ll not overweight distributing dividends.</p><p>A stock paying 4% today that&#8217;s growing its dividend at 10% annually will, after enough years, be yielding a much higher amount on your original cost. I also map this for the future in the application. Here are some examples based on my portfolio positions and how the net dividend projection may become:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!iPfE!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe87d8c65-a78c-41ff-af84-ab48c2fa3890_1600x570.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!iPfE!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe87d8c65-a78c-41ff-af84-ab48c2fa3890_1600x570.png 424w, /__u/substackcdn.com/image/fetch/$s_!iPfE!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe87d8c65-a78c-41ff-af84-ab48c2fa3890_1600x570.png 848w, /__u/substackcdn.com/image/fetch/$s_!iPfE!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe87d8c65-a78c-41ff-af84-ab48c2fa3890_1600x570.png 1272w, /__u/substackcdn.com/image/fetch/$s_!iPfE!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe87d8c65-a78c-41ff-af84-ab48c2fa3890_1600x570.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!iPfE!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe87d8c65-a78c-41ff-af84-ab48c2fa3890_1600x570.png" width="1456" height="519" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e87d8c65-a78c-41ff-af84-ab48c2fa3890_1600x570.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:519,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!iPfE!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe87d8c65-a78c-41ff-af84-ab48c2fa3890_1600x570.png 424w, /__u/substackcdn.com/image/fetch/$s_!iPfE!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe87d8c65-a78c-41ff-af84-ab48c2fa3890_1600x570.png 848w, /__u/substackcdn.com/image/fetch/$s_!iPfE!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe87d8c65-a78c-41ff-af84-ab48c2fa3890_1600x570.png 1272w, /__u/substackcdn.com/image/fetch/$s_!iPfE!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe87d8c65-a78c-41ff-af84-ab48c2fa3890_1600x570.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Dividend Projection</figcaption></figure></div><div><hr></div><h4>Correlation Analysis: What Diversification Looks Like</h4><p>The correlations tab was the most technically interesting thing to build, and also the most educational to actually use.</p><p>The basic idea of diversification is that you want your assets to behave differently from each other. If everything falls together when the market drops, you weren&#8217;t diversified, you just had the same bet in different packaging. The goal is to own things whose prices move with some independence, so losses in one part of the portfolio don&#8217;t necessarily show up everywhere else at the same time.</p><p>Correlation is how you measure this. A correlation of +1 means two assets move in perfect lockstep , if one goes up 10%, the other goes up as well. Zero means no relationship. Negative means they tend to move in opposite directions. In practice, most financial assets have correlations somewhere between 0 and +1. Fern&#227;o also gives me this data with a rolling correlation through time&#8202;&#8212;&#8202;take the example of Google and Apple:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!QOQ1!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe3e36356-50f8-43b9-84a7-f8bb4e484fad_1600x747.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!QOQ1!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe3e36356-50f8-43b9-84a7-f8bb4e484fad_1600x747.png 424w, /__u/substackcdn.com/image/fetch/$s_!QOQ1!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe3e36356-50f8-43b9-84a7-f8bb4e484fad_1600x747.png 848w, /__u/substackcdn.com/image/fetch/$s_!QOQ1!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe3e36356-50f8-43b9-84a7-f8bb4e484fad_1600x747.png 1272w, /__u/substackcdn.com/image/fetch/$s_!QOQ1!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe3e36356-50f8-43b9-84a7-f8bb4e484fad_1600x747.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!QOQ1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe3e36356-50f8-43b9-84a7-f8bb4e484fad_1600x747.png" width="1456" height="680" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e3e36356-50f8-43b9-84a7-f8bb4e484fad_1600x747.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:680,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!QOQ1!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe3e36356-50f8-43b9-84a7-f8bb4e484fad_1600x747.png 424w, /__u/substackcdn.com/image/fetch/$s_!QOQ1!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe3e36356-50f8-43b9-84a7-f8bb4e484fad_1600x747.png 848w, /__u/substackcdn.com/image/fetch/$s_!QOQ1!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe3e36356-50f8-43b9-84a7-f8bb4e484fad_1600x747.png 1272w, /__u/substackcdn.com/image/fetch/$s_!QOQ1!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe3e36356-50f8-43b9-84a7-f8bb4e484fad_1600x747.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Google vs. Apple Correlation</figcaption></figure></div><p><em>Google</em> and <em>Apple</em>: strongly correlated, as expected. Both are massive US tech companies, both exposed to the same macro environment (interest rates, ad markets, consumer sentiment, regulatory pressure) and both owned heavily by the same institutional investors. When one moves, the other tends to move in the same direction. Holding both gives you company-specific exposure, but it doesn&#8217;t help you from a diversification standpoint.</p><p>Apple against the total market is where it gets interesting. Apple has become so dominant in major indices, at various points the largest company in the world by market cap, representing a significant chunk of the S&amp;P 500&#8202;&#8212;&#8202;that it and the index have become nearly inseparable. The correlation between Apple&#8217;s stock and the Total Market is even larger:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!ohvI!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faf3b1a91-362f-47dd-b6b2-03ecc0b3015d_1600x703.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!ohvI!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faf3b1a91-362f-47dd-b6b2-03ecc0b3015d_1600x703.png 424w, /__u/substackcdn.com/image/fetch/$s_!ohvI!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faf3b1a91-362f-47dd-b6b2-03ecc0b3015d_1600x703.png 848w, /__u/substackcdn.com/image/fetch/$s_!ohvI!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faf3b1a91-362f-47dd-b6b2-03ecc0b3015d_1600x703.png 1272w, /__u/substackcdn.com/image/fetch/$s_!ohvI!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faf3b1a91-362f-47dd-b6b2-03ecc0b3015d_1600x703.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!ohvI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faf3b1a91-362f-47dd-b6b2-03ecc0b3015d_1600x703.png" width="1456" height="640" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/af3b1a91-362f-47dd-b6b2-03ecc0b3015d_1600x703.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:640,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!ohvI!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faf3b1a91-362f-47dd-b6b2-03ecc0b3015d_1600x703.png 424w, /__u/substackcdn.com/image/fetch/$s_!ohvI!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faf3b1a91-362f-47dd-b6b2-03ecc0b3015d_1600x703.png 848w, /__u/substackcdn.com/image/fetch/$s_!ohvI!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faf3b1a91-362f-47dd-b6b2-03ecc0b3015d_1600x703.png 1272w, /__u/substackcdn.com/image/fetch/$s_!ohvI!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faf3b1a91-362f-47dd-b6b2-03ecc0b3015d_1600x703.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Apple vs. Vanguard FTSE Total Market</figcaption></figure></div><p>Gold is where the picture changes. Against the total stock market, gold&#8217;s rolling correlation hovers around zero and regularly swings negative. This is what &#8220;flight to safety&#8221; looks like:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!w3DG!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d36d974-d7b1-48ca-b855-0f52b441e5ff_1600x730.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!w3DG!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d36d974-d7b1-48ca-b855-0f52b441e5ff_1600x730.png 424w, /__u/substackcdn.com/image/fetch/$s_!w3DG!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d36d974-d7b1-48ca-b855-0f52b441e5ff_1600x730.png 848w, /__u/substackcdn.com/image/fetch/$s_!w3DG!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d36d974-d7b1-48ca-b855-0f52b441e5ff_1600x730.png 1272w, /__u/substackcdn.com/image/fetch/$s_!w3DG!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d36d974-d7b1-48ca-b855-0f52b441e5ff_1600x730.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!w3DG!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d36d974-d7b1-48ca-b855-0f52b441e5ff_1600x730.png" width="1456" height="664" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2d36d974-d7b1-48ca-b855-0f52b441e5ff_1600x730.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:664,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!w3DG!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d36d974-d7b1-48ca-b855-0f52b441e5ff_1600x730.png 424w, /__u/substackcdn.com/image/fetch/$s_!w3DG!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d36d974-d7b1-48ca-b855-0f52b441e5ff_1600x730.png 848w, /__u/substackcdn.com/image/fetch/$s_!w3DG!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d36d974-d7b1-48ca-b855-0f52b441e5ff_1600x730.png 1272w, /__u/substackcdn.com/image/fetch/$s_!w3DG!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2d36d974-d7b1-48ca-b855-0f52b441e5ff_1600x730.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Physical Gold vs. Vanguard FTSE Total Market</figcaption></figure></div><p>This is why portfolio managers allocate to gold even though it doesn&#8217;t pay a dividend, doesn&#8217;t have earnings, doesn&#8217;t do anything except sit there and be gold. You&#8217;re not holding it for returns, you&#8217;re holding it to reduce how badly things go when stocks go down. The drag in good times is the price of the cushion in bad ones.</p><p>Bitcoin is the most interesting case study here, and it&#8217;s one where the data tells a pretty different story from the marketing. Some pitches for Bitcoin is &#8220;digital gold&#8221; , scarce, decentralised, a hedge against inflation and currency devaluation. Under that narrative, you&#8217;d expect it to behave like gold: low or negative correlation with equity markets, a place capital moves toward when risk appetite contracts.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!HAmN!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f9eeebd-e660-4bd4-bd0b-9c3a4a201d9f_1600x731.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!HAmN!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f9eeebd-e660-4bd4-bd0b-9c3a4a201d9f_1600x731.png 424w, /__u/substackcdn.com/image/fetch/$s_!HAmN!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f9eeebd-e660-4bd4-bd0b-9c3a4a201d9f_1600x731.png 848w, /__u/substackcdn.com/image/fetch/$s_!HAmN!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f9eeebd-e660-4bd4-bd0b-9c3a4a201d9f_1600x731.png 1272w, /__u/substackcdn.com/image/fetch/$s_!HAmN!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f9eeebd-e660-4bd4-bd0b-9c3a4a201d9f_1600x731.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!HAmN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f9eeebd-e660-4bd4-bd0b-9c3a4a201d9f_1600x731.png" width="1456" height="665" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/4f9eeebd-e660-4bd4-bd0b-9c3a4a201d9f_1600x731.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:665,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!HAmN!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f9eeebd-e660-4bd4-bd0b-9c3a4a201d9f_1600x731.png 424w, /__u/substackcdn.com/image/fetch/$s_!HAmN!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f9eeebd-e660-4bd4-bd0b-9c3a4a201d9f_1600x731.png 848w, /__u/substackcdn.com/image/fetch/$s_!HAmN!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f9eeebd-e660-4bd4-bd0b-9c3a4a201d9f_1600x731.png 1272w, /__u/substackcdn.com/image/fetch/$s_!HAmN!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f9eeebd-e660-4bd4-bd0b-9c3a4a201d9f_1600x731.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Bitcoin vs. Nasdaq 100 ETF</figcaption></figure></div><p>The rolling correlation chart between Bitcoin and the NASDAQ is&#8230; not that. They&#8217;re positively correlated. When tech stocks fall, Bitcoin tends to fall too. When tech stocks rally, Bitcoin often rallies alongside them. The likely explanation is that Bitcoin&#8217;s investor base and the NASDAQ&#8217;s investor base overlap heavily.</p><p>For Bitcoin to earn the uncorrelated status that gold actually holds, it would need to decouple from risk sentiment in a sustained, consistent way across multiple market cycles.</p><div><hr></div><h4>Okay But What Am I Actually Changing?</h4><p>Building this module gave me a lot of insights. I was feeling frustrated as an individual investor of not having a tool to deeply analyze my portfolio&#8202;&#8212;&#8202;most tools are too expensive or thought for institutional investors. Luckily, the AI world is giving us these abilities to build our own tools quite fast. This is not, of course, an enterprise grade application, but more of a personal cockpit that I&#8217;m building.</p><p>So let me just say what I&#8217;m doing differently after analyzing this data:</p><ul><li><p><strong>Stopping new capital going into individual stocks.</strong> Not selling everything immediately, some positions still have real theses and some would trigger unnecessary tax events if sold now.</p></li><li><p><strong>Switching to accumulating ETFs wherever possible.</strong> For every ETF where an accumulating version exists, I want the accumulating one.</p></li><li><p><strong>Fixing the emergency fund before anything else.</strong></p></li><li><p><strong>Reducing position count over time.</strong> I have too many open positions. I&#8217;m not going to make 90 sell decisions at once, but as positions naturally run off or hit sell criteria.</p></li></ul><div><hr></div><h4>What I Actually Learned From Building This</h4><p>I built this module in my Personal Assistant to get better financial information.</p><p>I&#8217;m lucky I built this when I still have a lot of years of investing ahead of me. The earlier you catch a bad strategy, the more time you have to compound the correct one instead. That&#8217;s the whole game, really.</p><p>What are your thoughts on using AI for this kind of personal financial analysis? And if you were building your own personal assistant, what module would you build first?</p><p>More modules to come!</p><p>&#8212; Ivo</p>]]></content:encoded></item><item><title><![CDATA[Weekly AI Digest: Half a Billion Reasons to Add Limits to your Claude]]></title><description><![CDATA[Week 22, 2026]]></description><link>https://thedatajourney.substack.com/p/weekly-ai-digest-half-a-billion-reasons</link><guid isPermaLink="false">https://thedatajourney.substack.com/p/weekly-ai-digest-half-a-billion-reasons</guid><dc:creator><![CDATA[Ivo Bernardo]]></dc:creator><pubDate>Sun, 31 May 2026 17:35:57 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!_Fhi!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c7249b4-6568-482e-97f4-704289d4c945_1838x974.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Welcome back. </p><p>This was the week AI and theology briefly collided, as Pope Leo had a word for the technocrats.</p><p>But the ones truly in need of divine intervention? The startup that (allegedly) accidentally got a bill $500 million after letting employees run Claude Code with zero guardrails. Meanwhile, two CEOs quietly walked back their robot apocalypse prophecies. Funny how IPO season inspires sudden optimism.</p><p>Grab your coffee and let&#8217;s dive into week 22.</p><div><hr></div><h4>&#187; Stat of the Week: $500 Million</h4><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!0Vfu!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5fb1c01d-e013-45f6-bd4a-0bbd2e6808fa_2048x2048.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!0Vfu!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5fb1c01d-e013-45f6-bd4a-0bbd2e6808fa_2048x2048.png 424w, /__u/substackcdn.com/image/fetch/$s_!0Vfu!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5fb1c01d-e013-45f6-bd4a-0bbd2e6808fa_2048x2048.png 848w, /__u/substackcdn.com/image/fetch/$s_!0Vfu!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5fb1c01d-e013-45f6-bd4a-0bbd2e6808fa_2048x2048.png 1272w, /__u/substackcdn.com/image/fetch/$s_!0Vfu!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5fb1c01d-e013-45f6-bd4a-0bbd2e6808fa_2048x2048.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!0Vfu!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5fb1c01d-e013-45f6-bd4a-0bbd2e6808fa_2048x2048.png" width="471" height="471" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/5fb1c01d-e013-45f6-bd4a-0bbd2e6808fa_2048x2048.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1456,&quot;width&quot;:1456,&quot;resizeWidth&quot;:471,&quot;bytes&quot;:4955879,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://thedatajourney.substack.com/i/200003002?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5fb1c01d-e013-45f6-bd4a-0bbd2e6808fa_2048x2048.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!0Vfu!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5fb1c01d-e013-45f6-bd4a-0bbd2e6808fa_2048x2048.png 424w, /__u/substackcdn.com/image/fetch/$s_!0Vfu!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5fb1c01d-e013-45f6-bd4a-0bbd2e6808fa_2048x2048.png 848w, /__u/substackcdn.com/image/fetch/$s_!0Vfu!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5fb1c01d-e013-45f6-bd4a-0bbd2e6808fa_2048x2048.png 1272w, /__u/substackcdn.com/image/fetch/$s_!0Vfu!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5fb1c01d-e013-45f6-bd4a-0bbd2e6808fa_2048x2048.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div><hr></div><h4>&#187; Industry Highlights: Market &amp; Product</h4><p><strong>Anthropic</strong> launched Claude Opus 4.8&#185; this week, an upgrade to its flagship model that brings improvements across coding, agentic tasks, and reasoning benchmarks. Fast mode now runs at 2.5&#215; the speed and costs three times less than previous models.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="/__u/substackcdn.com/image/fetch/$s_!_Fhi!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c7249b4-6568-482e-97f4-704289d4c945_1838x974.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="/__u/substackcdn.com/image/fetch/$s_!_Fhi!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c7249b4-6568-482e-97f4-704289d4c945_1838x974.png 424w, /__u/substackcdn.com/image/fetch/$s_!_Fhi!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c7249b4-6568-482e-97f4-704289d4c945_1838x974.png 848w, /__u/substackcdn.com/image/fetch/$s_!_Fhi!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c7249b4-6568-482e-97f4-704289d4c945_1838x974.png 1272w, /__u/substackcdn.com/image/fetch/$s_!_Fhi!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_webp, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c7249b4-6568-482e-97f4-704289d4c945_1838x974.png 1456w" sizes="100vw"><img src="/__u/substackcdn.com/image/fetch/$s_!_Fhi!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c7249b4-6568-482e-97f4-704289d4c945_1838x974.png" width="1456" height="772" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/6c7249b4-6568-482e-97f4-704289d4c945_1838x974.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:772,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:297325,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://thedatajourney.substack.com/i/200003002?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c7249b4-6568-482e-97f4-704289d4c945_1838x974.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="/__u/substackcdn.com/image/fetch/$s_!_Fhi!, /__u/thedatajourney.substack.com/w_424, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c7249b4-6568-482e-97f4-704289d4c945_1838x974.png 424w, /__u/substackcdn.com/image/fetch/$s_!_Fhi!, /__u/thedatajourney.substack.com/w_848, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c7249b4-6568-482e-97f4-704289d4c945_1838x974.png 848w, /__u/substackcdn.com/image/fetch/$s_!_Fhi!, /__u/thedatajourney.substack.com/w_1272, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c7249b4-6568-482e-97f4-704289d4c945_1838x974.png 1272w, /__u/substackcdn.com/image/fetch/$s_!_Fhi!, /__u/thedatajourney.substack.com/w_1456, /__u/thedatajourney.substack.com/c_limit, /__u/thedatajourney.substack.com/f_auto, /__u/thedatajourney.substack.com/q_auto:good, /__u/thedatajourney.substack.com/fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6c7249b4-6568-482e-97f4-704289d4c945_1838x974.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Opus 4.8 Benchmark vs. Competitors</figcaption></figure></div><p><strong>The launch happened when the company just closed its Series H round, becoming officially the most valued startup in the world.</strong></p><p><strong>Tied with the Claude models, an unnamed enterprise client</strong> made headlines for all the wrong reasons this week: an AI consultant revealed to Axios that one of their clients burned $500 million in a single month on Claude, simply because nobody thought to cap employee usage&#178;. Engineers were reportedly checking the weather with agentic workflows, at hundreds of dollars a pop. The same week, Microsoft confirmed it has been quietly scaling back internal Claude Code licences&#179;, and Uber admitted it blew through its entire 2026 AI budget by April&#8308;. </p><p><strong>Asana</strong> acquired Y Combinator-backed Stack AI for a reported $75 million, adding no-code AI agent building capabilities to its project management platform. Asana&#8217;s Q1 revenue hit $205 million, yet its stock sits 53% below its January peak, as the company has been a victim of the famous <a href="https://en.wiktionary.org/wiki/SaaSpocalypse">saaspocalypse</a>.&#8309;</p><p><strong>Lastly, DuckDuckGo</strong> saw nearly 28% more visits in the week following Google&#8217;s announcement that users love AI Mode. Will Google learn that people just want a choice? Or will they prove that when you embed AI into our product successfully, no one cares?&#8310;</p><div><hr></div><h4>&#187; Economic Highlights: The World of AI</h4><p><strong>Pope Leo XIV</strong> published his <a href="https://www.vatican.va/content/leo-xiv/en/encyclicals/documents/20260515-magnifica-humanitas.html">first encyclical this week</a>, <em>Magnifica Humanitas</em>, and it uses AI as a lens to diagnose something older and more structural: concentrated power, eroding democracy, and a tech elite that increasingly shapes the world to its own advantage. </p><p>The encyclical isn&#8217;t really about AI. It&#8217;s about who controls the future, and whether anyone is accountable for that. The Economist noted that the document explicitly attacks what it calls &#8220;technological messianism&#8221;, the quasi-religious belief that enough compute will solve enough problems. Is the pope really concerned about the future of society? Or just afraid to have another competitor in the religious field? &#8311;</p><p><strong>Speaking about heads of the technological church, Sam Altman and Dario Amodei</strong> both spent the week walking back earlier statements about AI causing a jobs apocalypse, a reversal that Fortune connected to the fact that both companies are preparing for blockbuster IPOs. Altman told an Asia-Pacific audience that mass job displacement is unlikely, even as he acknowledged entire categories (customer service, for instance) are disappearing. &#8312; &#8313;</p><div><hr></div><h4>&#187; Track of the Week: Greg Brockman in the Knowledge Project Podcast</h4><div id="youtube2-6JoUcQ1qmAc" class="youtube-wrap" data-attrs="{&quot;videoId&quot;:&quot;6JoUcQ1qmAc&quot;,&quot;startTime&quot;:null,&quot;endTime&quot;:null}" data-component-name="Youtube2ToDOM"><div class="youtube-inner"><iframe src="https://www.youtube-nocookie.com/embed/6JoUcQ1qmAc?rel=0&amp;autoplay=0&amp;showinfo=0&amp;enablejsapi=0" frameborder="0" loading="lazy" gesture="media" allow="autoplay; fullscreen" allowautoplay="true" allowfullscreen="true" width="728" height="409"></iframe></div></div><div><hr></div><p>This week had a little of everything: a pope diagnosing Big Tech's original sin, a company lighting half a billion dollars on fire because someone forgot to add a spending cap, and two AI CEOs discovering that "mass unemployment is coming" is a harder pitch when you're trying to get retail investors to buy your shares. </p><p>See you next week!</p><p>&#8212; Ivo</p><div><hr></div><p><strong>Footnotes</strong></p><p>1 <a href="https://www.anthropic.com/news/claude-opus-4-8">https://www.anthropic.com/news/claude-opus-4-8</a></p><p>2 <a href="https://www.msn.com/en-us/news/technology/company-accidentally-blows-500-000-000-on-claude-ai-in-one-month/ar-AA24nahv">https://www.msn.com/en-us/news/technology/company-accidentally-blows-500-000-000-on-claude-ai-in-one-month/ar-AA24nahv</a></p><p>3 <a href="https://www.theverge.com/tech/930447/microsoft-claude-code-discontinued-notepad">https://www.theverge.com/tech/930447/microsoft-claude-code-discontinued-notepad</a></p><p>4 <a href="https://fortune.com/2026/05/26/uber-coo-ai-spending-tokens-claude-code/">https://fortune.com/2026/05/26/uber-coo-ai-spending-tokens-claude-code/</a></p><p>5 <a href="https://thenextweb.com/news/asana-acquires-stack-ai-agent-builder-saas">https://thenextweb.com/news/asana-acquires-stack-ai-agent-builder-saas</a></p><p>6 <a href="https://www.pcgamer.com/hardware/duckduckgos-ai-free-search-saw-nearly-28-percent-more-visits-in-the-week-following-googles-insistence-that-people-love-ai-mode/">https://www.pcgamer.com/hardware/duckduckgos-ai-free-search-saw-nearly-28-percent-more-visits-in-the-week-following-googles-insistence-that-people-love-ai-mode/</a></p><p>7 <a href="https://techcrunch.com/2026/05/25/the-popes-ai-encyclical-isnt-really-about-ai/">https://techcrunch.com/2026/05/25/the-popes-ai-encyclical-isnt-really-about-ai/</a></p><p>8 <a href="https://thenextweb.com/news/sam-altman-says-an-ai-jobs-apocalypse-is-unlikely">https://thenextweb.com/news/sam-altman-says-an-ai-jobs-apocalypse-is-unlikely</a></p><p>9 <a href="https://fortune.com/2026/05/26/sam-altman-dario-amodei-walking-back-ai-jobs-apocalypse-prophecies-ipo/">https://fortune.com/2026/05/26/sam-altman-dario-amodei-walking-back-ai-jobs-apocalypse-prophecies-ipo/</a></p>]]></content:encoded></item></channel></rss>