A Risk Professional’s Guide to Physical Risk Assessments
Climate Risk

Companies Don’t Always Agree
Climate Services: The Business of Physical Risk


(00:00) Chris Struggles to Define Climate Risk Scores, Arbitrary Scores

(13:30) Incentive Structures: Information Asymmetry and Product Quality, Comparison with Other Scoring Industries

(23:50) Practical Examples: Zillow Real Estate Scores

(29:00) Uncertainty and Verification

(35:00) Climate Opinion Scores

(41:00) Geocoding

(50:00) Parametric Insurance

Side Note:
Several people have noted that when I’ve been thinking about something for a while/get excited about something I just start yapping without setting any context. You can observe this phenomenon in the first half of this podcast, I’ll be working on that so future episodes should be easier to follow.

(00:20) Double IPA

(01:20) The Gang Struggles to Define a Platform: Metadata, Storage, Compute, APIs

(11:20) The Value of a Platform

(16:00) What If, What If, What If We Replaced Geospatial Platforms with Claude

(24:00) Custom Use Case Development vs Platforms

(27:00) Money Money Money, Chris’ time as a traveling Google Cloud Salesman, Analytics Companies, Skytruth’s Sentinel-1 Pipeline, Honestly This Conversation Kind of Meanders a Bit.

(34:00) Ask Not What Platforms Can Do For You

(15:00) Bayesian Deforestation Detection, operation-oriented algorithms and platforms

(27:00) Landtrendr in the West Bank, CCDC

(35:00) Anomalous Change Detection, how to deal with co-registration issues.

Links:
Landtrendr

Bayesian Deforestation Detection

CCDC product on GEE ( I was wrong about this being MODIS)

Anomalous Change Detection

Dealing with co-registration issues with ACD

(00:00) Imagine useful geospatial models, it’s easy if you try.

(08:00) Academic papers, life at the paper mill, distilled .pub for geospatial.

(24:30) Weather data vs earth observation data consumption. Differences in value and monetization, difficulty in forecasting products.

(49:30) Opus 4.5, Claude code, AI tools and the death of meaning and effort at work.

• I am super, super grateful to all of you for subscribing and reaching out. I got to meet/chat with so many cool people in earth observation this year, and I hope this continues next year.

• Forecasted US almond yield.

• ERA5: don’t believe his lies.

• No one really wants to read about anomaly detection, even if it’s on fancy newly-released hyperspectral data.

• Foundation models: can’t live with ‘em, can kind of live without ‘em unless you’re trying to be an influencer in the niche field of earth observation.

• Benchmarks kind of suck, but we also need them because we don’t have anything else.

• Google released embeddings, they’re honestly pretty good, and I’m still planning on milking them for content in 2026 (but I think they shouldn’t have included all that auxiliary non-EO data in training).

• We released Geovibes and I used it to map catfish farms, so it’s totally not a useless demo.

Read on to see my slightly unhinged thoughts about my posts.

Subscribe now

Technically from 2024, but my first post on substack and my friends with kids say your favorite one is usually the first-born. They don’t say this out loud, but I can see it in their eyes and soul.

This one was particularly fun to write and iterate over. Initial results were extremely disappointing, but I read up on the almond growing season and realized I was modeling everything wrong: bloom occurs earlier in the year than greening, so it turned out NDVI from the prior year was extremely strongly correlated with yield in the next year.

Another fun one! I was quite pleased with this analysis as it resulted in some folks from a start up working specifically on measuring/forecasting precipitation reaching out for a chat. This is when I started realizing that I could use substack posts as a sort of social query to find interesting people to chat to.
I was astonished to see how different various sources of precipitation data were over CONUS. In previous lives at yield-modeling companies I think some of us might have taken for granted how similar different datasets can be. Funnily enough, I actually have an unpublished draft on a similar analysis of divergences in growing degree days over the US corn belt where the differences are perhaps even more striking. Maybe one day I’ll get back around to publishing it!This was quite early on in my writing (I mean it still is), and emojis were still kind of funny as an LLM-ism. I really dislike them now. You’ll also notice traces of LLM-isms in the titles, (‘why this matters’ etc…) which I also really dislike. We live and we learn.

This one was an interesting one: it received the most attention on social media out of any of my posts at that point, due to Wyvern’s open data release, but was the least read (only 90 reads). My original plan was to actually perform anomalous change detection (ACD) between pairs of images, which I find much more interesting and relevant to earth observation, but ran out of time, so settled for anomaly detection. I still plan to write about ACD one day, it is an underrated technique for change detection in remote sensing.

It was at this point it started dawning on me that I was wasting my time chasing social media clout writing about things I thought people were interested in, instead of just writing things I myself was interested in. I also thought people might be interested to see the code used to generate the images (they were not).
This one also has emojis. I’m sorry. Also the LateX didn’t render.

People are really interested in foundation models in EO, as well as contrarian takes. Look at this plot of my subscribers, can you tell when I started being a contrarian asshole?

I still think EO foundation models are a big money and time sink. If you don’t believe me, you should read Akis’ Spectral Reflectance post which is much more thoughtful than this post.

Fundamentally I’m very grateful though because when this post ‘took off’, I got to meet many people I would now call my friends, who ended up helping me with geovibes. We even have a secret slack channel where we complain about foundation models. Also, shoutout to Johannes Jakubik the lead author of TerraMind who chose to engage with me on the post (see the comments), and actually took the time to reach out and chat too!
Something I am still thinking about that is related to this post: what is the best you can do with a pixel-wise model + post processing on all of these benchmarks?

More of a commentary on how benchmarks don’t really reflect reality, and that there are some particularly tricky inference issues specific to geospatial that even major labs/releases are not immune to based on some of the artifacts I’ve seen in outputs this year. I had a lot of fun thinking about different types of geospatial errors, and got to make the Santa Fe Quartet.
Happy to report 0 emojis in this one.

The thought that sparked this one is still bothering me to this day: how much does it cost to make a map? I had heard from friends that worked at companies that had done global deployments that things sat in the $5k - $15k range. So I decided to try and extrapolate how much it might cost to develop a global map on GEE.
I still think about this a lot: no one knows how much anything costs! I had big plans to write continue writing about this subject, but unfortunately failed to make inroads with any organizations doing these things. I don’t blame them, money is a sensitive subject. If you are a company that would like to talk about this however, please do reach out!
A cool thing that came from this post is that Simon Ilyushchenko mentioned this post in an announcement stating that Google had changed the pricing structure of GEE!

I still think these embeddings are a little ‘overfit’ because of this, but damn they are pretty good! An experiment I’ve been meaning to run: sample built area over cities at different elevations in the world, can you train a model to recover elevation from the AEF embeddings? If you can, then they are a little overfit, if not then I stand corrected!

Would call this one a puff piece: when I started the blog I made a promise to myself that I would have to write a least a single line of code for each blog post. This one was pure commentary, but at least it wasn’t AI slop.

I have been thinking about this for a long time, so it felt really good to release geovibes and write about it!
The gist is: you can search embeddings with just a notebook and a downloaded duckdb table + faiss index. That’s really all you need. Making the embeddings is a different story! We should think about making these (unproven) tools as accessible as possible so people can test them out in the real world.

Dog-fooding geovibes and mapping aquaculture/catfish farms in Alabama. It turns out the CDL’s map is quite poor (confuses open water and aquaculture), but with geovibes it’s fairly easy to map these very distinctive levee ponds! Was happy to close out the year with this, and put geovibes to use.

Subscribe now

• I used embeddings to map aquaculture in Alabama.

• This map is much better than the one provided by CDL.

• When performing this kind of exercise it can be useful to think of first solving the problem at the tile/geometry level, then moving to pixel level if necessary.

• You too can use geovibes + existing embedding databases to perform this kind of exercise.

• New in geovibes: sampling + classification functionality

• New in geovibes: 'detection mode’ : iterate on your classifier outputs.

• New in geovibes: add any GEE layer onto your map to help you label.

Subscribe now

Two months ago,we released geovibes, a tool that allows users to interact with geospatial embedding databases. The motivation was to allow those who were interested in geospatial embeddings to ‘see for themselves’ rather than be told what embeddings can and can not do by LinkedInfluencers™.

Unfortunately, we released something that falls squarely in the “cool-demo” bucket, and not necessarily the '“actually useful” bucket: yes you could click and find things, but it was still not quite possible to make a map using it. This is an incomplete attempt to rectify this. There is still a lot of work to do!

Opus 4.5 in Claude Code was absolutely instrumental in implementing all of this. I have many thoughts about LLM-based coding/work which I’ll probably explore in a future post, but in my own litmus test of usage, I have not reverted back to OpenAI’s Codex or Cursor since I switched.

Embedding Retrieval

In fair Alabama, where we lay our scene: we start by using geovibes + the Earth Genome provided quantized SSL4EO-pretrained DINO-ViT embeddings which are generated from annual Sentinel-2 composites. We use google maps to search for potential catfish farms to seed our search:

We seed our search with this location in geovibes (still figuring out a better way to do this other than manually), and start searching using our duckdb + faiss index. This results in the following view, where samples all over Alabama are colored by their similarity. In the following video clip I’m:

• Seeding the aquaculture search at the Miller Catfish Farm

• Opening the tile layer, and sorting the samples by 'most Dissimilar’: aka the samples that are furthest from the initial sample, within the tile layer

• Panning around, and collecting those ‘hard negatives’: i.e things that look somewhat similar in the embedding space, that are definitely not aquaculture. You’ll notice many of these tiles are river banks/deltas.

In order to create a classifier: I’ll also label some of the aquaculture positives returned by geovibes. How do I know these are aquaculture? For one, they look like the smooth/green/blue ponds of the Miller Catfish Farm. However, we can also do some research and find out that earthen ponds are the primary method of cultivation for freshwater aquaculture in Alabama1: these are primarily made by digging out a bunch of dirt with a bulldozer and then filling it with water.

Multiple rectangular ponds are laid-out side by side, divided by straight levees. These are typically uniformly sized/shaped, facilitating management and harvesting. This article actually contains a nifty photograph of the harvesting process, which highlights how important the regular shape and proximity to a road is:

Embedding Classification

Now that we have our positives and *some* negatives: we should probably sample some more negatives from our dataset. We’ve implemented two ways to do this in geovibes: you can randomly sample from the duckdb table, or you can do a stratified sample by ESRI/IO landcover map for the year you’re interestsed in. The second method requires a GEE account.

We then filter the sampled negatives if any of them fall within a buffer distance of any of our hard-earned positive samples from geovibes.

Note that if the ‘thing’ you are looking for covers the majority of your area, this will cause some issues for your classifier as you are likely to sample ‘false negatives’. It’s possible that in the future we’ll enable people to use geovibes to post-filter the sampled negatives manually.

Once we have our geovibes positives, geovibes negatives + sampled negatives, we’re ready to train a classifier! We’ve implemented a pretty simple pipeline with xgboost that:

1. Trains an xgboost classifier on the dataset

2. Queries all of the embedding vectors from the duckdb table

3. Transfers the embedding vectors to a pandas dataframe/numpy array

4. Uses the trained xgboost model to score the embeddings from the duckdb table

5. Thresholds the detections by a probability score, and returns both the individual tiles + merged detections for a set probability threshold.

Somehow this process only takes around 40 seconds: our training dataset is quite small (~5000 samples total), but inference is over 5 million 384-dim embeddings! I was quite surprised by this, but then again the 5 million embeddings appear to fit quite comfortably in memory on my 32GB M1 pro, so perhaps this isn’t such a big dataset after all.

Classifier Iteration

We now have a set of detections over Alabama with associated classifier probabilities. We can re-use a lot of the tooling implemented in geovibes to re-label samples, and create a refined dataset for a second round of classification.

In the video below: I show how to load the output of our classification pipeline back into geovibes, I then sort the samples by lowest probability (closest to our threshold of 0.5) and begin labeling again. You’ll notice there are some positives that are on the edge of aquaculture ponds, but also many negatives that occur along the banks of rivers and lakes. In this process, if I am unsure whether a tile is an aquaculture pond or not, I leave it unlabeled.

This process allows us to quickly iterate on our initial embedding classifier using the same framework we did to create our original dataset: quickly viewing and labeling samples of interest. In this case we are using the classifier score/probability as our distance metric rather than a cosine distance.

Interestingly: this re-labeling process which is fairly fast, appears to greatly improve the quality of our classifier. Whereas the initial iteration appeared to struggle with various false postiives such as coastal areas and mining pools, relabeling ~ 300 negatives and positives, which takes around 5 minutes, allows us to improve the classifier a great deal as you can see in the figure below.

We can have a look at the spatial distribution of the effect of relabeling on our classifier, by looking at where the majority of detections have been:

• removed (v0 positive —> v1 negative),

• retained (v0 positive —> v1 positive)

• gained (v0 negative —> v1 positive)

As we can see: many of the removed points are clustered around waterways/water bodies in parts Alabama that most of our labeled aquaculture ponds are not in: to the south near Mobile, as well as North along the Tennessee River, and in the east of the state along the shores of Weiss Lake and the Coosa River. Meanwhile, new detections, in blue, seem to be primarily concentrated in the same region where most of our detections are retained: the Western portion of the Black Belt.

We can take a closer look at where detections have changed to get sense of how the decision boundary of our classifier has shifted: in the Black Belt many of our new detections are at the boundaries/edges of existing aquaculture detections:

Removals in the same area seem to typically be associated with water bodies which have a less regular shape:

Is this an aquactulture pond? I’m actually not sure (but this is important, more on this below), but it seems unlikely given what we know about levee ponds and aquaculture practices in this region. Another form of removal is found in this area on the edges of a larger set of detections:

Removals in other areas follow the same pattern: smaller bodies of water, with less regular shapes are now classified as not-aquaculture:

In fact, you can see this reflected in the fact that the total number of clusters detected is reduced by half (~800 to 400) after re-labeling, while the total area is reduced by a lower factor, indicating that the relabeling process has removed many of these smaller detections. After having looked at this set of detections, I’m fairly confident that we have improved the quality of our embedding-based aquaculture classifier!

Tiles to Pixels to Map

People think they want pixel-level maps. For many first passes I believe this is unnecessary and potentially counter-productive. If we had started solving this problem at the pixel level, we would have had to deal with many more rows in our machine learning problem. However, ultimately if we do want a pixel-level map I believe the following workflow is still beneficial: solve the problem at the tile/geometry level first, then within your tiles solve the problem at the pixel level. This greatly reduces the size of the problem.

In this particular case since we know what signal we expect to see over our tiles (water) we can use a land cover map like CDL to simply filter out the non-water pixels. It turns out CDL actually has an ‘aquaculture’ class (in cyan below), but it’s pretty poor and understandably seems to be heavily confused with the ‘open water’ class (dark blue).

In order to produce a map from this:

1. I manually validated the map, removing a small amount of remaining false positives such as industrial waste ponds and a few remaining coastlines.

2. I buffered the detections by a half tile.

3. I selected aquaculture + open water from the latest CDL year over those detections.

That concludes the process of using embeddings to produce a map of aquaculture over Alabama! The final postprocessed geojson can be found here, and the final geotiff at 30m resolution can be found here.

Concluding Thoughts

Looking through these detections, it’s clear to me that the levee-pond style of aquaculture appears to exist in a very geographically constrained area (the Black Belt). This is something that a simple google search would have revealed: a combination of climate, topography and soil type appear to have created ideal conditions for catfish farming. I clumsily georeferenced a soil map with our detections (in red) to highlight this :

Blackland prairie soil happens to be a dark, heavy, alkaline clay soil, with a very high water retention rate due to the clay content. This means if you dig a hole in it, and fill that hole with water, the water tends to stay! While we chose a data-driven approach to tackle this mapping problem, we could have started from a first-principles approach instead. A little bit of research could have really helped narrow down the area over which we deployed a classifier!

When iterating over the classifier using geovibes, it became obvious to me that the difficulty in these mapping exercises is in the edge cases: sure you can easily map regularly shaped levee ponds, but what about less-regularly shaped ponds, could those be aquaculture as well? How would you determine that/validate? I often found myself using additional information (i.e proximity to industrial parks, or mining areas) to determine if a sample should be labeled as positive or negative.

I think embedding-based workflows face a little bit of a catch-22: yes they are useful, but often a user will require information not contained in the embeddings to perform validation/iterate on their data. If the labels required to do this already exist, then perhaps the problem the user is trying to solve is not an interesting one to solve in the first place. Applications where embeddings will appear to shine initially are ones that can be solved via easy visual verification of (free) medium-resolution imagery: confined animal feeding operations, deforestation/mining, aquaculture. Additionally, there have been demonstrations of the reproduction of existing maps (think USDA’s Cropland Data Layer), while I think these are useful demonstrations of the signal contained in embeddings, it’s not clear to me how one would operationalize these types of models.

Subscribe now

(28:42) High-res SAR/Umbra data detour: building damage estimation in Jamaica, matching imagery up to vector datasets

(34:33) Embeddings mentioned

(37:10) Differences in seasonality, illumination and histogram matching: foundation models mentioned, discussions around Krishna’s model performance over Gaza

(04:30) Geographic generalization discussion. Husky vs Malamute vs Wellpad.

(11:32) Chris fails to map coconut palm because it’s hard. Read the paper carefully.

(15:30) Krishna maps oil slicks, but it’s hard.

(20:30) Cognitive debt, problem selection and failure modes in geospatial.

(31:00) The sales cycle, promises, inflated expectations. Limitations in geospatial.

(34:00) Problem selection in journalism. Are we running out of ideas?

(37:00) Turning a failure into a success. Cover crop mapping is hard.

(45:00) Turning failure into a success: predicting sugarcane yield is hard.

(49:00) Is building tooling easier than solving modeling problems?

(53:00) Vertical seems better than horizontal. Solutions are multi-modal.

A central theme in throughout this conversation is the use of auxiliary geospatial products, science and human intuition to validate maps.

(00:00) Introduction, what is validation?

(02:30) Chris talks about mapping oil palm using Sentinel-1

(25:00) Krishna talks about global land cover mapping using Sentinel-2

(51:00) What tools do Krishna and Chris use for validation. (Krishna loves QGIS)

1. Using Geovibes you can interact with a foundation model/embeddings using a jupyter notebook via similarity searching on your laptop.

2. This gives you a sense of your model’s vibes, i.e minutiae in the data/minority classes that perhaps only you care about, and aren’t measured in benchmarks.

3. We provide a script that enables you to build a searchable index from embeddings provided online.

4. We suggest the community uses the following schema for the release of embeddings: geoparquet files with [tile_id, geometry, embedding] where geometry is a (lat, lon) representing the centre of a tile. Each geoparquet file should contain the embeddings generated over an Military Grid Reference System (MGRS) tile.

5. Metadata on temporal range, tile size, overlap and pixel resolution should also be included

6. Potential future plans may involve: dealing with embedding rasters, making this more memory efficient, an embeddings pipeline for you to generate your own using custom models, experiments showing that this is useful.

   Subscribe now

Why?

I feel like earth observation research is currently a very `vibes-based` community.
Expensive foundation models get released without benchmarks. Global embedding datasets get released with limited benchmarking on downstream tasks. The benchmarks that do exist are narrow in geographic scope, and some are already maxed out by models that we know are just not that great. There also appears to be a general lack of procedure around benchmarking: some papers might benchmark on PANGAEA, others on Geo-Bench, and others on their own collection of tasks.

Most of us don’t read the actual papers released, we just go off the LinkedIn posts and the vibes. Myself included!

As an example just look at this set of tables from the DINOv3 paper: m-brick-kiln, m-eurosat, m-pv4ger, m-pv4ger-seg or 4/12 of these tasks are clearly saturated across all of the models being compared here. These should be taken out to pasture.

My feeling is that these ‘general’ sets of benchmarks targeted at measuring how foundational a given model is don’t reflect what users are actually interested in. We can acknowledge that those in academia must play the numbers game and over-index on novelty and benchmark performance as opposed to impact and utility (I have been guilty of this myself), but that does not mean that the rest of us should follow suit!

We see the same effect with LLMs coding agents: frontier labs will release benchmark results, but the thing I tend to pay attention to are real users vibe checks. After all, these are real users, not static benchmarks that can be gamed! The goal of geovibes is to provide a framework for users to vibe check embeddings/foundation models easily.

How Do I Use It?

Geovibes is a weekend project. It should not be used as a production tool. I’m not even sure how you would do that to be honest. There are probably a million small things that are broken, and some big ones. With that being said: you should use geovibes to create datasets and maps.

A pretty simple, universal workflow for mapping is:

• label positives and negatives with Geovibes

• you may need to sample more diverse negative samples since your dataset will only consist of ‘hard’ ones resulting from the search. Landcover is a great way to do this.

• train a classifier (XGBoost/Random Forest/LightGBM/Catboost) on top of those embeddings.

• Once you have a trained classifier: deploy this classifier over any set of embeddings generated by the same source model.

The classification workflow has not been implemented as of yet in geovibes, but you can subscribe to updates to be notified when it is.

Subscribe now

What is Geovibes?

We built on top of the open source ei-notebook released by the Earth Genome to address this issue: if we’re going to be vibes-based, then let’s do it for real! Yes, we are missing a strong set of universal benchmarks in EO, but then again is that really important? Each user has their own specific use-case they care about: we should instead empower them to easily get a ‘feel’ for a given set of embeddings aka the vibes.

Let’s allow the user to interact iteratively with these embeddings, the same way they would do with chatbots/coding agents to figure out if they prefer Gemini Pro 2.5, Opus 4.1, GPT-5-Codex or GPT-5-High or … you get the idea.

Geovibes is a tool that allows you to create or download local, searchable vector databases of embeddings produced by earth observation foundation models. It comes at a time when we have an increasing availability of embedding datasets (Earth Genome Softcon, Google Alpha Earth, TESSERA etc…), but no unified or easy way to interact with them.

Think of it as Earth Index, but it runs on your laptop. This allows you to checkout the vibes of any new embedding dataset or foundation model. The idea is relatively simple: click on a location, retrieve the embedding vector for that location, use it to query your database using similarity search, then return the top-k neighbors.

Human in the Loop Searching

You needn’t stop there though: what if the returned search results contain some relevant results, but some irrelevant ones? Geovibes enables you to label negative samples: essentially pushing your query vector to be more different than some selected points.

In the video below: I’m using Google’s Alpha Earth Foundations (AEF) embeddings in an attempt to find aquaculture in Alabama. I click on an example and zoom out: I see that because AEF embeddings do not contain any spatial information, it seems that both the Tombigbee River (West) and the Alabama River (East) are picked out by my search results (compare the image below and the video).

These are clearly not the type of thing I am looking for, and I can indicate this to my search: I draw a polygon around the Tombigbee River search results, add these as negatives to my dataset, and perform a new search with my updated query vector. We see that many search results from the Alabama River disappear as a result of this, even though I did not label those as negatives!

This type of operation is valuable for two reasons: it makes the similarity searching steerable to some degree AND it helps surface ‘hard negatives’ which are useful for training classifiers from your dataset.

How it Works

I want to specifically credit Ed Boyda from the Earth Genome with coming up with the idea to run this type of search locally: as far as I’m aware (correct me if I’m wrong Earth Genies), he implemented the original version of this which used annoy and held the embeddings in a GeoDataFrame in memory.

The ‘backend’ of geovibes relies on two core technologies: faiss for similarity searching, and duckdb for fast + local access to embeddings. Using the duckdb vss extension, you can technically do this without faiss, but we found that solution scaled poorly both in terms of index-build time, and search time.

• Given a set of embedding vectors: we build a faiss index over the embeddings, this tells us for a given vector, which vectors are similar for all vectors in the dataset.

• We link this faiss index to a duckdb table by tile_id: the faiss index returns which tile IDs are most similar to a given query,

• We then use duckdb to retrieve the embedding vectors for those tile IDs. This enables us to re-calculate and update a given query vector with the actual embedding values, as well as display them on the UI for the user to peruse.

Subscribe now

For now: you can use geovibes to checkout embeddings and foundation models. Which ones are useful for your specific use cases? Are you able to find your things immediately, or does it take several iterations of labeling?

Available Searchable Databases

We have pre-compiled some faiss indexes + duckdb tables for you to check out and play around with. For now, searchable indexes are available over Alabama and New Mexico. These are hosted on Source Cooperative (thanks Jed!).

Alabama:

• Earth Genome SSL4EO DINO ViT Sentinel-2 embeddings 2024-2025

• Earth Genome Quantized SSL4EO DINO ViT Sentinel-2 embeddings 2024-2025

• Earth Genome Quantized Softcon embeddings 2024-2025

• Google Satellite Embeddings v1 2024-2025

• Clay v1.5 NAIP embeddings 2019-2020 (via a heroic effort from Zoe, thank you!)

New Mexico:

• Google Satellite Embeddings v1 2024-2025

Please get in touch if you’d like us to generate other regions.

You can access these via the download_embeddings.py script in geovibes (thanks Noah!): if you don’t have a great internet connection, I recommend you start with a Google Satellite Embedding v1 database, as these are only 64-dimensional and were created with no tile-overlap, so are much smaller. The Clay v1.5 database is about 30 GB (!!).

Bring Your Own Embeddings

We provide scripts in geovibes such that users can generate their own searchable indexes which are compatible with the geovibes user interface. In order for these to work, your embeddings should follow this schema: [tile_id, geometry, embedding] where tile_id is a unique identifier, geometry is a (lat, lon) representing the centre of a tile and embedding is the embedding array.

We suggest that if others would like to make open tiled embedding datasets, that use the geoparquet format, separate out embeddings by MGRS tile, and use this schema. Additionally, metadata such as temporal range, tile size, overlap and pixel resolution should also be included in the filenames/folder structure.

Our hope is that the this tool increases utilization of some already existing embedding datasets, and encourages others to release more datasets knowing that they can now be searched over fairly large regions on a laptop.

What’s Next?

• Many embeddings appear to be released as rasters currently, at the pixel level. I think it’d be interesting to see how/if we can support those in this kind of workflow.

• We’ll be writing some posts about fun problems we encountered while doing this:

  • including how duckdb’s vss extension doesn’t appear to scale well,

  • why we think smaller embeddings are better,

  • how duckdb/faiss appear to scale with search size/dataset size/request size,

  • how far we think we can push this kind of local workflow on our laptops.

• We’ll also be using the tool to do some mapping, and probably finding lots of bugs along the way, you can expect us to update the tool as we use it, but you should feel free to fork it and modify it to fit your own purposes as well.

With LLMs/coding agents as they are: there is no need to have a ‘central’ repository. If you want to modify it to fit your purposes, or fix a bug, just ask Codex or Claude to do it, no need to submit a PR/github issue unless you really want to.

There is more than one way to skin a foundation model.

Subscribe now

Acknowledgements

Thank you to the Earth Genome, in particular Ben Strong, Hutch Ingold, and Ed Boyda for supporting this effort by providing embeddings and agreeing to open source the original ei-notebook implementation.

Thank you to Zoe, Noah and Isaac who contributed to Geovibes, and listened to me rant about foundation models and embeddings.
Thank you to Jed Sundwall, who enabled us to distribute these databases via Source Cooperative.

Throughout this conversation, a common theme explored is the nature of the abstractions implemented in Google Earth Engine, vs the open-source ecosystem.

If anyone from Google sees this: we would love more export tasks.

(00:00) Introductions and career paths

(08:00) Questions from Episode 1

(17:00) Geospatial platform discussion. Descartes Labs, Google Earth Engine, the Impact Observatory pipeline and more!

Our first episode is titled: “Embeddings in Context: Tools for Geospatial Problem Solving” and you can find it on spotify, apple and youtube.

Give it a listen to find out:

• what problems we solved use embeddings

• how I spent $40,000 in cloud credits on pre-training on satellite imagery only to be beaten by ImageNet weights

• why I think embeddings for search should be high recall across a broad range of problems at the expensive of precision, and more!

I’ll be writing about some of these topics in more depth too, you can subscribe below so you don’t miss those pieces.

Subscribe now

Timestamps sustainably sourced and hand crafted.

(0:00) Intro

(2:30) Background, earth observation and existing algorithms and the embedding fallacy.

(9:49) Validation, how to solve problems, seductive embeddings, the museum of cool demos.

(19:22) What are embeddings?

(21:00) Search vs embeddings, the development of Earth Index.

(25:00) Expert embeddings: the dumbest embeddings that work.

(30:25) Computer vision embeddings/ImageNet. Experiments in pre-training, how to spend $40,000 in cloud credits.

(37:22) Should we pre-train on satellite imagery? DINO v3

(44:30) What properties should embeddings have for search? Recall vs Precision/Landcover+

(50:00) Real world consequences of mapping

(54:00) Future outlook, Krishna is jaded

• Google Earth Engine lowered their prices! Simon Ilyushchenko very generously mentioned a previous post I had made referencing the confusing economics of earth engine. While I think removing pricing for individual seats and lowering online pricing is a great step in the right direction, it unfortunately is unlikely to have a huge effect on the calculations in my post which were all based on the batch API. What I think is more likely to happen is that companies will have to choose between switching their batch pipelines (exporting) over to online workflows (high volume API, streaming etc…). Perhaps the lowering of the online API price means we’re likely to see xee gain more adoption in commercial settings.

Subscribe now

TLDR;

• Sounds like The Moose was a good cat ❤️.

  

• Google shows that FMs don’t appear to scale effectively for linear-probing on tasks: I would not train another general EO FM right now.

• It does not make sense to try to build a business on unproven and expensive technology economically. It does make sense to try to sell cloud services.

• Even if embeddings were a proven geospatial company, it is impossible to compete with a killer unicorn when you are a show pony. Competing with Google’s distribution and subsidies is a brutal game

Thoughts on AlphaEarth Foundations

Google recently released global embeddings from their alphaearth foundation model (AEF). There’s been a lot of buzz around the embeddings themselves, here are some thoughts on what this means for Big Embeddings as an industry.

1. Read my lips: no new foundation models.

At least not in the general case. Google scaled their experiment beyond what any other company in the space could reasonably do. Have a look at the training data:

Although some sensors are conspicuously missing (notably MODIS and VIIRS), we can see they threw the kitchen sink at this model. Some interesting choices include: text from the Global Biodiversity Information Facility (GBIF) + Wikipedia, NLCD (only available in the US) and monthly aggregates of ERA5-Land. These are not all used as inputs. Let’s look at how the results scaled with increasing data:

Across many of the 15 tasks, linear-probe performance saturates around 10^7-10^8 observations, with additional data yielding changes within the confidence intervals. A few tasks still improve, but the slope is quite small beyond this point on most tasks. On some tasks, the performance actually decreases with increasing data. This suggests weak returns for general-purpose embeddings in linear-probe evaluation, making large-scale training a highly questionable investment unless targeting a specific task that demonstrably benefits from scale. I will note that AEF appears to scale better than some other existing foundation models, but higher performing ones such as TerraMind are missing from these comparisons.

While AEF appears to outperform all other methods on the chosen benchmarks, it is also worth noting that both CCDC and MOSAIKS, two unsupervised methods appear to perform solidly on all benchmark tasks at much lower complexity than foundation models. In general, I’m growing a little weary of squinting at benchmark plots and tables: if AEF achieves balanced accuracy of 0.22 on us_trees and Satclip got 0.18 I’m unlikely to use either for a task that ressembles us_trees. Benchmarks are just the price of entry: we need them to know that a model is worth even looking at, but the true test is the real world!

2. Embeddings don’t look like a business:

I see people claiming that the future of EO is based on ‘geo-embeddings’, and that Google must think so too, because they released these embedding. I think this is wrong: zoom out! The product here is not the embeddings but where they were released: Google Earth Engine and associated cloud products. If anything, the fact that these were released globally without an explicit pay-wall should tell you what value Google assigns to these. While this is indeed a big moment as it represents the first global set of embeddings which are so widely and easily accessible, I think this will play out more like the release of Dynamic World or Impact Observatory’s LULC map rather than ChatGPT (sorry Krishna I stole your point but imitation is the sincerest form of flattery). These are impactful data products that need to be used with others in order to be useful. Meanwhile, if you do want to access these, and use them in conjunction with other raster datasets, you know where to do that. And don’t forget to sign up for a cloud project, and can we also interest you in some cloud storage and compute while you’re here 😉?

Being an embedding application layer is simply a losing proposition for every company: you pay for cloud resources to host and store your embeddings, millions to train your own models (please don’t), engineers to build your APIs, and THEN you have to convince customers to hand over their cash for your embeddings. This equation changes when you provide the compute, storage, and your embeddings just happen to be a cherry on top that users can choose to use, provided they pay for cloud resources that you provide.

Playing devil’s advocate for a second: the flip side here is that you potentially be able to pick up customers that can’t and won’t use GEE. There is also a case to be made for problem-specific embeddings with verified gains in task cost, but we have yet to see anyone in the space show hard data on embeddings being a cheaper way to develop valuable applications.

3. The race to the bottom:

Google is offering $5k grants to support people building on this product. Have you ever heard of a company paying people to use their killer product? What are the implications of this? Even if we take the counterfactual position that these embeddings are indeed a killer product and Google is attempting to flood the market a-la Uber with their cheap rides, what does this mean for start-ups looking to compete against something that’s currently been assigned financial cost of almost $0?

From my initial experiments on retrieval on smallholder crops: these embeddings are appear to be potentially useful (more on this coming soon), although I have some reservations about how they were trained. It’s still clear we have not figured out how to effectively interact with geospatial embeddings, and what the true cost and time savings are from using them. Verification is still the biggest problem, and we encounter it over and over again in EO: you can build a great looking model, but how do you go about verifying it’s detecting the actual signal you care about? I would love to see more companies working to solve this problem, although the answer may be buy timely high-resolution imagery.

4. One more time with feeling:

What does this mean from an industry perspective? A complex solution looking for a problem is a major red flag: sometimes ideas consume so much capital and energy that they become too big to fail. Whole companies stake their future on them, people bet their careers, and investors bet their dollars. However, there are few companies in the space that can afford to sustain these types of ideas until they become useful, so if you are considering it please consider the economics of the endeavour as much as you consider the technology. Having organizations spin-up and fail unnecessarily erodes trust in the industry as a whole and is likely to harm it in the long term.

4. Shameless self-plug:

I’m starting a podcast with my good friend Krishna Karra. The first episode will probably be about embeddings, unless we all lose interest before we record.

Subscribe now

• It is cool that people are releasing embeddings, kudos to the Deepmind/Google team!

• Google’s Embedding Fields show large scale spatial variation that can be problematic for semantic search.

• DuckDB is awesome, and can serve as a vector database.

• Embeddings should be served at a scale that is larger than 10m pixels for ease of access.

• Embeddings for retrieval should compress what is hard to join later (texture, specific land-use, shape), not what is trivial (elevation, lat/long, other raster dataset).

• It’s important to do experiments with your embeddings: search and PCA are two easy ways to get a sense of the vibes.

  Subscribe now

  ---

As part of a recent project, I’ve been assembling various types of geospatial embeddings and testing out retrieval on them. This is all part of a bigger release of some open source tools that will hopefully enable you to do this type of analyses as well, which is coming soon!

Google Satellite Embeddings

One of the first embedding sets we turned to was the Google Satellite Embedding dataset: it’s a 64-dim embedding dataset that encodes ‘ temporal trajectories of surface conditions’ at each 10m pixel by ‘various Earth observation instruments and datasets’. It sounds like there’s a paper coming soon, but the only real information I could find was this conference poster, and this presentation. Between these two sources we can see that that various time-steps for various sources/sensors are encoded into a 64-dim latent space, and then decoded and used to reconstruct a set of amount of targets:

The poster gives us a little more clarity on exactly what these targets/reconstruction targets are:

These are certainly interesting choices as targets for learning, and I’m not sure this is a particularly good idea, at least in the context of using embeddings for search or at large spatial scales. I’ll spend the rest of this post discussing why.

Pixels to (Geo)Parquet

The first step we need to take is aggregate these 10m pixels to larger tiles. This is because it is simply impractical to search over every single 10m pixel, and the Tobler’s law of geography tells us that things that are close together tend to be more similar anyway.

As a note to embedding providers: if you are considering pixel-level embeddings, you should also consider releasing an aggregated version at a coarser scale, this will help people explore you data more easily.

We can achieve this by first defining our tiles, uploading these geometries to Google Earth Engine (GEE) as GEE assets, and then simply performing reduceRegions over them. These operations allows us to export the the median for each embedding ‘channel’ over a tile (I chose 250m x 250m) as a geojson. We then end up with a 64-dim vector associated with each tile centroid, which we need to conver to geoparquet. In the case of the state of Alabama, this resulted in about 2.1 million embeddings.

As an alternative, you could export the embedding geotiffs from ee.ImageCollection("GOOGLE/SATELLITE_EMBEDDING/V1/ANNUAL", and perform the zonal stats yourself using rasterio/geopandas.

DuckDB

Once we have our embedding geoparquet files, containing tile centroid and a 64-dim embedding vector we can use duckdb to fulfill two purposes: it can act as a database such that we can easily retrieve batches of embeddings for the parquet file using SQL queries, but it also can act as a vector database for similarity searching thanks to the vss extension. Creating an Hierarchical Small Navigable Worlds (HSNW) index on top of your database is extremely easy just run:
CREATE INDEX my_hnsw_cosine_index ON my_vector_table USING HNSW (vec) WITH (metric = 'cosine');

as you are creating your duckdb table.

This then allows you to use a 64-dim query vector to find ‘similar’ entries in your db. We ran into some interesting issues with scaling and performance that we will be writing up in some separate benchmarking posts, but this is the general gist of it!

Search

Now that we have a searchable vector db, we can simple choose a lat-lon and query Google’s Embedding Fields for similar points! In the video below, you can see me interactively querying the Embedding Fields for tiles that look ‘suburban’. We can see that it does a fairly good job with these! As we zoom out, we see detections concentrated on the periphery of major urban areas in Alabama. As we zoom into a different area, we see that the retrieved tiles have similiar ‘suburban’ characteristics to the query tiles.

This is a pretty neat way of getting a sense of any embedding datasets ‘vibes’ since you are able to tell how it does in terms of human-level semantics: you can search for any minority class of interest in your dataset, and see how well the embeddings have encoded that class as a ‘concept’. If the search results are similar tiles/things, then congratulations your model, and embeddings are good. At least when it comes to that particular search/class.

What Properties Should Embeddings Have?

As I started searching, I noticed some puzzling behavior: some searches seemed to work well (i.e the suburban neighborhood search from above). Others exhibited rather odd behavior, especially as I widened the scope of the search. For example: here I’m selecting a forest tile, and searching for the 25,000 closest neighbors in the embedding space. The search results show a very interesting spatial pattern!

In fact, I can reproduce the complement of this pattern, simply by seeding my search with a forest tile further south:

Interesting! I would expect these tiles to be somewhat semantically similar in any meaningful embedding space. I would also not expect to see this sort of dramatic geographic pattern, unless there was some underlying physical phenomena occurring driving changes in the satellite observations that constitute the embeddings.

In order to investigate this further, I performed PCA on the 2 millions 64-dim tile embeddings and plotted the first 9 components. We can see the occurence of all sorts of larger scale spatial patterns, and that PC1 and PC3 appear to line up very well with our strange search results! Incidently these explain 20% and 11% of the variance in the full embedding dataset.

So why might these larger-scale patterns appear? Is there some inherent physical signal driving them? Although I doubt that the embeddings are so well-attuned to the signals from South-Eastern forests that a whopping 20-30% of the variance in them can be attributed to them, we can image since elevation was one of the learning targets, it could somehow distort the latent space:

While this doesn’t appear to be a perfect match, and it’s highly likely that vegetation characteristics would change with elevation, one has to wonder whether they change sufficiently across this territory to warrant this much variance in the embeddings. Should forest in the south of Alabama really be almost semantically closer to wetlands, than forest in the north?

Conclusion

I’d like to commend the Deepmind + Google team for releasing these. These are the first 10m/pixel level embeddings that have been made available to the public, and they are very unique! I’ve had quite a lot of luck searching for something in them (suburbs, golf courses, different types of crops) so they clearly contain some useful information. However I think this sort of unexplained large scale variation across the embeddings for things that humans find semantically similar is ultimately an undesirable property. If I wanted to filter my searches by elevation, that would be a very simple thing to do using a DEM! However if that signal is explicitly incorporated in my embeddings, there is no way for me to get rid of it. Embeddings for retrieval should compress what is hard to join later (texture, specific land-use, shape), not what is trivial (elevation, lat/long, other raster datasets).

As we move towards more people releasing embedding datasets, we need to make sure that the public has the necessary tools and intuition to make use of them. It is also a good thing to do to think about what specific properties you might want your embeddings to have, and adjust model training to reflect that.

Finally: we’ll be releasing tools so that people can do investigations like these soon, so stay tuned!

Subscribe now

Acknowledgements

Thank you to Zoe Statman-Weil, Isaac Corley and Noah Golmant for listening to my embedding rants, and for helping me develop these ideas!

Some Ways to Help:

Please consider donating to help Central Texas flood recovery:

1. Kerr County Flood Relief Fund

2. TEXSAR

3. Kerrville Pets Alive

4. Salvation Army Kerrville

5. Mercy Chefs

   ---

In geospatial, it’s rarely good news. Weather data and satellite imagery never accompany “senior dog who was in the shelter for 2 years adopted” articles. Instead, earth observation images are often used to observe geopolitical and natural upheaval and the accompanying death and devastation. This is an important function: it helps bring awareness to just how devastating conflict and natural disasters can be. This function comes with an implicit responsibility: when your pixels represent real lives, empathy and accountability must guide your messaging and actions.

I’ve noticed a concerning trend in the industry recently where sharing information and raising awareness quickly morphs into engagement farming and marketing. If you’re in the business of information the line is unfortunately blurred; but when a tragedy such as the Kerr County/Texas floods happens there seems to be pervasive eagerness to point at it and exclaim “Ha! Our data showed this would happen, and it did!”, forgetting in the process that their pixels are people.

Companies instinctively soften this opportunistic marketing with with the equivalent of corporate “thoughts and prayers”: this can take the form of a call to action (“we must do better”), or an expression of sadness (“it is with a heavy heart…"). These thoughts and prayers appear to absolve the company of potential misgivings concerning their apparent enthusiasm. To illustrate my point, here are some examples of what I perceive as disaster opportunism, recently posted on LinkedIn

All of these examples are informative: they show proprietary data in order to help highlight the effects of the floods. Yet I still feel uneasy with these posts: their purpose is clearly to highlight the utility of proprietary data/analysis, by leveraging the occurrence of the Texas floods. One has to wonder whether these moments incentivize profit-driven behavior at the expense of empathy.

Rules concerning corporate messaging are complicated in the geospatial world. When your pixels are people, I believe there is an additional responsibility to tread delicately. If companies would like to do better, I’ve compiled a list of things they can do to offset the surely unintentional eagerness of their marketing and genuinely help:

1. Provide your data for free to journalists, and let them do the talking.

2. At the top of your post, include links so your readers can donate/help to funds specifically set up to address the catastrophe.

3. Provide your data for free to emergency response teams.

4. Make your data surrounding this particular catastrophe open, so that the wider public can use it and inspect it.

5. Provide concrete ways we can ‘do better’: for example if you have flood risk data, overlay it with overture maps data and identify the highest building density areas that experience risk similar to Camp Mystic. Publish this information. (This is a made up example, I’m sure you can think of better things to do!).

It’s not all bad though: Floodbase, for example, shared their data with journalists at The New York Times without overtly marketing their services during the tragedy:

Is this entirely altruistic? No, but at least they provided the data for free to journalists and helped provide useable, informative maps of the event to the public. I think this is a much more palatable form of marketing when it comes to catastrophes. I should also note that First Street did provide their flood zone maps to NPR, and it appears ICEEYE’s data supported disaster response coordination. While these are commendable acts, the blending of genuinely helpful actions with overt marketing still feels somewhat tone-deaf and unsettling.

We can’t take empathy and good vibes to the grocery store: companies have to sell their product to survive, and sometimes that product is forecasts and observations of death and destruction. Profitability is necessary, but I do hope companies in this space can strike a more careful balance between their business interests and genuine empathy toward those affected.

This criticism isn't limited to corporations: I’m guilty of the same pursuit of visibility. When these types of tragedy strike my immediate reaction is some variant of “I should write about this on my blog”, and my motivations are probably more self-serving than altruistic. Earlier this year, following the Palisades fires, I wrote a post demonstrating drought calculations using Google Earth Engine. While I hope this contributed valuable tools and insights, reflecting honestly I recognize there was also a rather large element of self-promotion. I also of course note the irony of writing a ‘meta-post’ about this very same subject.

When our pixels are people, it's vital that we continually ask ourselves: Are we genuinely adding value to important discussions, or merely capitalizing on tragedy for visibility?

Platform Hunger Games

Picture this: you’re a small, scrappy geospatial start up with a dream. Maybe you want to map floods, or maybe quantify soil carbon. Maybe you want to count cars in a parking lot, who knows! Don’t let your investor’s dreams be dreams. Where are you going to get imagery, merge it with compute in order to deliver beautiful models and maps?

Five years ago, the landscape certainly looked different: at the time, Descartes Labs1 (now EarthDaily) was certainly still a strong option, and possibly the only viable one at the time. Planetary computer has just launched. Google Earth Engine (GEE), while it had launched much earlier in 2010, still did not offer a commercial license.

These days: while Microsoft has launched Planetary Computer Pro (pricing tbd), GEE is probably the most established commercial geospatial platform as it stands2. There is of course a final option: roll your own! Yes, writing GDAL is scary, wrangling AWS Batch/Lambda is scary, scaling compute and storage access is scary, and geospatial data engineers are expensive! However you may be surprised to find that under some conditions, this might be the most economical solution.

Subscribe now

Earth Engine Economics

GEE pricing is quite interesting. We can see below that the ‘Professional’ plan costs a flat fee of $2000 per month.

But what on earth are EECU-hours?? From the page itself:

• An Earth Engine Compute Unit (EECU) is a mechanism for representing an amount of instantaneous processing power. Earth Engine tracks the total computational footprint of tasks as a function of their EECU usage through time (EECU-seconds, EECU-hours, etc.). Because Google has many different types of processor cores, architectures, etc., EECUs are a useful abstraction for talking about computational power.

Got it, so EECU-seconds ~ cloud compute.

Usage costs themselves are displayed a little further down the page, so reading between the lines we can see that on the Professional plan $2000 you get $200 of Batch credits, and $40 of Online credits.

There is tiered pricing to take into account too: as you use more, the cost per EECUs drops:

Earth Engine Compute Units

The esoteric nature of EECUs make it very difficult for companies to estimate their potential usage costs. The GEE team itself appears unsure about exactly how to do this, at least in the general case. In the following examples, I’m going to run some experiments myself, but it should be noted that even for the same workload, EECUs costs can be highly variable due to caching and changes to the GEE backend. Nonetheless, if you look at Len’s comment here, it provides a workflow by which to estimate costs: run some sample workflows on a subset of tiles/tasks, and extrapolate.

In this particular case let’s look at what is likely a common use case: access to imagery for model inference. Unfortunately this workflow still can not be performed fully in GEE. Yes there is a ee.Model.fromVertexAi, but using it is quite challenging, particularly due to the limit on maxPayloadBytes.

Typically in this particular workflow users will define imagery exports over areas, these will result in GeoTIFFs stored in a Google Cloud Storage (GCS) bucket, which will then be pulled for subsequent inference, rather than wrestling with the still experimental ee.Model.fromVertexAi.

I happen to be running a similar workflow for another project so here is what our export looks like:

1. Cloud mask using cloudscore+, and reduce to the median over 2024-01-01 - 2024-12-31.

2. Separate out each Sentinel-2 band, and export in its native resolution as uint16

3. Export by MGRS tile, over the island of Java in Indonesia.3

4. Tally up the EECUs cost per MGRS tile. Remember an EECU-hour is $0.4. In this particular case the total cost works out to around $258.

I’ve plotted the results both in terms of cost per MGRS tile and cost per km^2:

Since Java represents about 0.1% of the worlds land-mass, excluding the Arctic and Antarctic, and I can’t really find this information anywhere else, I’m going to extrapolate the cost of a GEE-generated global annual composite. Taking into account the tiered pricing this results in a cost of around $1600004.

I’m sure that the folks at Google could easily optimize this export process to be at least four times as efficient, so let’s set a range for a global annual composite between $40000 - $160000. This is a lot of money, and remember, we have not paid for compute/inference yet!

A Rock and a Hard Place

So what are your alternatives? In the case of Sentinel-2, could you simply access the open data yourself. Of course this means paying data engineers to build and maintain your data pipelines. I happen to have the privilege of knowing several people who have built or maintained custom global geospatial inference pipelines. Based on the aggregate of the estimates given to me on total costs to run a model on every Sentinel-2 pixel in the world for a year, I’d estimate this value to lie in the range of $15000 - $350005, including compute/inference time.

So let’s make some simple assumptions, and see how this plays out:

• It takes a Senior Data Engineer about 6 months full time to make a full imagery pipeline + inference pipeline.

• It takes the same engineer about 3 months to set up an equivalent pipeline using GEE as the image provider.

• Once set up, the GEE pipeline takes 0.1 FTE of the engineers time to maintain, while the custom pipeline takes 0.2 FTE.

• Senior Geospatial Data Engineers are paid $160000 annually.

• On the GEE professional plan, the company pays a membership of $2000 a month. The company pays the full membership each year (i.e no stop-starting).

• The company runs global Sentinel-2 inference once a year.

• Global inference costs around $40000 - $160000 on GEE, and $15000 - $35000 on the custom pipeline in terms of compute + data transfer/storage.

Running this simulation gives us the following plot:

Its interesting to note that except that even with these fairly generous assumptions: that my extrapolation is off by a factor of 4, that the custom pipeline takes double the amount of time to set up and maintain, the cost of running this process of GEE very quickly outstrips the cost of the custom pipeline. This tells us that GEE is fundamentally mis-priced: since it is essentially a premium on compute + storage that one must pay every time we wish to compute anything, as opposed to the cost of the custom pipeline where the price to set the pipeline up is amortized over time.

The Math ain’t Mathin’

So how should GEE be priced? Well for one, Google could stop charging both a membership AND for usage. How do our plots look in the case where Google drops the monthly membership?

Much better, as we’re reducing the cost per year by $24000. Although the custom pipeline still appears to be cheaper in a lot of cases over time. Let’s pull another lever: assume Google decides to drop usage based costs by 50%, what do our lines look like in that case?

While there certainly appear to be benefits in the first couple of years, GEE costs eventually eat these benefits up over the long term. Of course at this point we’re probably splitting hairs, and its unclear if geospatial companies plan on 5-year timescales anyway.

Conclusion

GEE is expensive, if you use it as an imagery pipeline. Of course the magic of GEE is that you can do so, so much with it. Want to train a pixel-wise harmonic regression and then train a random forest on top of your coefficients? You can do that. Want to do massive zonal stats computations? You can do that. Want to run Landtrendr, or CCDC, with a few lines of code? You can do that. Dynamic world runs and is available on GEE, as is cloudscore+ and countless other datasets. There is additionally also the community-maintained awesome-gee-community-catalog which contains an absolute treasure trove of datasets.

Every single dataset on GEE can be seamlessly overlaid/intersected with every other. This means you can easily ask questions like: “out of all the pixels in dynamic world which were forest in 2024, and are crop in 2025, which ones had the highest peak MODIS NDVI in 2025” in a couple of lines of code, and just let it run. No need to handle resampling, overlaying etc…

There is so much value in being able to perform these operations, and GEE makes it so easy. However, if we want geospatial companies to succeed on GEE, which I think Google does as this means more money in the long run: the current cost appears to be high.

My unsolicited advice to the GEE/Google team: remove the monthly membership costs, and reduce usage based costs by 25-50%, and you will be a much more compelling platform for companies.

Subscribe now

• Make maps, not war (just) benchmarks

• I made the Santa Fe Quartet as a geospatial equivalent to Anscombe’s quartet

• If you are interested in building this kind of map-benchmark let’s chat!

Subscribe now

Benchmarks are not enough

In a recent post on on Geospatial Foundation Models, I used publicly available benchmarking data to show that they did not appear to be scaling well. This prompted some healthy discussion around how we should actually measure the utility of some of these models.

I think a fairly obvious answer is: along with model + benchmark releases people should also deploy their model over an actual area and produce a map. This would enable interested users to immediately gauge the quality and the behavior of the model in the wild (aka '“model vibes”). These is just something that looking at the data can tell you that metrics can’t. An additional benefit is: researchers get to feel the pain of practitioners and are more likely to develop models that produce good-looking maps in an operational setting, i.e more useful models. A final hope is that: the community converges towards an easy way to:

• take a model

• pull input imagery tiles

• perform pre-processing

• perform inference

• stitch the model inference/predictions back together into a map

All preferably for low cost, and fast :).

I should mention that many researchers already do this in the form of Google Earth Engine apps/ ImageCollections, and I think this is absolutely awesome. This definitely makes it infinitely more likely for me to use the model outputs because they are available and easy to overlay with other layers. I would however love to see a DOFA flood map deployed on the Pakistan floods of 2022 that I could overlay with a TerraMind one and a plain U-Net one.

The Santa Fe Quartet

Anscombe’s quartet is one of my favorite datasets: it demonstrates the importance of graphing the data one is analyzing.

The four datasets depicted below all exhibit the same performance when measured against the linear regression line:

With geospatial data, we have an additional privilege which is that we can also plot our predictions based on their locations on the globe. This allows a variety of different errors to occur, all which tend to be easily addressed and diagnosable if you just look at the map.

To demonstrate this, I pulled a dynamic world tile as an example along with the underlying Sentinel-2 imagery. We’ll simplify this land cover map to just be built area vs non-built area for the purposes of this demonstration. Then we’ll artificially introduce commonly seen geospatial ML map errors, with the constraint that all maps should measure similar f1 score/ intersection1 over union despite the very different underlying processes generating the artificial errors.

In this case we treat the original dynamic world built area map as our ground truth for calculating f1/IoU.

• For the first one, I chose areas that were non-built and introduced random ‘salt’ noise. This is a common type of artifact from pixel-wise models.

• For the second one I used a binary dilation to expand the existing built area predictions. This can happen if your model is not so hot at classifying edges/boundaries.

• In the third I generated artificial tiling artifacts at regular spacing. This can happen if you do not stitch your predictions back properly after tiling.

• For the final one, I chose another class from dynamic world (crops), and introduced random salt noise only over that class. This can happen if you have issues in your original training set, i.e not enough differentiation between crops and built-area in this case.

As you can see, all four artificial errors result in f1 scores of ~0.73 and IoUs of ~0.582. If we were to just go by benchmarks, who knows which model we would actually want to use! By looking at actual predictions, especially on a map, we can easily diagnose what major sources of error are and remediate them.
It goes without saying, types of geospatial errors are not limited to these four, and all four can produce different combinations too, but I hope this is illustrative of the need for more than just benchmarks when considering geospatial models, foundational or not!

• Just as GPT-style cross-entropy predicts language utility, Waymo shows motion cross-entropy tracks autonomous driving safety; we have yet to find the same appropriate objective for geospatial/EO.

• Waymo shows their objective scales with compute budget in a similar fashion to LLMs.

• We should think more carefully about deriving closed-loop-like metrics for earth observation to measure model utility, rather than model performance.

  Subscribe now

Autonomous Driving Scales

Waymo, the autonomous driving company whose burning cars became one of the enduring images of the Los Angeles anti-ICE protests, recently released a paper titled ‘Scaling Laws of Motion Forecasting and Planning: A Technical Report‘. In this report, the authors note that analagous to LLMs, the abilities of motion forecasting models also follow a power-law as a function of training compute.

The plot below shows this scaling for several different metrics:

1. Cross-entropy (open loop): this is the training objective of the motion forecasting models. It is the likelihood that the model assigns to a given ground-truth trajectory/token sequence. Essentially: did the model think that what actually ended up happening was likely?

2. Minimum average displacement error (minADE): this measures the geometric accuracy of the best trajectory generated by the model relative to the ground truth. With this metric we ask the question: how close to the ground-truth can the model get if allowed to pick it’s best forecast.

3. Failure Count: the model is rolled out in a simulator and the number of failures in N scenarios is counted. Failures in this case could be collisions, going off-road, hard braking etc… here the key question is: is a given model safe if its own predictions feedback into decision making in a high-fidelity simulator.

Incredibly, we see that all three appear to improve with increasing compute budget, albeit with Failure Count appearing to flatten with larger compute budget. This tells us:

• Next-motion-token prediction is a scaling-friendly objective

• Lower loss appears to translate into safer simulated driving

Closed Loop Geospatial Metrics

Open- vs. closed-loop metrics are less common in the geospatial world; Waymo’s report illustrates them clearly. As you can see in the image above, only the Failure Count metric is ‘Closed Loop’: this means that the system is evaluated in scenarios where its actions directly influence the environment, and subsequently adjusts its behavior based on feedback received from the environment. This crucially evaluates an aspect of model performance that Open-loop metrics don’t by simulating driving ability in real-world driving conditions.

Why don’t we evaluate geospatial models in the same way? There’s been some discussion on whether benchmarks such as PANGAEA are useful, and my personal feeling is that benchmarks are the bare minimum, but ultimately we need some measure of the actually utility of these models, preferably in an operational setting. The tricky part in doing this if of course you need to connect to your end-user, and evaluate how changes in your model might affect them!

Of course, since we’re not automating an agent that interacts with the environment in geospatial (yet), the nature of the evaluation changes a little. However, below I provide some best attempts at imagining what some closed-loop/operational geospatial metrics could look like.

An Example: Flood Mapping

The Sen1Floods11 dataset is a part of the PANGAEA benchmarking suite, and is a “surface water data set including classified permanent water, flood water, and raw Sentinel-1 imagery.” A company in the insurance space may be interested in your fancy new model which blows every other model out of the water (no pun intended) on the Sen1Floods11 dataset. How can we move beyond just pixel-level skill in flood detection towards a more realistic and useful benchmark for such a business? By simulating how this model would actually be used in the real world.

For the sake of this thought experiment, let’s assume a parametric flood policy where the carrier agrees to terms such as: “if > X% of the insured polygon is inundated, as determined by the S1 flood model, then trigger a pay out”. This type of workflow is shown on the website of flood mapping company Floodbase:

A good closed-loop benchmark could be:

1. Replay an unseen year of Sentinel-1 acquisitions, task the model with emitting a trigger for each policy polygon when extent crosses >X%

2. Evaluate trigger F1 score for your polygons (did we fire on the right events/areas).

3. Evaluate basis-risk error (BRE) as defined by:

where Loss is the best available estimate for the monetary damage caused by the flood, Payout is what the model-drive trigger says the insurer must pay, and Sum Insured is the policy limit.

Now we get to see not only overall model performance at a pixel-level (IoU), but how it propagates through operational pipelines and ultimately affects customers. In effect we not only evaluate the ability of a given model to capture the signal of interest, but how model performance translates to decision performance and alignment with monetary loss. Thus when a new model is released, we can easily run it through our benchmark, and evaluate how IoU performance actually relates to our (fictional) bottom line.

Conclusion

There’s a lot of issues with the example I gave above: how can model developers access payout information, or insurance information? How can we accurately estimate the actually monetary value of the loss of a given property? These are questions that I do not have the answer to, but posing the question forces us to link model outputs to utility measured in dollars. A researcher I spoke to following the Geospatial Foundation Model diss piece told me that research in the field appears divorced from practitioners and end-users. They should rest assured that they are not alone in this: practitioners and companies themselves are often equally as separated from their clients and stakeholders.

Many organizations claim that the purpose of geospatial/earth observation is to drive decision making: in that case, why would we not also model that decision making process alongside the physical process we’re attempting to measure?

• Models good, scaffolding gooder.

• Embeddings based search available to anyone with access to a laptop and who can run a jupyter notebook, coming soon.

• I’ll try to map Indonesian aquaculture with this workflow to prove it scales.

• Thank you Earth Genome for releasing data + code!

Model Scaffolding

In ‘AI’ world, model scaffolding is any post-training structure you wrap around a foundation model to guide or enhance it’s behavior. Prompts, memory stores, tool-calling or even calls to other models can all be qualified as scaffolding.

What is scaffolding for geospatial models? Embeddings-based search and human in the loop labeling and ML is an area that many have been investigating for a long time, yet is somehow still in my opinion under-explored. Humans are still state-of-the-art (SOTA) when it comes to end-to-end geospatial tasks like “find all the aquaculture ponds in Bali”, so any tool that can allow humans to inject their vision, reasoning and domain intuition is extremely high leverage.

Subscribe now

Geospatial search is a great example of geospatial model scaffolding: it lets you

1. Click a point on the map.

2. Use embeddings to surface visually similar locations nearby.

3. Label positives and negatives on‑the‑fly to spin up a training dataset.

   What happens if the user needs more information than is available in the basemap/imagery to make a decision though?

The Earth Genome, released a notebook for embeddings-based crop classification recently. I’m currently working on modifying it to work with any custom set of embeddings, but I wanted to share a great example of additional model scaffolding to answer this question.

Street View

In the video below, I show how really simple model scaffolding can enable the incorporation of external information: the user can directly open Google Maps at the location, and use Street View imagery to confirm whether what they are looking at in the satellite imagery is aquaculture or not.

In the video (sound on to hear) I narrate how I:

1. Use the scaffolding to open a street view tab in my browser.

2. Explore the area to confirm that it indeed contains aquaculture.

3. Label a positive and begin searching

4. Find other instances of aquaculture in Bali.

With a single click, humans can now look at ground-level imagery before labeling satellite imagery: a trick inspired by the classic QGIS Street‑View plugin and by Soler et al. 2023’s clever crop‑type mapping in Thailand. The key here is integrating these existing tools with embeddings-based search, which allows users to scale their vision and reasoning abilities as well as their domain expertise on geospatial data.

This is an extremely simple thing to implement (~3 lines of code), and I can personally attest to how high leverage it is having used it to build datasets and perform mapping experiments:

import webbrowser
center = Map.center
url = f"https://www.google.com/maps/@{center[0]},{center[1]},15z" webbrowser.open(url, new=2)

Conclusions

• Human‑in‑the‑loop remains unbeaten for nuanced geospatial tasks; tiny scaffolds like this one multiply expert time. You have to look at the data, this is the cardinal rule of geospatial data science.

• Low‑hanging fruit first: rather than chase ever‑larger foundation models (which still lag supervised baselines on pixel‑wise segmentation), focus on interfaces that scale human level vision + reasoning + experience with geospatial data. This is how we will get adoption.

• I’ll release the notebook & backend soon so anyone can build an embeddings DB and start mapping with a set of embeddings parquet files.

• ipyleaflet’s look‑and‑feel is … serviceable. If you know a slicker JavaScript or Python framework or hacks to prettify ipyleaflet drop me a line!

• A big thank you to the Earth Genome for releasing both the notebook and the embeddings.

Subscribe now