It has been intoxicating to generate interactive visuals in minutes that used to take me days. Take this visualization I put together for a project where we are designing a graph-based research platform:

Getting the graph layout with annotated labels and interactions in a just few minutes (with a few pokes in the right direction) is crazy enough, but using scrolling to turn into a sticky mini-map that can interact with the text took only a few minutes more. This kind of thing used to take hours and hours of coding, and now I don’t even read the d3.js its spitting out. I just keep prodding it until I see what I want.

Now that we’re basically walking around with magical genie lamps in our pockets it’s time to make all the explorable explanations we’ve wished for but never had the time to commit to. I’ve started a few series, going all the way back to my masters thesis on smooth particle hydrodynamics:

Or this series on dimensionality reduction:

Kai took it to another level generating dozens of explainers spanning emergent phenomena, parallel coordinates and group theory along side data visualization series exploring earthquakes in Japan or EV battery technology:

I have to be honest though, it is very easy to end up with slop writing. It’s been a bit maddening to get compelling figures in a couple iterations but then having to constantly yell at Claude for using emdashes and overly-dramatic language.

I noticed that while its consistently effective to hone in on the interactions and visuals I want to see by iteratively prompting the prose never really satisfies. That’s often ok when I’m just trying to read an explanation for myself in a more digestible format than raw markdown, but it’s never really something I feel comfortable imposing on someone else to read. In the moonshine examples that I iterated on the most I could have probably written the natural language faster myself in the time I spent prompting it to de-slop.

So I thought, what if I could still write for myself without having to edit the raw html the model is spitting out? Could I somehow have the best of both worlds by writing prose in markdown but still visualizing in full browser glory? I decided to try and the result is a new command /moonshine:still that builds a small React project that allows the article to be written in markdown but rendered with interactive figures.

This opens a lot of direct manipulation opportunities beyond just jumping in and fixing things. Because markdown is a bunch of text blocks, and the figures act as a block, we can easily rearrange the article with controls. We also can have defaults configured for figures that you can override while editing and save them. An extension of this idea is to make it so you can re-layout graphs the agent has made directly:

One of the more surprising capabilities that I ended up adding was the ability to comment on any block, and then have the agent address those comments in bulk. I found myself getting really annoyed trying to give specific feedback on different parts of the article and having to tab back and forth between my browser and my agent. The commenting system works with files and is not harness specific, it just makes sure all the context an agent would need is available with your comment.

I’ve written up the guide to all of moonshine’s features using none other than moonshine:

https://enjalot.github.io/moonshine/moonshine-guide/

It’s got a bit more detail on how these features work if you’re interested, but the easiest thing to do is just point your agent at https://github.com/enjalot/moonshine and ask it to make a still for you.

I hope you make something complex a little easier for us mere humans to understand, and if you do, please share it with me!

Latent Scope 0.6

While we were at it, we released Latent Scope 0.6 that brings several UX improvements to the explore interface as well as support for LanceDB behind the scenes. I’m happy with Lance because it both makes a compact and efficient way to share / publish scope data (it powers the API behind the live demos) and it improves performance of the local nearest neighbor search in the explore UI.

One of the biggest UX wins that Jimmy pioneered was a universal search bar, which allows you to search by similarity, select a cluster or filter by an SAE feature:

That was a key development to enable mobile support! We wanted the demos to work on mobile and took a lot of inspiration from Google Maps. I’m personally quite happy with the result:

Navigating by Similarity

Something that grew out of the new front-end code is an interactive article on Sparse Autoencoders:

Since training and deploying my first Sparse Autoencoder, I’ve wanted to try and articulate a vision for using SAEs to steer embedding-based similarity search. This article let’s you play with this new UI on the dad jokes dataset. It also tries to build an intuition for what Sparse Autoencoders are, how they work and what they let us do.

The power of filtering data by SAE features is already supported in Latent Scope, for example you can see dad jokes about “cattle and bovine animals” and I hope this article makes it more compelling to embed and process your own dataset with an SAE.

Designing with AI 2025 Talk

I’m also excited to share that I’ll be speaking about Latent Scope at the Rosenfeld Designing with AI 2025 conference. I’ll be speaking along side a very interesting mix of designers and researchers sharing case-studies about their work with AI:

Why I’m Excited About Embeddings

I highly recommend everyone reading this to check out Jimmy’s interactive article Why I’m Excited About Embeddings! It’s a very intuitive introduction to embeddings with some great analogies:

Thank you for reading, if you want to discuss any of these updates please join us in the Latent Interfaces discord!

Hidden States is an unconference for people who want to work with ML below the API. We had 50 people from a variety of backgrounds come together for a full day of discussions on everything from sparse autoencoders to agents and autonomy. I wanted to recreate the vibes from the d3.unconf days, where no one person is able to know everything in an exciting new space, but coming together with curious practitioners creates opportunities to connect and learn. It felt like we achieved that aim, thanks to the hard work of my co-organizers Casey and Stella and the humble, curious and phenomenal attendees.

The event built on the energy of the recent Embeddings UX meetup in New York, with two panelists kicking off the day with inspiring keynotes: Leland McInnes and Linus Lee.

Leland gave a wonderful talk about the value of exploration and the challenges in trying to communicate high-dimensional data. He showed tons of interesting applications of UMAP across many domains, some of which you can see in the responses to this tweet. You can get a sense for the breadth of applications from the examples showcased in the UMAP docs including interactive, exploratory analysis and scientific papers.

Perhaps my favorite example from the talk was this paper mapping mathematical formulas across scientific domains. Learning

Linus gave a thoughtful prompt on representations and interfaces, covering a depth and breadth of perspectives that was very motivating. He considered the variety of representations humans have used to communicate from language to data visualizations as well as the emerging machine-friendly representations (hidden states!). Tying these together he encouraged us to think about the interfaces we have yet to design, looking at the ways direct manipulation and agency have been incorporated into all the tools and instruments we already have. We weren't able to record either of the keynotes but Linus does intend to publish the materials in some form, which I will be very excited to share once he does!

After the keynotes we had a full day of talking to each other, from big discussion groups on explorable explanations or agents to small groups chatting over lunch or in the tea lounge. The lightning talks were also a great hit, with 12 demos in 45 minutes we got to see many cool ideas and prototypes in rapid succession. The one thing we could have done better is coordinate the discussion topics more effectively. We either ended up with two big groups or dozens of tiny groups, and there is room to do a little more digital coordination for folks to find each other. Out of the chaos some great ideas emerged! We had a few "speed dating" sessions where people had 90 seconds to talk to someone nearby they hadn't met yet as well as splitting everyone into 3 groups and hosting essentially group therapy. We prompted folks to share their greatest challenges in their work, which my group agreed included developing UI (struggling with javascript and the browser).

Given the good feelings we got from finding a community of curious and kind people, we will certainly want to do it again! A not insignificant amount of people traveled from the East Coast to attend, so we may host the next iteration in a new place. Some feedback we got was that people wanted more time together, perhaps learning via workshops or hacking together. In the heyday of d3.unconf we had a 3-day event with workshops, discussion and hacking in that order. Perhaps this community will grow to such a gathering, though we always capped the attendance at 100 to keep the communal feel.

If this sounds like something you would like to be a part of in the future, fill in the registration form on hiddenstates.org and you'll hear from us as soon as we plan the next one!

Latent Scope 0.5

When I applied to the Mozilla Builders program I had to organize my thoughts and put together a plan for a focused iteration on the project. I realized that there was a lot of functionality exposed in Latent Scope but much of it felt duct taped together. The main goal, catalogued in this github milestone, was to refactor the UI and make the existing functionality more accessible and more robust. Thanks to help from the Mozilla team and huge contributions from Jimmy, we accomplished quite a bit!

Some notable changes to the UX include a redesign of the Explore page, allowing the map to take up more space (and even dynamically resize the width of the table to give it even more prominence). The filtering system has also been cleaned up and streamlined, making it a bit simpler to reason about what is being filtered. We decided to cut out editing / curating features for the time being to focus more on the act of exploration, we felt that getting exploration right is a precursor to further editing or annotation.

One exciting new way to filter is by Sparse Autoencoder feature, if you've embedded your data with nomic-embed-text-v1.5 you can process it for SAE features from latent-taxonomy! We have more UI improvements on the way to visualize and process these filters, but I'm excited to incorporate SAEs as a first-class feature.

We also made a bunch of improvements for the Setup process, aiming to make it as straightforward as possible to go from text dataset to exploring. A big way we accomplished this was to make the process go step-by-step, dedicating the whole screen to each step one at a time. In this way there is always a preview of your data relevant to the setup step you are on.

We also made the cluster labeling functionality much more robust. You can search for chat models form HuggingFace, you can choose from any ollama models you have installed and you can even configure a custom url to hit any OpenAI compatible endpoint like a llamafile server. We also make it easier to control how many tokens you're using by choosing the number of samples to summarize, and they are sampled by distance from the center of the cluster.

One last thing we added is the ability to upload and download scopes via HuggingFace datasets. This feels exciting to me because it made it much easier to share what I found while exploring new datasets. It was immediately useful in setting Jimmy up with good example datasets, and now I can finally share my dad-a-base!

What's next?

Right now we are actively working on improving the SAE functionality and making it easier to browse and filter by features. The next major functionality is supporting images and image embedding models like CLIP and SigLIP. I'm also looking into switching the backend from parquet files to lancedb files as it looks like it would improve both image storage and embedding storage as well as provide easy and performant nearest neighbor search. Lastly, we really want to have some kind of deployed version of the explore experience to make it even easier to share scopes!

We'd really appreciate any feedback on Latent Scope, you can respond directly to this email or fill out this form anonymously to help us improve in the right latent directions: https://forms.gle/8Kbm1xi44mqc2ojZ7

Thank you for reading!

P.S. Our discord has been growing as well with #hiddenstates, #latentscope and #sae channels, join the chat.

This is an exciting time for embeddings, perhaps you know them as hidden states, neuron activations or latent vectors. Whatever you call them, they are starting to get their moment in the sun. The idea that we can create new user experiences by directly working with hidden states isn't exactly new, but it is taking off with researchers and startups in some exciting ways. I've got some new research developments to share in this space, but first I want to introduce an event we're putting together tomorrow in New York:

Embeddings UX NYC

Wednesday, Sept. 25th at Nomic HQ w/ Modal Labs & Latent Interfaces

We're pulling together a panel discussion on the latest in user interfaces for embeddings with 5 people doing state of the art research and prototyping in the field. I'll briefly introduce the panel with links to their work, as that alone is a good overview of where things are going:

Leland McInnes is a researcher at the Tutte Institute and well known for his work on UMAP, HDBSCAN and DataMapPlot. I'm also excited to see his nascent EVoC project optimized for clustering embeddings.

Linus Lee is a researcher and prototyper who just moved from Notion to Thrive Capital. His work on Prism has inspired me (and many others) to dig deeper on Sparse Autoencoders for better UX with embeddings.

Adam Pearce is a veteran ML visualization developer, formerly at Google Research and now at Anthropic. Adam's contributions to Scaling Monosemanticity (i.e. the Golden Gate Claude paper) introduced state of the art visualizations of Sparse Autoencoders on LLMs.

Yondon Fu is a co-founder of Livepeer and has been doing very interesting prototypes of new user interfaces like Feature Packs. These prototypes focus on image models, demonstrating visually just how compelling Sparse autoencoders for embedding based UX can be.

Andriy Mulyar is a co-founder of Nomic, building not just a new kind of UX via Nomic Atlas but also training SOTA embedding models and releasing open source libraries like deepscatter.

As you can imagine, I'm incredibly excited to hear from these folks and I have the privilege of moderating the discussion. While the event won't be recorded, I will be writing a follow-up with my learnings from the event. I also want to do more of these, in San Francisco as well as online, so let me know if you'd be interested in the next one!

New research: Sparse Autoencoders

You might have noticed a theme on the panel: 3 of the 5 have done work with Sparse autoencoders. SAEs are a powerful technique for examining the inner layers of LLMs. Recently, I trained my own Sparse autoencoder on the nomic-text-embed-v1.5 sentence embedding model. 

The experience was illuminating, as it was the first model I've actually trained from scratch, and I've done my best to write-up what I did and what I learned in this detailed article.

Why Sparse Autoencoders?  They allow us to try and tease out unique directions in the latent space of a model in an automated way. You may have heard of style vectors (this distill article on augmenting human intelligence is my favorite articulation of the idea) but that usually involves a lot of manual work finding each vector. SAEs let us find thousands or millions of directions at once. Then of course the trick is interpreting (i.e. labeling) those directions, where the current best practice seems to be asking an LLM to summarize data samples that highly activate for each direction. Once you've produced and labeled the SAE features for a given model, you can then use them as a lens to look at your data.

I'm working towards integrating this technique into latent scope in several steps:

1. Training SAEs on popular embedding models, I've already started with nomic-text-embed-v1.5. This work is being done in the latent-sae repo.

2. Interactive taxonomy of an SAE model, for reference and downstream uses of the SAE. This work is being done in the latent-taxonomy repo. You can play with the 25k features currently supported in this interactive webapp.

3. Extract SAE features of a dataset directly in Latent Scope, this work is being tracked in this issue in the latent-scope repo.

It will be exciting to be able to filter your datasets by a specific feature, like stress related to buttons, plant identification strategies, or cybersecurity vulnerability detection. The breadth of concepts captured by the SAE from embeddings is impressive, and this is just the small version! I'll likely do another newsletter soon about all of the things I'm experimenting with around SAEs as the technical and design challenges are numerous.

Join the discussion on the Latent Interfaces discord or just reply to this email if you want to chat about anything related to embeddings UX!

We're all generating and collecting enormous amounts of data, yet much of it remains unexamined because looking at it is unfeasible with traditional data analysis techniques. Recent advancement in LLMs, particularly in embedding models, have unlocked a new way to pull understanding from unstructured data. Putting these new capabilities into practice is fraught with operational challenges and the constantly evolving model landscape. That's why I've been building Latent Scope, an open source tool that streamlines the process of getting insights from unstructured data.

To demonstrate this process I've published an example analysis of 50,000 U.S. Federal Laws since 1789. One insight that immediately jumped out is that there are neat clusters of ~3,000 laws about building bridges, ~2,500 laws concerning Native Americans but surprisingly there are no obvious clusters of laws about Civil Rights.

In this way, Latent Scope can act as a kind of observability for embedding applications. Without being able to visualize the clusters and the search results it would be very difficult to know how well represented (or not represented) various concepts are in my dataset.

Combining structured and unstructured data
I put together an example analysis of my 10,000 tweets over 16 years on Twitter to highlight the potential for augmenting existing data with the meaning extracted from unstructured text. While it may not be a surprise to anyone who's been following this newsletter a long time, my most liked tweets revolve around D3.js and data visualization. The area of the map that has the most overlap between tweets, retweets and replies are all clusters related to D3!

By combining the structured data with the extracted categorizations we can do things like see which topics got the most likes and retweets. It's usually very difficult to act on metrics alone, there is always some context missing, now it's possible to get the context systematically! 

You can run latent scope on any decent computer, many of the open source embedding models can process thousands of rows per minute with just a CPU. It only took me 5 minutes to run my 10,000 tweets through the entire process end-to-end on my M2 mac.

Latent Scope is now on version 0.2, with a simpler setup process and a more robust and unified interface for filtering and curating data. The export format has been iterated on to make it easier to incorporate in other analysis tools. There is now more thorough documentation with a guide to your first scope and how to use the explore & curate page.

Why does Latent Scope exist?

I've developed Latent Scope to systematically do what I've done in an ad-hoc way at my previous jobs. I'm excited about this process but I'm tired of reinventing the wheel! 

I've been doing data engineering & visualization professionally for the past 20 years and the mandate is always the same: impose structure on data to highlight patterns and extract insights. The discipline is well developed and I've always enjoyed the challenge of encoding data into visualizations, but then in 2016 I felt the ground shift when I learned about the concept of latent space. 

I was introduced to t-SNE by building the interactive visualizations for the popular Distill article about the algorithm. Getting a feel for the power and flexibility of t-SNE, and soon after, UMAP primed me for the revelation of embeddings. Before t-SNE I had to manually figure out which features of a dataset were interesting enough to show, but with this new tool I was able to automatically organize data in interesting ways.

Then the Distill researchers patiently taught me about latent space, we explored embedding visualizations of a Handwriting model and then visualized neural network activations in The Building Blocks of Interpretability and Activation Atlas. The very thing that made neural networks so effective was captured in their hidden layers and it turned out that if you could look at what was in those layers you could get insight into what the machine had learned. 

During that time I was helping curate training data for SketchRNN and then contribute to open sourcing the QuickDraw dataset. When I applied t-SNE to the SketchRNN embeddings of the drawings I became fully convinced of the power of Machine Learning for Data Visualization.

Here was a dataset that defied traditional data visualization techniques, sure you could organize it by what country it was drawn in, or how many strokes were in each drawing, but to get at the actual meaning of the drawing was out of reach for traditional tools. With embedding visualization it was possible to organize the data by its actual content, finding clusters of subjectively good drawings. We also got an insight into the unexpected ways people were playing the QuickDraw game!

In 2018 I began applying this idea of visualizing embeddings with some colleagues on Google Cloud. We built out a suite of tools for making sense of usage patterns for hundreds of Google Cloud products, finding patterns in user behavior and building out data-driven personas. While I can't share too many details we did find surprising insights into how our users were combining product offerings in ways our product teams had not foreseen. 

It was exciting to see a valuable business application of the concept, but at the time it took quite a lot of resources to execute on the backend and I wanted to focus on building data interfaces.

So in 2020 I took a small detour from embeddings and joined Observable to help make advanced data visualization more accessible to teams. There I continued to curate data and refine my visualization practice until the end of 2022 when I felt it was time to strike out on my own.

I got back to embeddings with some work for Stability.ai, helping curate training data for iterations of Stable Diffusion. Re-implementing the process on a new infrastructure reinforced the concepts but highlighted the drag of having to reinvent the wheel. It was also exciting to see the rapid developments of open source models like CLIP and StableDiffusion with capabilities blowing past the VAEs, RNNs and CNNs I had been working with in 2017. While I was focused on curating good data to improve models, I stumbled upon a dramatic insight. Perusing the clusters of user generated images, certain disturbing images showed up. Because of this we were able to rapidly identify, classify and put a stop to an influx of harmful image generations.

What's Next?

The next step for Latent Scope is to take this process into a production environment where data is being underutilized because its value isn't accessible. I'm especially interested in datasets that grow over time. Some use-cases I can imagine are:

Curating training data for fine-tuning domain-specific models

We all know the adage: "garbage in, garbage out", but how do you know what's in a multi-million or multi-billion token dataset? I'd like to use Latent Scope to generate detailed reports on dataset composition and quality. The outcome would be actionable ways to augment the training data.

Curating knowledge bases for RAG applications

Similar to curating training data, knowing the composition of your dataset will greatly impact the quality of your RAG responses. If certain concepts or documents are underrepresented, the LLM won't have the context it needs to answer queries. We should be able to systematically determine the document distribution and create an actionable report for addressing deficiencies.

Insight discovery in low signal-to-noise datasets

Data may be the new oil but a lot of it is very crude. That doesn't mean we can't process it to find the good stuff. I'd like to use Latent Scope to sift large unstructured datasets to extract meaningful data. I'm especially interested in training classifiers based on what we find to filter production data as it comes in.

In my experience being able to see what's in the data that's fed into production processes will always generate surprising insights, then after the surprise wears off you gain a new sense of confidence in the process.

I also know that in real-world workflows a tool like Latent Scope is going to be a part of a larger process and integrating it will take some customization and adapting. My hope is to find a client that I can work closely with to add value to their data and make Latent Scope more robust in the process.

If that sounds like you or anyone you know, please get in touch!

I am excited and a bit nervous to share a project with you that in some ways has been years in the making. Since I learned about the power of embeddings and their potential for understanding data I've been trying to refine a combination of ML and data visualization techniques for exploring large unstructured datasets. I feel I finally built the tool I was missing in my explorations, a workflow that supports experimentation and paves a path through the dizzying array of choices in this fast moving environment.

I give you Latent Scope!

https://github.com/enjalot/latent-scope

I want to start by sharing what it does, because there are two important sides to it: the python-based data process and the web-based interactive visualization.

Exploring

The goal of the process is to get your data into the interface for exploring your dataset. The main idea is to be able to zoom in on interesting subsets of your data while staying in the context of all of your data. Interesting subsets can be similarity search results, the "retrieval" part of the popular RAG technique. They could also be the clusters determined by carving up the latent space, giving you a sense of how well represented various concepts are. Even just zooming in, clicking around and selecting data points can be an enlightening exercise because you stay in context as opposed to just pulling random values.

You can try exploring some demo datasets, and then read on to see how to run your own data through the process to explore it.

Trust the process

The data process captured in the tool is a series of scripts that help you ingest a tabular dataset (CSV, parquet or pandas dataframe), calculate embeddings for a text field from each row, run UMAP on the embeddings, cluster the UMAP points and optionally label the clusters with an LLM. The scripts can be run from the web UI or individually via the command line.

While the steps are linear, at each stage you may find yourself wanting to do it again with a different choice. You may want to embed your dataset with multiple models to see if they capture more or less meaning. Experimentation is key, you may want to play with the UMAP or HDBSCAN parameters, as there is no predetermined best choice. When you experiment you want to keep track of what you've tried, you want to be able to back track and go down a different path.

Latent Scope was designed to facilitate this, each run of each step has both its output and the metadata capturing the choices made written to flat files. It should be easy to see what you've done and it should be easy to pick up the process from any point.

Another important aspect of the process is that all of the steps are using indices into the original input dataset. This means we don't modify the input dataset and there is no possibility of copy errors. It also has some handy UX implications I'll get into in a future update!

Getting Started

The README has detailed instructions, but it really is as simple to get started as:

pip install latentscope
ls-serve ~/latent-scope-data

Try it out locally with a bunch of Transformers models like Jina v2, Nomic and e5 or setup API keys for popular services like OpenAI, Mistral, Cohere, Voyage and Together.

Just the beginning

I wanted to build this as an open source python module because I'm hoping that we can have a solid foundation for processing messy unstructured data into structures that facilitate exploration and understanding. I've tried to make it as easy as possible to install and use, and have the output be standard and portable. I have a whole bunch of things I still want to build, so I've started listing them as GitHub issues, and a couple would be great starting points for anyone who wants to get involved.

If you want to chat about using it, or need some support getting started you can now join me on the Latent Interfaces discord!

I’d love to hear from you!

P.S. Some Thanks & Acknowledgements:

This isn’t exactly a scientific publication but I have received a lot of support that enabled me to put this project together.

I need to thank Erik Hazzard for tons of feedback and encouragement. My wife Agnes for endless encouragement and patience. EJ Fox for early prototyping sessions and especially the idea to summarize clusters before I saw anyone else doing it. David Ha, Shan Carter and Chris Olah for patiently teaching me the foundation for everything I know about ML when we worked together years ago.

I had been working for 24 hours straight... so I decided to call it a day.
find more in the dadabase

I am embarking on a new phase of my career, late last year I left my full-time job and reflected on the kind of work I’ve been doing for the last 15 years. What I do is make data interfaces and I’m launching a company to explore the possibilities of applying my accumulated experiences with design, prototyping and development to sophisticated data problems. I call it, Latent Interfaces.

I’d like to dig into these possibilities of applying my process to sophisticated data problems. If you’re reading this you probably agree with the adage that a picture is worth a thousand words, and a prototype is worth a thousand meetings. So I’ll share some visuals and prototypes from a couple of projects I’ve gotten involved in recently that outline my process for how I'm approaching data problems with my company Latent Interfaces.

Mapping the Human Genome

Recently I've had the opportunity to be involved in a fascinating research project whose goal is to create intuitive interfaces for massive amounts of genomic data. This research is led by Wouter Mouleman at the Altius Institute and it is a prime example of the type of problem I want to tackle.

While we are still in the early stages of development, I've already gotten to push every aspect of my data interface development practice. Whether it’s designing a new file format to efficiently access data mapped to anywhere on the human genome or prototyping new interactive visualization techniques, we are at the forefront of a new way of seeing our own DNA:

Hello, Python

In the process I’ve been refreshing my Python skills, something I now see as an essential complement to my JavaScript skills. Being able to process larger datasets into manageable pieces for the browser has been game changing. Of course, you can’t write a newsletter in March 2023 without mentioning ChatGPT, and it has been incredibly helpful speeding up my orientation in Python land:

Design by a thousand papercuts

Mapping the genome isn’t just math and algorithms, it’s also about creating a place that people can visit. It’s about orienting yourself amidst torrents of data and being able to share a perspective that you have found. I realized that I couldn’t design just starting at the screen, so I printed the genome out on cardstock and cut each chromosome into a puzzle piece:

This exercise gave me an intuition for laying out the map and convinced me it would be worth it to make a digital tool for adjusting the layout:

While the layout is not yet finalized (this actually turned up new concerns we now need to address) I wanted to share this work-in-progress because it was both an effective way for us to move the design forward and it was a lot of fun to do!

Data Architect

I want to take a moment to plug my new favorite YouTube channel by Eric W. Reinholdt, an architect who documents his process beautifully. I feel there is a lot for data design professionals to learn from architects, and Eric’s videos are a free master class!

APIs need design too

One of the things I came to appreciate working at Observable is a well designed API. I got to see Mike Bostock and Fil design and build Plot commit by commit, and the design of that API is incredibly thoughtful and ergonomic, it's like a data visualization glove that glides right on to your brain and makes you faster and stronger without even thinking about how to use it.

Then there is the other kind of API, the (hopefully) RESTful kind that lets you fetch data from somewhere and offloads storage and processing to the server allowing you to focus on frontend interface development. A well designed API increases the potential energy of an Observable notebook by an order of magnitude. And I am still constantly making Observable notebooks for my clients.

StabilityAI API

The StabilityAI API is currently in beta, and you can use it to generate StableDiffusion images without needing a fast GPU in your computer. I’ve taken some time to familiarize myself with it and made a couple of notebooks that demonstrate how to call it from the browser. The first is a simple Text-to-image example which lets you write a prompt and submit it for generation. It’s a basic REST API call powered by a form.

The second notebook is as bit more interesting, as it enables Image-to-image generation:

It’s interesting because in order to correctly call the API you need to submit an image in the right format (binary data) at the right size (width and height must be a multiple of 64). The binary part isn’t so bad, as HTML5 Canvas provides a canvas.toBlob() function, but the code to resize properly is a bit gnarly:

I had translated that code from a Python example and was curious to get an intuition for what it was doing. I always get nervous about resizing images and maintaining aspect ratios so I figured I’d make a little interface that let me simulate different image dimensions and see what the results would be:

You can play with it yourself near the bottom of the Image-to-image notebook.

Latent Interfaces

As you can see, these data interfaces can take many forms and involve a wide variety of tools and techniques. I’m excited to share more work in the coming months, especially a project I got to collaborate on with my main man  EJ Fox and the data visualization studio he's starting!

I’m also contemplating making some YouTube videos that go into more detail and it’s extra motivating when people let me know. So please reply to this email if you want to see a more detailed video on any of these.

Thank you

I'd like to thank my close friends EJ Fox and Erik Hazzard for feedback on the newsletter and support in starting this new venture. I'd also like to thank my wife Agnes for her unyielding support and priceless insights.

First though, I want to talk about decisions and how we think about them. I also want to talk about what I don’t hear as often: how we feel about them. More precisely I want to talk about how we think and feel during decisions. When we are making a data visualization we make tons of big and small decisions by the time we publish it. How often do you think about how you feel while you are making those decisions?

https://observablehq.com/@enjalot/emotion-tree

How many times have your groaned in your head “ugh, this is going to take forever” when you realize you need to reshape the data? That’s feeling overwhelmed, maybe bored and a little scared. What about those times you say “woah, I didn’t expect to see that!” That’s joy, curiosity and inspiration. There’s always the crunch-time “why did I ever agree to do this project!?!” which is of course feeling stressed and frightened.

Then there are the countless moments that pass without intense feelings but are nonetheless felt. Choosing a higher contrast color scheme, feeling the difference it makes to your eyes. Choosing the formatting and font of the axis tick marks and feeling the slight comfort of additional context. Choosing the sort order, the statistical metric, to filter or not to filter the outliers and missing data, feeling the uncertainty as each choice is made. These are the choices of our profession, which are often discussed in terms of perception and statistics, correctness and effectiveness. I’d like to focus on how it feels to make these choices.

In September of 2011 I made my first “live coding” video about D3, a 40 minute quest to make a bar chart coded in the terminal. I don’t recommend watching it 10 years later, for one thing we are 5 versions further along with D3. I did skim through it though and realized that it does highlight the cost of all of those decisions. As I narrate my thought process I need to type quite a bit of code to enact a decision, if I get it at all wrong I need to stop and think about the implementation. Then I plod along trying to implement a series of decisions heavily based off prior experience and conventional wisdom, rather than the dataset in front of me. All of that feels frustrating.

Today in 2021 I can make an incredible amount of decisions about a bar chart almost at the speed of thought. The implementation of D3 has been abstracted into heuristics and defaults in Plot. Here is the same bar chart remade in less than two minutes:

https://observablehq.com/@enjalot/simple-bar-chart-plot

This feels good, it makes me eager and energetic, I am often awed by what pops up rather than anxious about what is causing my visualization not to show up at all. I get to be surprised and curious because of how quickly I can try out a different decision, I can be confident in my trials instead of nervous about my implementation. If you’re interested in the technical differences that enable these different feelings, I recommend Toph’s excellent Plot for D3 users. If you’re curious about how these improvements were arrived at I recommend starting with Mike Bostock’s 10 years of open source data visualization.

The thing is, Plot isn’t done and neither is Observable. Ive been on my own quest to pursue these feelings while I create for a long time, and it continues. For me it started with Tributary, when EJ Fox and I saw the Bret Victor Inventing on Principle talk in 2011 it opened our eyes to a new way of thinking. We realized we could build tools to improve the way it felt to code, to learn and to design.

This was fun and educational time, we kept adding features and fun things that let us try new ways of making. After a year or so though, the codebase itself got too complicated for me to think about and I would feel dread anytime I wanted to fix or change something. I stopped being able to add to it. At that time I started working for Lever as a “full stack” engineer and learned a lot about product development and software engineering. These skills enabled me to think more clearly about building software and they came in quite handy when I decided to start Blockbuilder in 2015:

At this time bl.ocks.org had grown as an incredibly powerful and popular resource for learning D3. Creating and consuming examples was (and is) one of the best ways to learn a technology with such combinatorial possibilities. Creating examples at that time required quite a bit of comfort with the command line, an environment and way of thinking that many people coming to D3 were not familiar with. Using the same technical underpinnings as Tributary I was able to make an in-browser editor compatible with bl.ocks.org.

This felt great, people who wanted to make examples, or try messing with existing ones didn’t have to think about the process of saving, editing and deploying them. They could just do. As with Tributary though, with time, the weight of software development built up and I was no longer able to move the project forward.

This past weekend after 5 years of serving the D3 community, and a couple of years of barely keeping it running I finally decided to put it to rest. Of course it’s open source, and we’ve archived over 40,000 blocks that were indexed by our search engine, but it is no longer running as a service. This makes me feel sad, I’ll miss what this phase of my journey. I’m grateful that I don’t have to miss the people that it connected me with or the ideas it gave me. I’m already feeling a bit lighter as I move forward in my current phase where I get to work with a phenomenal team on helping people think with data. I feel excited about that.

I’d like to hear how you feel. Let me know <3

I’ve put together this outline of the material, but what I’m really excited about is this student workbook that I’m hoping folks in the workshop will fork and fill out as we go.

The goal of the workshop is to give a solid foundation in web-based data visualization with d3.js along with a taste of building more advanced animated and interactive visualizations.

With Observable it will be easy to avoid a ton of painful setup, allowing us to jump straight into visualizing rather than worrying about local servers, local data, file paths, coding editors. I’m hoping that the starting points I chose strike a balance between providing motivation and providing enough context. D3 covers a ton of underlying browser technologies as well as a programming language many in the audience may not be super familiar with. I’m hoping we can cover enough basics to keep moving while leaving a trail of breadcrumbs for deeper exploration when folks want to apply this to their own data.

Test Run: October 9th

I’d like to run a test of the workshop Friday, October 9th at 1pm EST. If you’re interested in spending around 3 hours learning some d3 and giving me feedback I’m hoping to get a small group of 5 folks to try it out together.

💌 Reply to this email if you want to join! 💌

Video Process

I’m getting really into making and editing videos lately. I’ve always enjoyed playing with motion, it’s one of the reasons I love using d3 so much. I consider myself an amateur video editor, I learned basics of Adobe Premiere many years ago and occasionally play with those skills. Lately I’ve been using LumaFusion on the iPad to edit timelapsed screen recordings. You can see the result in the workshop trailer at the top of this email.

Here is another attempt I made where I did a voiceover as well. Doing the voiceover is actually way more work than everything else combined, and pretty hard to get right.

Instead I’ve been putting music from friends behind the edited down videos. EJ Fox actually scored the beat to the workshop trailer in Logic:

EJ deserves extra shoutouts for giving me a ton of feedback on my edits, and encouraging me to hand draw the thumbnail. It’s really nice having someone on a similar creative wavelength to resonate with!

I’m hoping to make more videos like this! It’s been really interesting to expand my expressive capabilities into video and motion. I’m making more gifs and small videos, for everything from data visualizations to my son Maxwell’s drawings:

I’ve begun collecting quite a few ffmpeg commands that feel like superpowers:

If anyone is interested I can share these somewhere more accessible.

Coding with Fire

EJ and I are still streaming Coding With Fire every Sunday! You can see the results of past streams in this collection of notebooks. The more we stream the more we are figuring out what works for us. We are excited to be having more guests come on and build with us. Tonight Colin Megill is joining us to visualize data around the US Senate.

Next week Fabian Iwand is going explore programatic animation with us, look what kind of stuff comes out of his mind!

Follow us on Twitch to get notified when we go live!

It’s been a little over a month since I joined Observable and I’ve got a few irons in a few fires already. The one I want to think a bit about today is live streaming and moving meetups online.

There are two streams I’m working on: Coding with Fire and Observable. I’ve learned a bit already from both, and feel like things are really getting interesting. At the same time we recently had a pretty awesome d3 meetup with hundreds of folks from all over the world.

It has me thinking, what is the difference between a stream and a meetup?

A key aspect to the success of the recent meetup was the active chat: folks shared the cities and countries they were tuning in from, questions were asked and answered and people shared their recent work. Chat is something that has been integral to the game streaming experience for a long time, and streaming about code and design is no exception.

A big draw for a meetup is both hearing from and having the opportunity to connect with the speakers. The recent d3 meetup had a pretty exceptional lineup with talks from Mike Freeman and Amelia Wattenberger and an AMA with Mike Bostock. Of course this made for a big moment, and the excitement showed up in the chat.

It helped that the everything about the event was made possible by fellow Observable colleague Wayne Sutton. Until now my experience as a meetup organizer was always in a hobby capacity, learning from someone who has professional experience bringing people together has been illuminating.

I’m also excited for streams like we are seeing from Shirley Wu, sharing her individual process. Anytime you watch you can see that Shirley is consistently creating a space for folks to connect and learn, not just with her but with each other too!

All this has given me some thoughts about streams as “venues” where events can happen, and am looking forward to trying it with some future d3 meetups.

Coding with Fire Stream

I really can’t talk about streaming without proper deference to my good friend EJ Fox. He is the producer, director and host of Coding with Fire.

Each stream we make something. We’ve gone in several different directions and each time it challenges my skills and my tools. EJ and I have been on the same vibe since we met at visual.ly and hacked on Tributary almost 10 years ago. Having a live-coding environment of Observable’s caliber finally enables us to explore the kinds of ideas we have in a way that’s interactive with folks who are tuned in.

I think it’s fun to do and entertaining because the dialogue we have is just as important to the outcome as the code. We can have a design discussion and try things out at almost the same time, which means we can also take in suggestions and challenges from people in the chat. We are continuously iterating on the format and subject matter, so be sure to tune in!

Observable Stream

Ive been humbled to stream alongside Anjana Vakil on the newly established Observable Twitch stream.

We’ve done one stream already and are doing another tomorrow on building d3 visualizations in Observable. It’s funny because the format of the stream is that I’m teaching Anjana how to make a particular d3 visualization, but I’m the one learning so much from working with her… clearly communicating highly technical content in a way that feels like fun is hard work. Near the end of the first stream it was so gratifying when we finally got a slick transition going on the scatter plot we built and the crowd cheered!

There’s going to be all kinds of cool stuff on the Observable stream soon. If you’ve been wanting to try Observable but are feeling in any way shy, Anjana also made an awesome set of tutorial videos to get you started.

I’m really excited for tomorrow’s stream, tune in here and say hi!

Blockbuilder

This week I want to focus on a story that is coming to a close: Blockbuilder.

Almost 5 years after the Blockbuilder's successful Kickstarter, I’ve made the hard choice to acknowledge that it is time to sunset Blockbuilder so I can focus on other things.

By the end of the year Blockbuilder will become a searchable archive of blocks. I’ve written up a detailed sunsetting plan with context and next steps, so in this post I will be focused more on my personal perspective and how I worked on it.

Blockbuilder was built on the concept of using GitHub gists as the database for d3.js example established by Mike Bostock’s bl.ocks.org. I saw so many awesome d3 examples as blocks, and got a lot of help from the community by making my own. It became clear that it needed to be easier to make them.

The pitch was a simple in-browser editor that handled editing, saving and forking the gists for you. I mocked up a prototype and setup a kickstarter, and the response was exhilarating.

The funding goal was blown past and almost two hundred people backed the project. I felt a lot of pride and a lot of responsibility.

Since 2015 thousands of people have made tens of thousands of blocks. The community quickly recognized that with all of these new blocks popping up, it would be great if they could be searched. But since there was no central database of blocks I set out to put one together.

Inspired by block explorer by Irene Ros I collected public blocks everywhere I could find them and pulled out all of the usernames. Using the GitHub API I was able to download and process all of the public gists of those users (code repo). This allowed us to index those blocks and make them searchable.

image credit: Nadieh Bremer

The project is open source and I can’t thank the contributors enough, with special thanks to Erik Hazzard for setting up the initial React app and Micah Stubbs for numerous improvements to the UI and search functionality.

Between work and life I haven’t been able to devote the attention and leadership I would have liked to give the project recently.

It became more apparent that for Blockbuilder to continue to add value as an in-browser code editor it would take a team effort to build out. Meanwhile Observable has introduced a paradigm shift for in-browser coding with a jaw-dropping team behind it. A team I’m incredibly excited to join!

image credit: Micah Stubbs

Blockbuilder still offers something pretty unique, the searchable index of 40 thousand blocks. I plan to keep the search alive indefinitely as well as make it easier to work with the underlying data. I’ve made an observable notebook that can be used as a starting point for programmatically working with the data. I’m planning to upload a few different flat-file views of the complete database to make it easier to visualize the entire collection in various ways. I’m sure there are very interesting things we could learn from visualizing such a large amount of work done by the community!

My main regret is that I didn’t have enough conversations with the large number of folks who have used Blockbuilder in their work, teaching and learning. I’d love to hear if you used Blockbuilder in your own journey. If it’s still an essential tool for you I want to talk about the best options for your workflow.

You can reply to this email or join us in the #blockbuilder channel on the d3.js slack (which also has an #observable channel).

Either way, I always like hearing about what you’re excited about in data visualization, so reach out!

You have been temporarily transported into my perspective and I can only hope you will enjoy the view. My goal is to show you what I see, and how I get to the vistas that I find. It’s a process, a journey sometimes, many small steps and seemingly unrelated connections that come together to form windows into the world that would otherwise not be.

Four years ago I joined Google to see what I could see from the vantage point of one of the largest computer systems on this planet. I learned a great deal, which I’ve spoken about, published and helped to open source. Now I’m moving on, with an intent to focus on helping others obtain the skills, tools and resources needed to create their own windows into the world.

I’m going to take a couple weeks off, to rest and recalibrate, after which I will be very excited to share how I’m planning to move forward. You will hear it here first!

love,
enjalot

Sign up now so you don’t miss the first issue.

Subscribe now

In the meantime, tell your friends!