Line charts are common data visualisation tools. They are usually the first chart we reach for when we want to visualise something that changes with time, depth (e.g well logs) or any other ordered axis. They are great when we want to display a handful of lines, however, when we start adding more than several onto that plot, it can become hard to read and starts to resemble spaghetti.

This is the equivalent of the overplotting problem we previously saw when looking at scatter plots / crossplots. Where a scatter plot fails is when the points on the chart start to lie on top of the same position resulting in a dense indistinguishable blob, but a line chart on the other hand fails when the series cross over each other and it can become difficult to follow them. Even though we can make the chart look good through styling, it now longer supports what the line chart was supposed to do.

To illustrate the problem with line charts, we will work with a set of twenty synthetic daily temperature records throughout a single year.

import numpy as np
import matplotlib.pyplot as plt

rng = np.random.default_rng(11)
days = np.arange(365)
n_stations = 20

def temperature_series(mean_c, amplitude_c, noise_sd=1.8, peak_day=200):
    seasonal = mean_c + amplitude_c * np.sin(2 * np.pi * (days - peak_day + 91.25) / 365)
    raw = rng.normal(0, noise_sd, days.size + 10)
    weather = np.convolve(raw, np.ones(11) / 11, mode=”valid”) * 3.0
    return seasonal + weather

stations = {}

for i in range(n_stations):
    stations[f”Station {i+1:02d}”] = temperature_series(
        mean_c=rng.uniform(4, 19),
        amplitude_c=rng.uniform(5, 15),
    )

fig, ax = plt.subplots(figsize=(8, 5))
cmap = plt.get_cmap(”tab20”)
for j, k in enumerate(stations):
    ax.plot(days, stations[k], color=cmap(j % 20), lw=1.2, label=k)
ax.legend(ncol=2, fontsize=7)
plt.show(

When we run the above, we get the following chart.

You can still read a few things from this, such as the overall trend of being cold at both ends of the year and warm in the middle, and you could probably identify the single warmest or coldest line at a given moment.

Beyond that, the data separates from the intended purpose of the line chart. You can’t properly follow Station 12 from January (day 0) to December (day 365), and you constantly have to refer to the legend to remember what line you are tracing, especially if colours are similar.

By having a chart like this, it makes the reader do a lot of hard work to try and extract anything meaningful from it.

Why the Line Chart Fails

The failure in this example has a few separate causes that compound on top of each other:

• Lines cross, so a colour you were tracking disappears behind others and re-emerges somewhere you weren’t looking

• There are more series than distinct colours the eye can hold apart, so the palette recycles and two stations end up near-identical shades

• And the legend sits away from the data, which forces a constant back-and-forth between the key and the plot that gets harder the more entries there are.

None of this can be fixed by looking harder at the plot, and it becomes a case of trying to ask too much of the chart at one time, such as:

• follow one specific station

• compare two stations

• judge the whole population

• spot the outlier(s)

It can’t serve all of these purposes on a shared axes once the series count is high. As with overplotting on a crossplot, the plot doesn’t warn you that there is a problem. That is down to you when you are creating it.

More Colours Won’t Save It

The first instinct you may have is to start looking at the colours of the lines and selecting a bigger palette in the hopes that if every station has a unique colour it may help resolve the issue. However, the human eye can only reliably separate a handful of categorical colours at once. Pushing past roughly ten or twelve colours produces shades so close that neighbours start to become indistinguishable from one another.

A twenty-colour or thirty-colour spaghetti plot is essentially a more colourful spaghetti plot, rather than a clearer one. The same goes for changing line styles or markers. These can help a little when we have a handful of series, but they clutter the chart as we add more.

What we need to try and do is to stop demanding that a single chart answer every question we have, but instead we need to pick a plot that answers the questions you have. This may mean several different smaller plots, but each should answer a question you have.

If You Need to See Each Station: Small Multiples

When the task is to inspect the individual behaviour of each series then it is best to give each one its own small panel and keep the axes common across all of them.

This results in small multiples or subplots appearing on the figure (also known as faceting).

fig, axes = plt.subplots(4, 5, figsize=(15, 9), sharex=True, sharey=True)
for ax, k in zip(axes.ravel(), stations):
    ax.plot(days, stations[k], color=”#1f3b57”, lw=1.0)
    ax.set_title(k, fontsize=9)
plt.tight_layout()
plt.show()

With the small multiples, you can still compare across them, which will make identifying a flat, mild year looks from a swinging one at a glance, and an odd station stands out from its neighbours. However, what you may end up losing, is the ability to see everything superimposed, and it comes with a size limit, which can be restrictive with small screen space.

If You Care About One or Two Series: Highlight and Context

Often you do not need to see all of your data series equally, but rather you may want to see how one or two specific stations behave against the rest.

In this case, you would draw the series you care about in strong colour and push everything else into a light grey background, which de-emphasises

fig, ax = plt.subplots(figsize=(8, 5))
for k in stations:
    ax.plot(days, stations[k], color=”0.8”, lw=0.9, zorder=1)
ax.plot(days, stations[”Station 07”], color=”#C44E52”, lw=2.4, zorder=3, label=”Station 07”)
ax.plot(days, stations[”Station 14”], color=”#4C72B0”, lw=2.4, zorder=3, label=”Station 14”)
ax.legend()
plt.show()

In this example, we will make two distinctive stations show on the plot.

Both of the series we have selected have annual means are almost the same, 10.6 °C and 11.4 °C, which is close enough that they may be missed in a summary table, however, when we look at where the stations are they are nothing alike. Station 07 is maritime: it moves through a range of about 11 °C across the whole year, roughly 7 °C in January and 14 °C in July. Station 14 is continental, swinging through 37 °C, from about −3 °C in January to 28 °C in July.

In January, Station 14 is the 17th warmest of the twenty and Station 07 sits 6th. By July they have switched places almost exactly: Station 14 is 3rd warmest, Station 07 has fallen to 18th. Two stations with the same annual average, ranked at opposite ends of the group depending on when you look. That is the kind of thing that looking an annual mean can hide, but was also completely invisible in the earlier spaghetti plot. The grey context still shows the population that these two series sit within, so you can still have a sense of scale without trying to untangle the lines.

This technique is common in climate charts, where a single year is highlighted against decades of others in grey. It works whenever the question is “how does this one compare to the rest.” It can also easily be applied to any other chart where you want to compare one against a large number of samples.

If You Only Care About the Population: Aggregate

Sometimes the individual identities don’t matter at all and you only want the overall behaviour and its spread. In that case, one option is to summarise by plotting a median line and shade a band between the full dataset’s percentiles.

Y = np.vstack(list(stations.values()))

p10 = np.percentile(Y, 10, axis=0)
p50 = np.percentile(Y, 50, axis=0)
p90 = np.percentile(Y, 90, axis=0)

fig, ax = plt.subplots(figsize=(8, 5))
ax.fill_between(days, p10, p90, alpha=0.25, label=”10th-90th percentile”)
ax.plot(days, p50, lw=2.2, label=”Median”)
ax.legend()
plt.show()

This throws away all of the individual station data, and what you get instead is a clean view on the typical seasonal cycle and how much stations vary around it. The band will widen or narrow to show where the population agrees or has minimal spread or where is has maximum spread.

This can be the right plot when you’re characterising a group of measurements rather than any particular member of it. However, it becomes the wrong plot when someone wants to ask something about a specific station, because that information is gone.

The Other Spaghetti: Overlaid Distributions

There is also a second problem where we may have too many distribution lines on a plot that overlap each other.

With three or four groups that reads fine, however, when we start to have ten or twelve, the smooth curves cross and stack into their own kind of spaghetti. This results in you losing the ability to see any single distribution’s shape.

The plot to help alleviate this is the ridgeline plot, which I have spoken about a few times on here. On the ridgeline plot each distribution gets its own baseline, stacked vertically with a little overlap, so the whole set reads like a landscape of ridges (and it also is pretty to look at!).

In this example, we have thirty years of daily temperature at one continental station, split by month, January at the bottom to December at the top.

from scipy.stats import gaussian_kde

month_len = [31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31]
edges = np.concatenate([[0], np.cumsum(month_len)])
years = [temperature_series(mean_c=11.0, amplitude_c=15.5) for _ in range(30)]
monthly = [np.concatenate([y[edges[m]:edges[m + 1]] for y in years]) for m in range(12)]
t_grid = np.linspace(-20, 36, 400)
overlap = 1.8

fig, ax = plt.subplots(figsize=(8, 7))
colours = plt.get_cmap(”coolwarm”)(np.linspace(0.05, 0.95, 12))

for i, samples in enumerate(monthly):
    dens = gaussian_kde(samples)(t_grid)
    dens = dens / dens.max()
    base = i * (1.0 / overlap)
    ax.fill_between(t_grid, base, base + dens, color=colours[i], alpha=0.85,
                    zorder=12 - i)
    ax.plot(t_grid, base + dens, color=”white”, lw=0.8, zorder=12 - i)

ax.set_yticks([])
ax.set_xlabel(”Temperature (°C)”)
plt.show()

The staggered baselines are what make this one work. Each month’s distribution is readable on its own, and because they share a horizontal axis you can still track how the whole seasonal cycle moves, the peaks sliding right through spring into summer and back left again through autumn.

You also can get something from this chart that was not easily identifiable on the line chart and that is that the summer months are visibly narrower than the winter ones, so this station’s winters are not just colder, but are far more variable. If we overlaid each of these on a single axis we would end up with an overlapping mess.

The trade-off with this chart is vertical space and a small amount of occlusion where a tall ridge hides the foot of the one behind it, but for comparing many distributions in an order that means something, it’s far more legible than the overlaid version.

Which One to Reach For

There’s no single replacement for a spaghetti plot, because the spaghetti plot was failing to answer several different questions at once, and each of those questions has its own better chart. Decide what you actually need to see, then pick accordingly: small multiples when you need to read each series, highlight-and-context when one or two matter against the rest, an aggregate band when only the population does, and a ridgeline plot when you’re comparing many distributions rather than many time series. Reaching for more colours is the one move that never helps.

The line chart isn’t wrong, any more than the scatter plot or the histogram was. Line charts are designed to answer the question “how does this series change”, however, when you ask it to answer that same question for twenty series and more at once on the same axes, it becomes painful. It is at this point that you need to stop adding lines and start choosing the plot that fits the question you’re really asking.

The ridgeline above relies on two settings that quietly decide whether the comparison is honest: every curve is normalised to the same peak height, and every KDE is evaluated over one fixed x-range. There is a third setting I left on default, and it is the one most likely to mislead you. The default gaussian_kde bandwidth over-smooths, and it will flatten a genuine right-skew into something that looks symmetric. Over on Substack I’ve written the full build of this chart, from scratch and with joypy, covering bandwidth choice, overlap tuning, and why 4 to 20 groups is the range where a ridgeline actually earns its place.

→ Ridgeline Plots in Matplotlib: An Underused Way to Compare Distributions

The histogram is on of the default ways to look at a dataset’s distribution. We essentially hand it a column of numbers and it shows where the values are concentrated, how widely they are spread out, and whether the distribution appears skewed. It’s simple, yet a familiar and useful data plot.

However, a histogram is not always a direct picture of the data. Before the chart appears on the screen, the values from our curve are divided into bins. The number of bins, their width and the position of their boundaries all affect the resulting shape of the histogram.

If we change any of those parameter, then the same dataset can appear smooth, bimodal or unexpectedly noisy.

This does not make the histogram a wrong choice, but it does mean that the histogram shows the data in a particular way, and how we view and set those parameters can influence the conclusions we draw.

Creating a Distribution With Two Modes

To see how this happens, we can create a synthetic porosity dataset representing two geological formations. Each formation contains 300 observations, with mean porosities of 0.15 and 0.20, and a standard deviation of 0.014.

import numpy as np
import matplotlib.pyplot as plt

rng = np.random.default_rng(3)

n = 600

pop_a = rng.normal(0.150, 0.014, n // 2)
pop_b = rng.normal(0.200, 0.014, n // 2)

por = np.clip(
    np.concatenate([pop_a, pop_b]),
    0.05,
    0.30
)

plot_range = (0.05, 0.30)

The two populations overlap, but they are sufficiently separated for the combined distribution itself to contain two modes.

Two geological groups do not always produce two peaks in a histogram. If their distributions overlap strongly enough, the combined data may be genuinely unimodal. In this example, we already know that two modes are present, so we can see how well the histogram reveals them.

The Bin Count Decides What You See

The number of bins controls the resolution of the histogram. With too few bins, the observations from different parts of the distribution are combined into the same broad intervals. This can make the real structure can disappear.

With too many bins, all of the small variations caused by random sampling can start to look like meaningful features. To illustrate this we can hold the data and plotting range constant whilst changing only the number of bins.

fig, axes = plt.subplots(
    1,
    4,
    figsize=(16, 4),
    sharey=True
)

for ax, bins in zip(axes, [6, 12, 18, 40]):
    ax.hist(
        por,
        bins=bins,
        range=plot_range
    )

    ax.set_xlim(plot_range)
    ax.set_xlabel(”Porosity (fraction)”)
    ax.set_title(f”{bins} bins”)

axes[0].set_ylabel(”Count”)

plt.tight_layout()
plt.show()

This gives us the following figure:

With six bins, the centre of the distribution becomes a broad block. There is no clear valley separating the two populations, and it would be difficult to justify a claim that the data contains two modes.

At twelve bins, some structure begins to appear.

At eighteen bins, the two concentrations and the lower-density region between them are much clearer.

At forty bins, the broader pattern remains visible, but the bars also begin to jump up and down in response to relatively small differences in sample count. Some of those fluctuations could easily be mistaken for additional peaks.

Each histogram is a small summary of the same data, and can support different visual interpretations.

A low-resolution histogram can hide genuine structure, whilst a high-resolution histogram can give random sampling variation more visual importance than it deserves.

There Is No Universally Correct Number of Bins

It is tempting to select the 18 bin histogram as the correct one, but there is no single bin count that is best for every dataset and every question.

A histogram designed to show the broad range of a variable may need fewer bins than one intended to investigate subtle modes. Sample size, variability, skewness and outliers can all affect what works well.

This is why several rules have been developed for selecting bin widths automatically.

One commonly used option is the Freedman–Diaconis rule, which uses the interquartile range and sample size

fig, ax = plt.subplots(figsize=(6, 4.5))

ax.hist(
    por,
    bins=”fd”,
    range=plot_range
)

ax.set_xlim(plot_range)
ax.set_xlabel(”Porosity (fraction)”)
ax.set_ylabel(”Count”)

plt.show()

Other options include "scott", "sturges" and "auto".

These rules are useful because they respond to the data rather than applying the same arbitrary bin count to every variable.

However, they should be treated as starting points and worked on from there.

A bin-width rule attempts to balance resolution against sampling noise, however, it does not know whether twe are trying to investigate skewness, tails, outliers or multiple geological populations.

It can still produce a reasonable histogram whilst obscuring the particular feature you care about.

Bin Position Matters Too

Bin count is often the choice most will go to when changing parameters on a histogram, however, the less obvious choice is where the bin boundaries begin.

Two histograms can use exactly the same bin width and still look different because their bin boundaries begin in different positions. Shifting the first boundary shifts every subsequent boundary with it. Values near an edge may then move from one bar to another, changing the heights of neighbouring bins.

In the following example, the bin width remains fixed at 0.018. The second histogram shifts the boundaries by half a bin width.

width = 0.018
offsets = [0.0, width / 2]

fig, axes = plt.subplots(
    1,
    2,
    figsize=(10, 4.2),
    sharey=True
)

for ax, offset in zip(axes, offsets):

    edges = np.arange(
        plot_range[0] + offset - width,
        plot_range[1] + width,
        width
    )

    ax.hist(por, bins=edges)
    ax.set_xlim(plot_range)
    ax.set_xlabel(”Porosity (fraction)”)

axes[0].set_title(”Original bin alignment”)
axes[1].set_title(”Shifted by half a bin”)
axes[0].set_ylabel(”Count”)

plt.tight_layout()
plt.show()

Both histograms still contain evidence of two modes, but the apparent shape changes.

The relative heights of the peaks differ, as does the depth and position of the valley between them. One alignment produces sharper individual bars, whilst the other gives the populations a broader appearance.

Nothing about the observations has changed. Only the location of the bin boundaries has moved.

For a clearly separated distribution, this may alter only the visual emphasis. For a borderline case with more strongly overlapping populations, the shift may be enough to determine whether a small valley appears at all.

Bin position is particularly easy to overlook because it is rarely stated on the finished figure. The reader sees the bars, but not the arbitrary grid that produced them.

Can We Avoid Binning?

One way to check the distribution is to compare the histogram with a plot that does not use bins.

The empirical cumulative distribution function, or ECDF, plots each observed value against the proportion of observations at or below it.

xs = np.sort(por)
ys = np.arange(1, len(xs) + 1) / len(xs)

fig, ax = plt.subplots(figsize=(6, 4.5))

ax.plot(
    xs,
    ys,
    drawstyle=”steps-post”
)

ax.set_xlim(plot_range)
ax.set_ylim(0, 1)

ax.set_xlabel(”Porosity (fraction)”)
ax.set_ylabel(”Cumulative proportion”)

plt.show()

The ECDF has no bin count, bin width or bin origin. Every observation contributes directly to the curve.

It is particularly useful for reading quantiles, comparing distributions and understanding the proportion of values above or below a threshold.

Changes in concentration can also appear as changes in slope. Steeper sections indicate that many values occupy a relatively narrow interval, whilst shallower sections indicate that the observations are more dispersed.

However, an ECDF is not necessarily the best plot for identifying modes.

The two concentrations are visible through changes in the slope of the curve, but they are less immediate than the peaks in a well-chosen histogram. Someone unfamiliar with ECDFs may find the histogram much easier to interpret.

It is therefore better viewed as a complementary plot rather than a complete alternative.

Kernel density estimates offer another smooth view of a distribution without discrete bins, but they introduce a bandwidth parameter. Too much smoothing can merge populations, whilst too little can produce artificial bumps. Kernel density plots avoid discrete bins, but they still require us to choose how much smoothing to apply.

A More Reliable Way to Use Histograms

When working with histograms, there are a few things we can do to settle on the most optimal bin parameters for the figure we are working on:

1. Plot several reasonable bin counts whilst keeping the plotting range fixed.

2. Try shifting the bin origin, particularly when a possible mode depends on one or two bars.

3. Use an automatic rule such as Freedman–Diaconis as a starting point, and work from there

4. Compare the result with an ECDF, density estimate, rug plot or the known geological groupings.

Known categories are especially important.

If formation labels are available, plotting each formation separately may reveal that two overlapping geological populations produce only one broad peak in the combined data.

Share

The Histogram Is Still Useful

A histogram remains one of the fastest ways to obtain a first impression of a continuous variable. It can reveal skewness, unusual tails, gaps and possible modes in a form that most readers understand immediately.

However, the danger begins when we forget that its shape depends partly on decisions that we make with the parameters. Bin count controls how much structure is resolved. Bin position determines which observations are grouped together.

Neither the bin count nor the bin position comes from the underlying data, but both can change how we interpret the final plot.

The histogram’s blind spot isn’t unique to histograms. Every standard chart compresses the data somehow, and each has a point where the thing it compresses away is the thing you needed to see. Over on Substack I walk through nine common scientific plots and the single blind spot that trips each one up, from scatter plots and boxplots to correlation matrices and PCA.

→ 9 Scientific Data Plots and When They Fail

Many articles and guides on data visualisation focus on when to use certain charts and how to get the best answers from your data. A histogram for understanding data distributions, a scatter plot (crossplot in petrophysics) for analysing relationships, a heatmap for matrices. There is nothing wrong with that advice, however many of those articles overlook the situations where those charts are not the best choice.

All data visualisations and chart types make trade-offs when it comes to showing data. Some charts compress your data down to bins or several key numbers. For example, boxplots compress a data distribution into five numbers so you can easily compare twenty groups at a glance. The trouble begins when the thing your data needs to say is exactly the thing the chart compresses away.

Here, we will look at nine common data plots and one aspect where each one starts to fail. The examples have a geological and petrophysical flavour, but the same issues apply across engineering, data science and any discipline where data needs to be visualised.

1. Scatter Plots and Overplotting

Scatter plots (also known as crossplots in geoscience) are one of the most common charts in science and data analysis. They allow us to easily compare two variables without needing to calculate a summary statistic first.

For small or moderately sized datasets, they work extremely well. We can identify trends, groups of data, data gaps, outliers and changes in variance.

However, the problem begins when the number of points being displayed on these plots increases.

Once points start landing on top of one another, the scatter plot no longer shows every observation clearly. Instead, it starts to turn into a structureless blob with points around the edges only being easily identifiable.

A region containing 50 points may look almost identical to one containing 5,000 points. Dense areas become solid blocks, while clusters and secondary trends can become indistinguishable due tooverlapping markers.

Two scatter plots showing synthetic porosity and permeability data. The first contains 800 points and shows the underlying structure clearly. The second contains 60,000 points and has collapsed into a dense cloud.

This can be particularly dangerous because the resulting chart still looks valid. It has labelled axes, sensible markers and maybe even a fitted trend line. There is nothing obvious to tell the reader that a large amount of information is hidden.

Transparency and reducing point size can help, especially when the overlap is moderate. However, there comes a point where adjusting marker properties is no longer enough.

At that stage, alternatives such as hexbin plots, two-dimensional histograms, density contours or careful sampling may give a more honest view of the data.

20,000 points as a hexbin plot, which allows a secondary group of points to show through.

The important question here is not simply whether all the observations have been plotted, but rather whether the reader can still see where those observations are concentrated.

The way this deteriorates at scale, and the alternatives worth reaching for, are covered in more detail here -> What Your Scatter Plot Stops Showing You Once It Gets Crowded

2. Histograms and Bin Sensitivity

A histogram divides a continuous variable into multiple bins and counts how many observations fall into each one. It may seem like a straightforward process, but the resulting shape of the histogram will heavily depend on how those bins are chosen.

If you use too few bins then genuine features may be smoothed away. This could lead to the loss of information. For example, a bimodal distribution, which may represent two different rock types or experimental populations may appear as one broad peak.

If you use too many bins, the random variation starts to look like meaningful structure and the distribution becomes a noisy group of peaks and gaps.

The same synthetic bimodal porosity dataset shown using 6, 20 and 80 bins. The first appears to contain one broad population, the second reveals two populations, and the third produces a noisy comb-like pattern.

There is no universally correct number of bins. However, there are rules such as Sturges, Scott and Freedman-Diaconis that can provide useful starting points to decide an appropriate number of bins. But, they don’t remove the need to inspect the result. Different rules are based on different assumptions and may produce very different charts.

This means that bin choice should not be treated as a minor plotting setting as it can influence the conclusion we draw from the data.

A useful habit that I have got into is to inspect the histogram using several different bin widths.

Kernel density estimates can provide another view of the data distribution, although they can introduce their own decisions around bandwidth and smoothing parameters. The example below shows a combination of a KDE, histogram bins and key statistics.

A step-by-step walkthrough of taking a matplotlib histogram from its defaults to something more informative is here: From Default to Publication-Ready: Transforming Matplotlib Histograms | by Andy McDonald | Data Science Collective | Medium

3. Boxplots and Hidden Distributions

Boxplots are excellent for comparing distributions across several groups. I have covered these and their alternatives several times recently, and some of the issues that may be encountered whilst using them.

Boxplots display the median, the upper and lower quartiles, whiskers and any values identified as potential outliers. This compact summary makes it possible to compare ten or twenty groups without filling the figure with individual observations.

However, the compression of that data into a succinct summary comes at a cost.

Very different distributions can produce similar boxplots. A normal distribution, a uniform distribution and a bimodal distribution can all share approximately the same median and interquartile range.

When shown as boxplots, they look very similar to each other.

Three similar boxplots shown beside their underlying observations. The raw data reveals normal, bimodal and uniform distributions.

This matters because the missing structure may be scientifically important.

A bimodal distribution often suggests that two populations have been combined. In a petrophysical dataset, this could mean that two facies or depositional settings are being treated as one group. In another discipline, it might indicate two experimental regimes or two different responses within a population.

The boxplot can’t show this because it wasn’t originally designed to describe the detailed shape of a distribution. This doesn’t mean that we should ignore them or drop them entirely from our toolkit. They remain useful when our primary goal is to compare medians, spreads and potential outliers across many groups.

The problem, and some practical alternatives, are covered in more detail here: Boxplots Are Hiding Your Data. Here’s a Simple Fix

However, problems can come up when we interpret them as complete descriptions of the underlying data.

There are a number of alternatives that can be used to reveal more about the data distribution including violin plots, strip plots and raincloud plots Another simple option is to overlay the individual data points on top of the boxplot, particularly when the sample size is manageable. However, my favourite recently is the raincloud plot that can be seen below.

You can find out how to create raincloud plots in several easy steps here: From Boxplot to Raincloud: Building a Better Distribution Plot in Python

4. Heatmaps and Information Overload

Heatmaps are often used to display a large number of values within a compact space and be visually appealing to look at. Essentially, they work well when the grid is reasonably small and the colour scale has a clear meaning. A correlation matrix containing eight variables, for example, can be easy to inspect, however, if that matrix were to grow to 100 variables, then readability breaks down.

A heatmap containing dozens of rows and columns may technically display every value, but that does not mean the information can be interpreted. Individual cells become too small, labels can become crowded and difficult to read, and subtle differences in colour can become difficult to distinguish.

A 40 by 28 annotated heatmap where the labels overlap and individual values are difficult to interpret.

At this point, the heatmap becomes a block of patchwork coloured boxes rather than a useful chart. The deeper problem is often that the heatmap has been created without a specific question in mind. Displaying every sample against every feature may feel like it is serving a purpose, but it is left to the reader to work out what they are supposed to notice.

Sorting or clustering the rows and columns can reveal structure and reducing the number of variables may also help. In some cases, a focused heatmap showing only the most important features communicates far more than the complete matrix.

The colour scale also matters. If we choose a poorly suited colourscale , the we could inadvertently exaggerate small differences or hide important ones. Diverging colour maps should normally be centred on a meaningful reference value, such as zero in a correlation matrix.

A heatmap should also help answer a question. In the example below, we are trying to answer the questions “What wells have complete logging sets and what curves are missing and where”. When the plot tries to display everything at once, it often answers nothing particularly well.

Interested in creating your own interactive heatmap with Plotly? If so, check out How to Visualise Missing Data with an Interactive Plotly Heatmap

5. Pie Charts and Too Many Categories

Pie charts show how a total is divided between categories. With three or four clearly different categories, they can provide a quick impression of the overall composition. Problems begin when the number of categories increases or when several slices are similar in size.

They are are a love it or hate it type of plot. If you search up pie chart problems on Google, you will find hundreds of articles pointing to the same problems. Some with very strong views on when not to use the plot.

Humans are not particularly good at comparing angles or areas, and it becomes harder when trying to compare multiple copies of the pie chart to see what has changed. It is easy to identify the largest slice in a simple pie chart, but becomes much harder to decide whether one category represents 11 per cent and another represents 13 per cent.

Twelve similarly sized lithology categories shown as a pie chart and as a sorted horizontal bar chart.

The reader is then forced to move repeatedly between the chart and its legend, and similar colours become difficult to distinguish. If the pie slices are not ordered by size, then ranking the categories requires more effort than it should.

A sorted bar chart can be a better way to communicate the same values more clearly. The categories share a common baseline, making small differences much easier to compare. These can be further enhanced by including data labels to make it easier for the reader.

The pie chart does have one useful property in that it makes it visually clear that the categories form parts of a whole. However, that advantage is quickly lost when there are too many slices or when precise comparison between slices or multiple charts matters.

For a small number of distinct categories, a pie chart may be suitable. For most scientific or technical comparisons, particularly those involving several categories, a bar chart is usually easier to read.

6. Line Charts and Spaghetti Plots

A line chart is one of the clearest ways to show how a value changes across time, depth or another ordered axis.

One line is easy to follow and five lines may still be manageable. However, by the time eighteen lines are shown on the same axes, the chart has usually become a spaghetti plot.

Eighteen synthetic well production decline curves overlaid on the same axes, with a large legend and several intersecting lines.

The problem isn’t simply that the figure looks messy, but rather that the chart can no longer support many of the tasks we expect it to perform.

We may still be able to see the overall range of values or identify a major outlier. However, following an individual series becomes difficult. Lines cross, colours repeat and the legend becomes detached from the data.

Increasing the number of colours rarely solves the problem. It often produces a more colourful version of the same confusion.

The appropriate alternative to the mess of spaghetti depends on the question being asked of the data.

We can use small multiples to show each series in a separate panel while retaining a common scale. This makes it easier to inspect individual behaviour without losing comparability.

Another option is to highlight one or two important lines and display the remaining series in the background. This works well when the aim is to compare a selected well, experiment or model against the wider population. It is often used when comparing yearly temperatures to previous years.

Grouping and summarising the series may also help. A median line with an uncertainty band can communicate the overall behaviour when individual identities are not important.

A line chart containing many series is not always wrong and it may still be useful for showing an envelope or identifying extreme cases. It begins to fail when we expect the reader to follow every line individually.

7. Correlation Matrices and Non-Linear Relationships

Correlation matrices provide a neat and compact summary of the relationships between several variables.

In most cases, the values shown on the correlation matrix are Pearson correlation coefficients. Pearson’s correlation measures the strength of a linear relationship between two variables and ranges between -1.0 and +1.0, with a value close to 0 indicating no linear relationship.

Two variables can have a strong and predictable non-linear relationship while producing a Pearson correlation close to zero.

A correlation matrix showing a correlation near zero between X and Y, shown beside a scatter plot that reveals a clear quadratic relationship.

Looking at the matrix above alone, we might conclude that some of the variables are unrelated. The scatter plot on the other hand shows that Y is strongly determined by X, but the relationship is curved rather than linear.

This can cause problems during exploratory analysis and feature selection. A variable may be removed because its correlation with the target is weak, even though a non-linear model could make good use of it.

Spearman correlation can detect monotonic relationships, including some curved relationships, because it works with ranks rather than raw values. However, neither Pearson nor Spearman reliably captures a relationship such as a parabola.

Other approaches, including mutual information and distance-based measures, can identify broader forms of dependence. These methods also come with assumptions and limitations, so they should not be treated as automatic replacements.

The correlation matrix is useful for screening a set of variables, however it should be supported by other plots such as scatter plots.

Share

8. PCA Plots and Interpretation

Principal Component Analysis (PCA) is often used to reduce a dataset containing many variables into two or three dimensions.

The resulting scatter plot of the Principal Components can reveal broad clusters, outliers and changes in structure, however, these plots are also easy to over-interpret.

The first issue that you may encounter is scaling. PCA identifies directions that explain the largest amount of variance within a given set of features. If the input features use very different numerical scales, the result may be dominated by the variables with the greatest variance.

For example, a gamma ray curve measured over a range of hundreds of API units may dominate a density curve varying between approximately 2 and 3 g/cc.

The same four synthetic well log curves projected using PCA before and after standardisation. The unscaled version is dominated by one feature, while the standardised version produces a different structure.

Applying standardisation to the features gives each variable a more comparable influence, but this is also a modelling decision. In some cases, the original scale may contain meaningful information, whereas in others, it simply reflects the units used for measurement.

The second issue is interpretation.

Principal Components are essentially weighted combinations of the original variables. They don’t have an inherent physical meaning, even when the resulting axes appear to separate familiar groups.

It may be tempting to assign a label of “Lithology” to PC1 and “Porosity influence” to PC2, unless it is supported by evidence and domain knowledge

Variance is also not the same as importance. A PC can explain a large proportion of the variation in the dataset whilst being unrelated to the prediction or scientific question we care about.

Finally, if we only look t a two-dimensional PCA plot we are only displaying part of the original feature space. Which may mean what appear to be nice data clusters or data overlaps may change when additional PCs are considered.

PCA plots are useful for orientation and gaining insight into the overall structure, as well as help identify unusual observations. They can become unreliable when we start treating the projected axes as measured physical properties.

9. Violin Plots and Sample Size

Violin plots combine a summary of the distribution with a kernel density estimate. Unlike boxplots, they show an estimated shape, which can reveal skewness, multiple peaks and differences in spread that would otherwise remain hidden.

However, the smooth shape of a violin plot can create a false sense of confidence in the data.

A Kernel Density Estimate (KDE) will produce a continuous curve even when it is based on only a small number of observations. A group containing nine samples can therefore produce a violin that looks just as smooth and complete as one containing five hundred.

Two violin plots with similar smooth shapes. One is based on 9 observations and the other on 500. Strip plots beside them reveal the difference in sample size.

With a small sample size much of the apparent structure is controlled by the bandwidth used by the density estimator. A narrow bandwidth may create several peaks, whilst a wider bandwidth may merge them into one broad shape.

The reader may also have no way to tell how much data sits behind the violin unless sample sizes are reported. Much in a similar way to boxplots.

This is where raw points become particularly useful. By overlying a strip plot or by using a raincloud plot the reader can see both the estimated distribution and the observations that produced it.

For large samples, showing every point may create overplotting, so a sampling strategy or alternative density representation may be needed. For small groups, however, there is usually little reason to hide the individual observations.

Leave a comment

The Pattern Underneath

These nine plots fail in different ways, but the underlying problem is similar.

Each chart relies on assumptions about what matters.

A scatter plot assumes the points remain individually visible. A histogram assumes the selected bins provide a useful representation of the distribution. A correlation matrix assumes that a coefficient can describe the relationship we care about. A PCA plot assumes that variance is a useful direction in which to view the data.

The data plot continues to render even when those assumptions are broken.

That is what makes these failures easy to miss. A crowded scatter plot and an overfilled heatmap can all look clean and publication-ready. Nothing in the plotting library warns us that the chart is hiding an important part of the story. It is down to us, as the interpreter, to understand what each chart compresses in the data and how we can get the best from them.

Knowing what each chart compresses is the starting point. Getting the figure itself right before those trade-offs start to matter is the next step. Over on Substack I’ve covered six specific matplotlib changes that take a figure from functional defaults to something you’d put in a report: spine cleanup, intentional colour choices, appropriate axis scaling, gridlines that support rather than clutter, annotation, and export settings that don’t clip your labels.

Six Matplotlib Changes That Improve Your Figures for Reporting and Publication

Imagine that you have just finished building a regression model to predict missing well log data You then check how well it has performed on an actual vs predicted scatter plot (also known as a crossplot in petrophysics).

The result may look convincing, with a dense cloud of data points following the trend of the one-to-one line and hugging close to it. However, a large portion of those points are likely sitting directly on top of each other. Therefore, what may look like a reassuring cloud to begin with, may be systematically wrong across a whole range of values. From this chart alone and without further tweaks to the plot, it can become difficult to tell.

This is a case of overplotting, and it’s one of the common ways that a simple scatter plot can become an unreadable mess.

This does not mean that the data and the plot are entirely usless. There are a few fixes we can make to the scatter plot to help clear things up, such as switching to other styles of visualisation within the bounds of the scatter plot in order to understand the underlying relationship between the actual and predicted data.

We will see some of those fixes in this article. Also, one thing to note, even though this article focuses on actual vs predicted data, the reuls and ideas can equally apply to any scatter plot.

Setting Up a Stand-In Dataset

To illustrate overplotting we will create some synthetic well data that will allow us to display a predicted versus measured scatter plot (also known as a crossplot in petrophysics, hence the name make_crossplot_data). Here most of the points will fall in a broad cloud along the one-to-one line, with a tighter secondary concentration of data points that we will deliberately bury inside it.

import numpy as np
import matplotlib.pyplot as plt

rng = np.random.default_rng(42)
def make_crossplot_data(n):
    n_main = int(n * 0.7)
    n_knot = n - n_main
    measured_main = rng.normal(0.15, 0.05, n_main)
    measured_knot = rng.normal(0.22, 0.012, n_knot)   # the hidden concentration
    measured = np.concatenate([measured_main, measured_knot])
    predicted = measured + rng.normal(0, 0.02, n)      # model error around 1:1
    measured = np.clip(measured, 0, 0.35)
    predicted = np.clip(predicted, 0, 0.35)
    return measured, predicted

LIMS = (0, 0.35)

def style_axes(ax):
    ax.set_xlim(*LIMS)
    ax.set_ylim(*LIMS)
    ax.set_xlabel(”Measured porosity”)
    ax.set_ylabel(”Predicted porosity”)
    ax.plot(LIMS, LIMS, color=”0.4”, lw=1, ls=”--”, zorder=0)  # 1:1 line
    ax.set_aspect(”equal”)

With the data generated, and some plotting defaults set up, we can create our first scatter plot, one that many of us will jump to after carrying out a continuous variable prediction.

measured, predicted = make_crossplot_data(20000)

fig, ax = plt.subplots(figsize=(6, 6))
ax.scatter(measured, predicted, s=10, color=”#1f3b57”)
style_axes(ax)
plt.show()

On first looking at this chart we may think this looks like a decent prediction for porosity. However, most of the points closer to the 1-to-1 line are actually sitting on top of each other.

Watching a Scatter Plot Fail

The easiest way to visualise overplotting is to watch it happen.

To do this, we will draw the same distribution at increasing sample sizes and see where the scatter plot starts to break down.

First, we start small with a hundred points. At this level, we can see that the scatter plot is doing it’s job well. We are able to see individual observations, judge the spread, and pick out anything unusual.

m100, p100 = make_crossplot_data(100)

fig, ax = plt.subplots(figsize=(6, 6))
ax.scatter(m100, p100, s=25, color=”#1f3b57”)
style_axes(ax)
plt.show()

If we push it to a thousand points, the chart still holds up, but the densest region is starting to merge together, which makes it difficult to count the individual observations in there. However, the overall shape of the data remains readable, for the most part.

m1k, p1k = make_crossplot_data(1000)

fig, ax = plt.subplots(figsize=(6, 6))
ax.scatter(m1k, p1k, s=15, color=”#1f3b57”)
style_axes(ax)
plt.show()

Now, if we increase it to twenty thousand points, the chart fails.

The centre becomes a uniform dark blue blob with many overlapping observations. And as a result, any structure that existed within that blob has now become invisible.

fig, ax = plt.subplots(figsize=(6, 6))
ax.scatter(measured, predicted, s=10, color=”#1f3b57”)
style_axes(ax)
plt.show()

Why This Happens

Each observation gets its own visible mark on the scatter plot. Once two markers occupy the same pixels, the second one adds no information and sits on top of it. The hundredth point on the same position adds none either. Dense regions start to saturate early and then hide everything that arrives afterwards, which means the regions where most of your data lives are precisely the regions you can see least clearly. Meanwhile, if we have a single stray outlier at the edge of the plot, it remains perfectly visible and can draw far more attention from the reader than its importance deserves.

There is no universal point count where this issue kicks in and it will depend on the size of the markers, figure dimensions, screen resolution and, most of all, how many data points there are concentrated close together.

A tight bivariate relationship can saturate with a few thousand points, whilst a more diffuse one might last far longer before it becomes saturated and we lose any meaningful pattern or structure.

What we need to watch for is: if you can no longer distinguish individual observations in the densest region,the chart can no longer answer the question we initially set out to answer.

Temporary Fix: Transparency and Marker Size

The usual first response to improving this is to make the markers semi-transparent. Overlapping points then stack visually, so denser regions render darker and you get a crude signal for data density.

fig, ax = plt.subplots(figsize=(6, 6))
ax.scatter(measured, predicted, s=10, color=”#1f3b57”, alpha=0.1)
style_axes(ax)
plt.show()

This can help for a while. Ten overlapping points at ten per cent opacity already render as effectively solid, so the darkness scale saturates quickly. You can, however, distinguish sparse data from dense data, but not dense from very dense.

To buy us more headroom, we can shrink the markers and reduce the impact of overlapping data points. This may work for datasets in the low thousands. However, beyond that, both tricks run into the same issue: a finite number of pixels and far more observations than pixels to put them in.

Sub-Sampling: Honest but Lossy

Another option is to plot a random sample of the data rather than the full dataset. Draw a couple of thousand points from a couple of hundred thousand and the scatter plot becomes readable again. This might mean we are only displaying 10% of the full data volume.

For quick exploration this may be a reasonable move, however, the cost is that you’ve thrown away most of your data to make the chart legible.

Rare behaviour, small clusters and sparse outliers may easily be removed and become less prominent when the chart is drawn. Sampling is a workaround for the display problem, rather than a solution to it.

The Real Fix: Ask About Density Instead

Everything so far tries to preserve the original question, “where are my individual observations”, in a situation where it can no longer be answered. The better move is to change the question to “where is my data concentrated”.

This is what a hexbin plot does.

The hexbin plot divides the plotting area into hexagonal cells, counts the observations falling into each one, and maps the count to colour. No overlap is possible, because nothing is drawn per observation. Dense regions are no longer hidden, instead they become the brightest cells on the chart.

fig, ax = plt.subplots(figsize=(6, 6))
hb = ax.hexbin(measured, predicted, gridsize=40, cmap=”viridis”,
               mincnt=1, extent=(*LIMS, *LIMS))
ax.plot(LIMS, LIMS, color=”0.7”, lw=1, ls=”--”, zorder=3)
ax.set_xlim(*LIMS); ax.set_ylim(*LIMS)
ax.set_xlabel(”Measured porosity”); ax.set_ylabel(”Predicted porosity”)
ax.set_aspect(”equal”)
fig.colorbar(hb, ax=ax, label=”count”, shrink=0.8)
plt.show()

If we look at the bright cell just above the middle of the one-to-one line, that’s the secondary concentration we built into the data at the start. This was what we lost when we displayed the vast number of points earlier. Here, the hexbin surfaces it without any special effort.

Using a hexbin plot does come with a trade-off, in that you lose individual observations, and you can no longer point at a specific well or sample. Outliers can become single faint cells that’s easy to miss.

Choosing the right bin size also matters. If the cells are too large smear then the underlying structure is not there any more, whereas cells that are too small reintroduce noise. For the core task of understanding where a large dataset really lives on the plot, then the hexbin chart can help with that.

Density Contours

Where the underlying data is continuous, kernel density estimation offers a related view.

Instead of counting points in cells, it estimates a smooth density surface and draws it as contour lines or filled bands, much like elevation contours on a map.

from scipy.stats import gaussian_kde

xy = np.vstack([measured, predicted])
kde = gaussian_kde(xy)
xx, yy = np.mgrid[LIMS[0]:LIMS[1]:200j, LIMS[0]:LIMS[1]:200j]
density = kde(np.vstack([xx.ravel(), yy.ravel()])).reshape(xx.shape)
fig, ax = plt.subplots(figsize=(6, 6))
cf = ax.contourf(xx, yy, density, levels=10, cmap=”viridis”)
ax.set_xlim(*LIMS); ax.set_ylim(*LIMS)
ax.set_xlabel(”Measured porosity”); ax.set_ylabel(”Predicted porosity”)
ax.set_aspect(”equal”)
fig.colorbar(cf, ax=ax, label=”density”, shrink=0.8)
plt.show()

Contours are great for describing the overall shape of a distribution and for overlaying two groups on the same axes, something that can become impossible when using saturated scatter plots.

However, one thing to bear in mind is the smoothing is also the weakness. KDE can paint density into regions where no data exists, which matters if your variables have hard physical limits, such as a porosity that can’t go below zero. Treat contours as a description of shape, and check the edges against the raw data.

When You Have Millions of Points

If you truly have a large dataset, for example in the millions and beyond, tools like Datashader can be useful to explore. Rather than drawing markers at all, they aggregate the data into an image directly, computing a value per pixel. Every observation contributes to the overall picture with nothing

You Don’t Need Big Data to Have This Problem

Most discussions of overplotting treat it as a big data problem. It can also appear, and often more severely, when your values are discrete.

Well log data is a good example. A gamma ray curve recorded to the nearest whole API unit could contain the same value thousands of times across a field. On a crossplot, every one of those observations lands on exactly the same coordinates. Not nearly the same pixel, but actually the same pixel. A chart with only three thousand points can be badly overplotted if those points take a few hundred distinct value pairs, and you’ll see the same effect with any rounded measurement, integer counts, categorical codes plotted numerically, and sensor data with limited resolution.

n = 3000
base = rng.normal(0, 1, n)
gr1 = np.round(np.clip(70 + 12 * base + rng.normal(0, 6, n), 40, 110)).astype(int)
gr2 = np.round(np.clip(70 + 12 * base + rng.normal(0, 6, n), 40, 110)).astype(int)

fig, axes = plt.subplots(1, 2, figsize=(12, 6))
axes[0].scatter(gr1, gr2, s=12, color=”#1f3b57”)
axes[0].set_title(”Scatter: 3,000 points on a grid”)
axes[0].set_xlabel(”GR curve A (API)”); axes[0].set_ylabel(”GR curve B (API)”)
axes[0].set_aspect(”equal”)
hb = axes[1].hexbin(gr1, gr2, gridsize=30, cmap=”viridis”, mincnt=1)
axes[1].set_title(”Hexbin: where the weight actually is”)
axes[1].set_xlabel(”GR curve A (API)”); axes[1].set_ylabel(”GR curve B (API)”)
axes[1].set_aspect(”equal”)
fig.colorbar(hb, ax=axes[1], label=”count”, shrink=0.8)
plt.show()

This case here is sneakier than the large data point version, because the number of points being plotted gives you no warning.

The chart could look sparse enough to trust, and adjusting the transparency of the points can help a little or even some point jittering (adding small random offsets to separate identical values, although it means plotting data that doesn’t quite exist).

Using a density view can handle that more cleanly.

Subscribe now

Summary

A scatter plot answers the question: “where are my individual observations?” That’s a good question to have when every observation is visible.

However, once observations start disappearing underneath each other, whether from the sheer volume of data or from repeated discrete values, the chart is no longer answering that question. You may try changing the styling, however, that may not be enough.

At that point, you should stop asking where the data points are and start asking where the data is concentrated. This is where hexbin plots, density contours, or a separate aggregation tool, can show you what the blob was hiding.

Overplotting is one way a default chart hides your data, but it isn’t the only one. A boxplot does the same job on a distribution, reducing it to five numbers so a bimodal formation with a gap in the middle can look like an ordinary spread. Over on Substack I build a raincloud plot one layer at a time, stacking the density, the raw points and the boxplot summary so you can see the shape and check it against the actual observations.

→ From Boxplot to Raincloud: Building a Better Distribution Plot in Python

Machine learning has become a common part of petrophysical workflows, from predicting missing curves to classifying facies. Most of the data we feed these models is continuous, eg. gamma ray, density, neutron porosity, sonic compressional. But, alongside the curves, almost every project carries categorical data too. This includes formation names, facies schemes, zonations, and lithology descriptions.

The problem is that machine learning models can’t use them directly. A linear regression, a neural network, or a support vector machine works entirely in numbers. It can multiply a gamma ray value by a coefficient, but it cannot multiply text strings by anything. So before a formation column can go into a machine learning model, it has to be converted into numeric values. There are different ways to do this, and choosing which method can impact how the model(s) treat those values and information.

In this article, we will look at what encoding is, the common options, and why one hot encoding is usually a safe choice. We will then test it on a simple, but real dataset from the Volve field to see how much difference the choice of encoding method makes.

What is encoding?

First, we need to understand what encoding is.

Encoding is simply the process of converting categorical values into numerical values that a model can work with. That sounds like a mechanical step, and it is often treated as one, a line of preprocessing code to get through before the interesting work starts. But, every encoding scheme carries information into the model that in turn tells the model something about those categories.

Numbers have an order, and as a result they have distances between them. Whatever numbers we choose, the model will take them seriously, whether we intended them or not.

Here are a few common encoding techniques.

Label encoding (also called integer encoding) assigns each category a whole number: 0, 1, 2, 3 and so on. It’s simple and compact, one column in, one column out, and often the first thing we think of doing when converting lithologies or formations to numeric values.

In some cases, the label encoding of lithology values is already done as part of data deliverables or converted as part of a company standard. For example a coarse grained arkosic sandstone is always encoded as 12345, and a fine grained laminated shale is always encoded as 54321. Even though the numbers are very different, they still carry information and distances.

When using tools like scikit-learn’s LabelEncoder each category is assigned an integer based on alphabetical order. This is illustrated below:

The encoder has placed the formations on a number line, at equal spacing, in alphabetical order. To us that looks like a lookup table, however when a model looks it this it considers it a measurement scale. A linear model given this column has to fit a single coefficient across it, which means treating the step from Draupne Fm to Heather Fm as the same size as the step from Hugin Fm to Tor Fm, and treating the Draupne Fm as two times as far from Heimdal Fm as the Tor Fm is.

Ordinal encoding is the same idea, but with the integers assigned in an order you choose deliberately, because the categories have a real ranking. Coarse to fine grain size across facies would be a candidate, as would geological age.

One hot encoding creates a new binary column for each category, which we will come to shortly.

There are more exotic options too, such as target encoding and learned embeddings, which are useful when a categorical has hundreds of levels, but for the handful of formations or facies in a typical petrophysical dataset, the three above cover almost every situation.

What is one hot encoding?

One hot encoding takes a different approach. Instead of one column of integers, it creates one binary column per category. Each sample gets a 1 in the column belonging to its category and a 0 everywhere else.

Which prediction algorithms this matters for comes down to a single question: does the algorithm treat the numbers you feed it as positions on a scale? The ones that do are exactly the ones that get misled by label encoding, and exactly the ones one hot encoding can work well with.

Distance-based methods are the clearest case. For example, K-nearest neighbours decides what a sample is by measuring how far it sits from others, and self-organising maps and clustering methods like K-means work the same way. Under label encoding, the Skagerrak at code 6 is literally six units from a formation at code 0 and one unit from its alphabetical neighbour, so the algorithm treats alphabetical adjacency as geological similarity. One hot encoding fixes this because every category sits the same distance from every other, so no false neighbours are created.

Linear and additive models are the next case. Multiple linear regression fits one coefficient per feature, so a label encoded column forces a single straight-line relationship across categories that have no order. With one hot encoding, each formation gets its own coefficient and can take whatever value the data supports, independently of the others. The same reasoning covers logistic regression and, to a degree, support vector machines.

Neural networks sit here too. A network multiplies each input by a weight, so a label encoded formation enters as a single input whose magnitude the network reads as meaningful, and larger codes push harder on the first layer for no good reason. One hot encoding gives the network one input per category, each with its own weight, which is why it is the usual starting point for categorical inputs that are not being handled by a dedicated embedding layer.

The common theme between these models is that all of them treat input magnitude as information. The main family that does not are tree-based models. A decision tree, random forest, or gradient boosting model splits a feature into groups rather than scaling it, so it can carve a label encoded column into sensible partitions on its own. Encoding still matters there for other reasons, but the false-ordering problem is much weaker.

So one hot encoding is generally the safe default whenever the model reads numbers as positions on a scale, and the categories have no natural order. It makes no ordering claim, so it cannot make a wrong one.

One issue that can arise from using One Hot Encoding is that one column can suddenly become many. This can start to become a handful when we have 100s of categories being converted into 100s of input curves. However, for most petrophysical work, this is unlikely to become a huge issue due to the smaller amount of categories and formation names we work with.

One column too many: dropping the first category

There is one wrinkle worth understanding before we use One Hot Encoding, because it appears as an argument in most encoders. When you one hot encode a formation with, say, four categories, you get four binary columns. But, those four columns carry a hidden redundancy: because every sample belongs to exactly one formation, the four columns always sum to one across any row. Knowing three of them tells you the fourth for free. If it is not Heather, Hod, or Hugin, it must be Draupne.

For many models that redundancy is harmless, but for linear regression it is a real problem. A regression usually includes an intercept, which is effectively a column of all ones. Since the full set of dummy columns also sums to one on every row, the dummies and the intercept become perfectly collinear, and the regression can no longer find a unique solution. This is sometimes called the dummy variable trap.

The fix is to drop one category, which is what drop="first" does in scikit-learn (pd.get_dummies offers the same through drop_first=True). The dropped category does not disappear from the model. It becomes the baseline, and every remaining coefficient is then read as the effect relative to that baseline.

For tree-based models and regularised models the collinearity is not fatal, so you will often see the full set of columns kept there, but for a plain linear regression, dropping one is the safe habit.

A real example from Volve

To see how much the choice actually matters, we can test both encodings on the same task with real data: well 15/9–19 A from Equinor’s openly released Volve dataset, with formation tops from the Norwegian Offshore Directorate factpages. The task is predicting the sonic curve (DT) from gamma ray, bulk density and the formation column, using a multiple linear regression.

First, we load the well and assign a formation to each depth sample from the tops:

import lasio
import numpy as np

tops = pd.read_csv(”wellbore_formation_top.csv”, encoding=”utf-8-sig”)

def add_formations(df, wellbore_name):
    t = tops[(tops.wlbName == wellbore_name) & (tops.lsuLevel == “FORMATION”)]
    out = pd.Series(None, index=df.index, dtype=object)
    for _, row in t.iterrows():
        mask = (df.MD >= row.lsuTopDepth) & (df.MD < row.lsuBottomDepth)
        out[mask] = row.lsuName.replace(” FM”, “”).title()
    return out

df = lasio.read(”159-19A_LFP.las”).df().reset_index()[[”DEPTH”, “LFP_GR”, “LFP_DT”, “LFP_RHOB”]]
df.columns = [”MD”, “GR”, “DT”, “RHOB”]
df[”formation”] = add_formations(df, “15/9-19 A”)
df = df.dropna(subset=[”GR”, “DT”, “RHOB”, “formation”])

That gives us 3,850 samples across ten formations, from the Blodøks down to the Skagerrak. Before modelling anything, it is worth looking at what label encoding would do to this well. Plotting the mean DT of each formation against its alphabetical integer code:

The means zigzag along the alphabet. The Draupne hot shale sits up at around 120 us/ft in second position, between the much faster Blodøks and Heather, and the chalks and sands land wherever their names happen to fall. The dashed line shows the single slope a linear model would be forced to fit through that. It is wrong for nearly every formation at once.

Now the test. We fit the same regression twice, once with the label encoded formation column and once with the one hot version, and score both with shuffled 5-fold cross validation:

from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import KFold, cross_val_score

le = LabelEncoder()
df[”form_code”] = le.fit_transform(df[”formation”])
y = df[”DT”].values
X_int = df[[”GR”, “form_code”, “RHOB”]].values
ohe = OneHotEncoder(sparse_output=False, drop=”first”)
X_ohe = np.column_stack([df[[”GR”, “RHOB”]].values, ohe.fit_transform(df[[”formation”]])])

kf = KFold(n_splits=5, shuffle=True, random_state=42)

kf = KFold(n_splits=5, shuffle=True, random_state=0)
r2_int = cross_val_score(LinearRegression(), X_int, y, cv=kf, scoring=”r2”).mean()
r2_ohe = cross_val_score(LinearRegression(), X_ohe, y, cv=kf, scoring=”r2”).mean()
print(f”label encoded: R2 = {r2_int:.3f} | one hot: R2 = {r2_ohe:.3f}”)

The label encoded model scores an R squared of 0.613. The one hot model scores 0.888.

Even though this is a simple model, we have the same data, same model, and the same validation process, with the only difference between them is how the formation names were turned into numbers.

Summary

Encoding is the step where categorical data becomes something a model can use, and it is a choice rather than a formality. Label encoding is compact but invents an ordering, and unless you assign the codes deliberately, that ordering is alphabetical and geologically meaningless. One hot encoding trades a few extra columns for making no assumptions at all, and for the unordered categories that dominate petrophysical work, formations, facies, lithologies, that is usually the right trade. On a real Volve well, it was the difference between an R squared of 0.613 and 0.888.

Leave a comment

Data: Equinor Volve open dataset (well logs) and the Norwegian Offshore Directorate factpages (formation tops), used with thanks under their respective licences.

The example in this article used a linear regression, to keep the focus on the encoding rather than the model. If you would rather see curve prediction done with a neural network, I have a walkthrough on Substack that builds one in Keras to predict bulk density from other logs, using the same Volve data. It also shows how much of a difference normalising the inputs makes to the result.

→ Well Log Measurement Prediction Using Neural Networks with Keras

PtitPrince can create a raincloud plot in a few lines of code. That is one of its main attractions, especially when you want a quick view of how several distributions compare.

The default output is useful for exploration, but it still looks like something produced inside a notebook. The layout can feel cramped, the labels come directly from the DataFrame, and the relative weight of the cloud, box and individual points is left to the library defaults. This may be fine whilst you are analysing the data, but less so when the figure needs to appear in a report, presentation or article.

To continue with my series on From Default to Publication Ready, this article will take the default PtitPrince output and improve it one step at a time. We will change the orientation, apply a more accessible colour palette, rebalance the components, improve the labels, remove unnecessary clutter and add sample sizes.

The idea is not to produce one perfect design, but to show the small decisions that can turn a basic working plot into a figure that can easily be included in a publication and communicates its message clearly.

If you are new to raincloud plots, I covered the individual components in an earlier article.

The data

For this article, we will generate some synthetic data that represents porosity distributions from four different geological formations.

The names and values are illustrative rather than real geology: two roughly normal distributions at different levels, one bimodal formation, and one that is skewed. This mix allows us to visualise different data distributions and to show the benefits of a raincloud plot

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import ptitprince as pt

rng = np.random.default_rng(42)
formations = {
    "Vardan": rng.normal(0.21, 0.025, 400),
    "Kelvik": rng.normal(0.15, 0.02, 320),
    "Stenby": np.concatenate([rng.normal(0.10, 0.015, 20),
                              rng.normal(0.18, 0.015, 20)]),
    "Morvik": rng.beta(2, 5, 20) * 0.25,
}
df = pd.DataFrame(
    [(name, val) for name, vals in formations.items() for val in vals],
    columns=["Formation", "Porosity"],
)

With the data in place, we can call upon the raincloud plot like so:

f, ax = plt.subplots(figsize=(8, 6))
pt.RainCloud(x="Formation", y="Porosity", data=df, ax=ax)
plt.show()

The default output is readable, however, the vertical layout squashes each cloud into a narrow column and we may start competing for space on the x-axis when we add more formations or formations with longer names.

Step 1: Flip the orientation

Raincloud plots often read better horizontally, in my opinion, particularly when the group labels are long or several distributions are being compared.

The cloud gets room to spread out and the rain falls in a wide band instead of a narrow vertical strip, and we can display the group names on the y axis, which can be great when we have long names and need to avoid rotating the labels.

f, ax = plt.subplots(figsize=(10, 7))
pt.RainCloud(x="Formation", y="Porosity", data=df, ax=ax,
             orient="h")
plt.show()

Horizontal raincloud plot giving each formation’s cloud and rain room to spread, with formation names on the y axis

We have also changed the figure size to give more width than height.

Step 2: Apply a colour palette

The default blue colour is fine if you want to keep the groups consistent, but sometimes it is good to be able to split the groups apart by colour, especially if we have a handful of groups.

We can easily apply a different colour palette, and it often helps to pick one that is easily accessible to readers, especially those that have varying forms of colour blindness. Additionally, adding a slight bit of transparency can also help soften the colours if they appear to harsh.

f, ax = plt.subplots(figsize=(10, 7))
pt.RainCloud(x="Formation", y="Porosity", data=df, ax=ax,
             orient="h",
             palette="colorblind", alpha=0.65)
plt.show()

The same chart using seaborn’s colorblind palette with softened fill transparency

The colorblind palette ships with seaborn, so there’s nothing extra to install.

Step 3: Tune the cloud, the rain and the box

A raincloud plot consists of three components sharing a single axis, and the defaults for the plot give each component an arbitrary share of the space.

With a few width and jitter changes, we can rebalance them so the cloud dominates, the box stays a compact summary, and the rain forms a clear band underneath.

f, ax = plt.subplots(figsize=(10, 7))
pt.RainCloud(x="Formation", y="Porosity", data=df, ax=ax,
             orient="h",
             palette="colorblind", alpha=0.65,
             width_viol=0.7, width_box=0.25,
             point_size=3, jitter=0.08, move=0.25,
             box_showfliers=False)
plt.show()

Rebalanced widths with a fuller cloud, slimmer box, smaller rain points, tighter jitter and no duplicate outlier markers

The values here are a reasonable starting point rather than a rule. width_viol and width_box control how much room the cloud and box take, point_size and jitter control how the rain reads, and move shifts the rain away from the box so the two don’t overprint. With more groups or more points per group, these tend to need another review.

The boxplot layer displays outlier markers by default. Setting box_showfliers=False removes them. In a raincloud plot, displaying them can amount to double counting because every observation, including the outliers, is already shown in the rain.

By having this on, matplotlib draws hollow circles for its outliers, exactly as a standalone boxplot would. In a raincloud that’s double counting: every point, outliers included, is already in the rain. The box’s outlier markers sit at the box position rather than in the rain band, so they read as extra data that isn’t there. ptitprince routes any argument prefixed with box_ through to the underlying boxplot, which is also how you’d pass other boxplot settings if you need them.

Rainclouds are one of a handful of chart types that most people never reach for, even when they’d tell the story far better than a histogram. I’ve covered three of them, with before and after examples on well log data, over on Substack. Link at the end.

Step 4: Say what the chart shows

The chart still labels its axes with DataFrame column names, which is not the best, especially if the names are more generic or contain special characters such as an underscore or dash.

We can tidy this up with a few small changes:

ax.set_xlabel("Porosity (v/v)", fontsize=12)
ax.set_ylabel("")
ax.set_title("Porosity Distribution by Formation",
             fontsize=14, fontweight="bold", loc="left", pad=15)

The empty y label is intentional. The formation names already tell the reader what that axis is, and a “Formation” label would repeat them.

For reports and presentations, a left-aligned title is easy to scan and works well on a wide figure. Journal figures often leave the title to the caption, so this is a design choice rather than a fixed rule.

Step 5: Remove what isn’t data

Matplotlib draws a full box around every plot by default. This is often considered “chart junk” by many and can make plots look ugly or distract the reader.

By adding light grid along the porosity axis we can help the reader’s eye across the chart. This is the quiet step that does the most for how finished the figure feels.

ax.spines[["top", "right"]].set_visible(False)
ax.xaxis.grid(True, linestyle="--", alpha=0.4)
ax.set_axisbelow(True)

Labels and decluttering combined. Titled chart with axis units, open top and right spines, and a light dashed grid behind the data

set_axisbelow(True) pushes the gridlines behind the clouds and rain. Without it, dashed lines cut across your data, which can look messy.

Step 6: Show your sample sizes, then export properly

A raincloud plot invites the reader to trust the cloud, and the cloud is a smoothed estimate. When looking at the rain, it can also be difficult for the reader to estimate how many points are present for each group. For example between Vardan and Kelvik there are only 80 points difference, however, it would be difficult to know that by looking at the rain.

As a result the sample count is often worth putting directly on the figure, particularly when the groups are uneven. We can append the counts directly onto the group label on the y-axis rather than adding additional annotations.

counts = df.groupby("Formation", sort=False).size()
labels = [f"{name}\n(n={counts[name]})" for name in counts.index]
ax.set_yticks(ax.get_yticks())
ax.set_yticklabels(labels)

The finished figure. Publication-ready horizontal raincloud with sample counts under each formation name.

The Final Script

Here’s everything in one place, from data to exported figure.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import ptitprince as pt

rng = np.random.default_rng(42)
formations = {
    "Vardan": rng.normal(0.21, 0.025, 400),
    "Kelvik": rng.normal(0.15, 0.02, 320),
    "Stenby": np.concatenate([rng.normal(0.10, 0.015, 20),
                              rng.normal(0.18, 0.015, 20)]),
    "Morvik": rng.beta(2, 5, 20) * 0.25,
}
df = pd.DataFrame(
    [(name, val) for name, vals in formations.items() for val in vals],
    columns=["Formation", "Porosity"],
)
f, ax = plt.subplots(figsize=(10, 7))
pt.RainCloud(x="Formation", y="Porosity", data=df, ax=ax,
             orient="h",
             palette="colorblind", alpha=0.65,
             width_viol=0.7, width_box=0.25,
             point_size=3, jitter=0.08, move=0.25,
             box_showfliers=False)
ax.set_xlabel("Porosity (v/v)", fontsize=12)
ax.set_ylabel(")
ax.set_title("Porosity Distribution by Formation",
             fontsize=14, fontweight="bold", loc="left", pad=15)
ax.spines[["top", "right"]].set_visible(False)
ax.xaxis.grid(True, linestyle="--", alpha=0.4)
ax.set_axisbelow(True)
counts = df.groupby("Formation", sort=False).size()
labels = [f"{name}\n(n={counts[name]})" for name in counts.index]
ax.set_yticks(ax.get_yticks())
ax.set_yticklabels(labels)
plt.tight_layout()
plt.savefig("porosity_raincloud.png", dpi=600, bbox_inches="tight")

Closing Thoughts

The difference between a default raincloud plot and a finished figure is not a complicated block of code, but rather a series of small and deliberate choices.

Changing the orientation gives the distributions more room. Adjusting the palette and transparency can help improve readability. Tuning the cloud, box and rain makes each component contribute without competing for attention and for space. Clear labels provide context, while removing unnecessary spines and adding a light grid help the data stand out. Showing the sample sizes gives the reader a better sense of how much evidence sits behind each distribution.

Not every figure needs exactly the same settings. More groups, longer labels or larger datasets may require different dimensions, point sizes or spacing. The useful habit is to treat the default output as a starting point rather than the finished result.

One setting we have deliberately left untouched is the bandwidth used to estimate the cloud. That choice affects how smooth, noisy or even multimodal a distribution appears, and it can change the story the plot seems to tell. This was covered briefly in Raincloud Plots with PtitPrince: See What Your Data Is Really Doing.

Polishing a raincloud plot only pays off if the raincloud was the right chart to begin with. Alongside it sit ridgeline plots, bump charts and beeswarm plots. Over on Substack I’ve put together a piece covering all three, with before and after examples using well log data, so you can decide which one your data deserves.

→ 3 Python Charts You’re Probably Not Using (But Should Be)

In my last article I built a raincloud plot from scratch in matplotlib. This was a combination of a half-violin for the shape (cloud), the raw points jittered underneath (the rain), and a boxplot at the bottom giving a summary of the data distribution.

Even though we can build the plot by hand (and as usual) there is a dedicated library which wraps most of that up. This allows you to write fewer lines of code to achive what you are looking for.

ptitprince is the Python port of the original RainCloudPlots project that introduced the chart, and it sits on top of seaborn.

Setup and data

First, we need to install the library from PyPI:

pip install ptitprince

For the data, we are using a synthetic dataset representing porosity for four geological formations, built to show different distributions we may expect in the subsurface.

Formation A is bimodal, the sort of split you get when a sand and an interbedded shale share a zone. B is tight and unimodal, C is right-skewed, and D has only eighteen samples.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import ptitprince as pt

rng = np.random.default_rng(42)

n = 300
form_a = np.concatenate([rng.normal(0.10, 0.015, n // 2),
                         rng.normal(0.22, 0.020, n // 2)])   # bimodal: sand/shale mix
form_b = rng.normal(0.18, 0.012, n)                          # tight, unimodal
form_c = 0.06 + rng.gamma(2.0, 0.018, n)                     # right-skewed
form_d = rng.normal(0.15, 0.020, 18)                         # very small sample

data = pd.DataFrame({
    "formation": (["A"] * len(form_a) + ["B"] * len(form_b)
                  + ["C"] * len(form_c) + ["D"] * len(form_d)),
    "porosity": np.concatenate([form_a, form_b, form_c, form_d]),
})

Generating the Basic Raincloud Plot

The library’s main entry point is RainCloud, and the basic call looks similar to how seaborn calls plots.

fig, ax = plt.subplots(figsize=(8, 5))
pt.RainCloud(x="formation", y="porosity", data=data, orient="h", ax=ax)
ax.set_xlabel("Porosity (frac)")
ax.set_ylabel("Formation")

Once we run the above code, we get back the following chart.

The default raincloud output from ptitprince.

With essentially all three layers are there for each formation, we can gain a good understanding of our data very quickly.

Compared with the thirty-odd lines of the hand-built version, that is a real saving.

The defaults are not perfect and almost looks like a basic matplotlib plot. Nothing wrong with that, but if you have followed my previous articles you can see how much I enjoy transforming matplotlib figures into something much better looking.

In this case, the boxplot sits directly on top of the rain, so the box hides some of the points it is meant to summarise, and the outlier fliers on the box duplicate points that are already visible in the rain. The layers are all present, but they are all fighting for the same space.

Tidying Up the Raincloud Plot

Customisation in ptitprince works in two ways.

A handful of top-level parameters control the layout, and everything else passes through to the underlying seaborn calls using prefixed keyword arguments: anything starting cloud_ goes to the half-violin, box_ to the boxplot, rain_ to the strip plot, and point_ to the optional mean marker.

The single most useful parameter I have found to tidy up the overlapping boxplot is move This shifts the rain (points) downwards and away from the box:

fig, ax = plt.subplots(figsize=(8, 5))
pt.RainCloud(
    x="formation", y="porosity", data=data, orient="h", ax=ax,
    move=0.2,                 # shift the rain clear of the box
    width_viol=0.7,           # cloud width
    point_size=3,             # rain marker size
    alpha=0.6,                # transparency for cloud and rain
    box_showfliers=False,     # outliers already visible in the rain
)
ax.set_xlabel("Porosity (frac)")
ax.set_ylabel("Formation")

Tidied up version of the rain cloud plots with the rain sittin clear of the box, no fliers and no competing or overlapping elements.

By changing a few extra arguments we can make the chart tidier and easier to read.

Each layer now has it’s own space, the duplicated outliers are gone, and the transparency lets dense regions of rain show as darker bands. This is enough for making something quickly to help with the analysis.

Making it publication ready

For a figure going into a report or a paper, the tidied version still needs the usual finishing work: consistent colours, axis labels with units, sample counts, a title, and gridlines.

There are two ways we can get there.

The first is the prefixed kwargs handle anything that belongs to a layer, and the second, is by adding standard matplotlib calls because RainCloud draws onto an ordinary matplotlib Axes.

formations = ["A", "B", "C", "D"]
colours = ["#4C72B0", "#55A868", "#C44E52", "#8172B3"]

fig, ax = plt.subplots(figsize=(9, 6), facecolor="white")
pt.RainCloud(
    x="formation", y="porosity", data=data, orient="h", ax=ax,
    palette=colours,
    move=0.2, width_viol=0.7, point_size=3, alpha=0.6,
    box_showfliers=False,
    box_linewidth=1.2,
    box_medianprops={"color": "#2c3e50", "linewidth": 1.6},
)

# Post-hoc polish on the returned axes
counts = data.groupby("formation")["porosity"].size()
labels = [f"{f}\n(n={counts[f]})" for f in formations]
ax.set_yticks(range(len(formations)))
ax.set_yticklabels(labels)
ax.set_xlabel("Porosity (v/v)", fontsize=12, fontweight="bold", labelpad=10)
ax.set_ylabel("Formation", fontsize=12, fontweight="bold", labelpad=10)
ax.set_title("Porosity Distribution by Formation", fontsize=14, fontweight="bold", pad=15)
ax.grid(axis="x", alpha=0.3, linestyle="--", linewidth=0.8)
ax.tick_params(axis="both", labelsize=10)

plt.tight_layout()
plt.savefig("raincloud_porosity.png", dpi=150, bbox_inches="tight", facecolor="white")

Another tidied up version of the raincloud plot that would be more suited for including in apublication.

The split matters more than the individual styling choices. Layer properties, like the box line weight and median colour, go through the prefixed kwargs.

Everything that belongs to the axes rather than a layer, such as labels, title, gridlines, and the tick text, is plain matplotlib after the raincloud plot call.

The sample counts under each formation name do the same work the rain does, but provides an instant number for the reader and saves them having to count each point (very few people will do that!).

This is also where you meet the limits of ptitprince’s raincloud plot. If you need to restyle an element the kwargs don’t reach and you then end up digging through the axes’ children to find it, and at that point the hand-built version from the previous article can appear to be less work, not more.

The Parameter That Changes the Interpretation of the Cloud

One argument is worth singling out when creating the raincloud plot, because it can change what the chart says rather than how it looks.

bw sets the bandwidth of the kernel density estimate behind the cloud, and it controls how much smoothing is applied to the shape you see.

fig, axs = plt.subplots(1, 3, figsize=(12, 4), sharey=True)
for ax, bw in zip(axs, [0.1, 0.3, 0.8]):
    pt.RainCloud(x="formation", y="porosity",
                 data=data[data.formation == "A"],
                 orient="h", ax=ax, bw=bw, move=0.2, point_size=2.5)
    ax.set_title(f"bw = {bw}")
    ax.set_xlabel("Porosity (frac)")
    ax.set_ylabel("")
axs[0].set_ylabel("Formation")
plt.tight_layout()

The same bimodal formation drawn at three bandwidths. Two sharp humps at 0.1, two clear humps at 0.3, one broad hump at 0.8

This is the same formation A in all three panels.

At a bandwidth of 0.1 the two populations are sharp and separate.

At 0.3 they are still clearly there.

At 0.8 they have merged into one broad hump, and the cloud now tells you this is a single, wide distribution.

For a petrophysicist that is the difference between one lithology and a mixed zone, decided by a single smoothing parameter.

The rain is the safeguard, because the two clusters of points stay where they are, no matter what the smoothing does. If the cloud and the rain disagree about how many populations you have, then we should believe the rain.

When To Use ptitprince or matplotlib

For exploratory work the ptitprince library is an easy choice. One call gets you a complete, readable raincloud for a first look at new data or a quick comparison across formations, and move plus a couple of styling arguments carry it a long way. With axes-level polish on top, it will take you all the way to a publication figure too.

The hand-built version is for when the wrapper gets in the way, for example when an element the prefixed kwargs can’t reach, or an environment where a thinly maintained dependency is more trouble than thirty lines of matplotlib you own and can control.

Either way, the same rule from the previous article applies, and figure 4 is the reason: the cloud is an estimate and the rain is the data. Whichever tool draws the chart, when the two disagree, believe the rain.

Leave a comment

Throughout this article I have mentioned the hand-built version as the fallback for when the wrapper gets in the way. If you want to see how that works, I have written a full walkthrough on Substack where I build the raincloud plot from scratch in matplotlib, layer by layer, using the same four-formation dataset as this article.

→ From Boxplot to Raincloud: Building a Better Distribution Plot in Python

The boxplot is the common tool for comparing measurements across different geological formations, and it can also hide the thing you need to see. Boxplots reduce a distribution to five key numbers, so two formations with completely different shapes can produce similar looking boxes.

This is where raincloud plots can help. A raincloud plot stacks three views of a distribution in one: a half-violin for the shape (the “cloud”), the raw points jittered underneath (the “rain”), and a boxplot for the summary. You get the quick read of a boxplot without letting it hide what the data is actually doing.

Even though there there is a ready-made library for this in Python (ptitprince), we will be building one by hand using matplotlib. Once you have built it you can see precisely what the chart is showing.

The data

For this article, we will generate some synthetic data that represents porosity distributions from four different geological formations.

These distributions were chosen to show off the chart and may not necessarily represent real geology.

We can also import the required libraries, which are numpy, pandas, matplotlib and scipy.stats.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from scipy.stats import gaussian_kde

rng = np.random.default_rng(42)

n = 300
form_a = np.concatenate([rng.normal(0.10, 0.015, n // 2),
                         rng.normal(0.22, 0.020, n // 2)])   # bimodal: sand/shale mix
form_b = rng.normal(0.18, 0.012, n)                          # tight, unimodal
form_c = 0.06 + rng.gamma(2.0, 0.018, n)                     # right-skewed
form_d = rng.normal(0.15, 0.020, 18)                         # very small sample

data = pd.DataFrame({
    “formation”: ([”A”] * len(form_a) + [”B”] * len(form_b)
                  + [”C”] * len(form_c) + [”D”] * len(form_d)),
    “porosity”: np.concatenate([form_a, form_b, form_c, form_d]),
})

formations = [”A”, “B”, “C”, “D”]
groups = [data.loc[data.formation == f, “porosity”].values for f in formations]
colours = [”#4C72B0”, “#55A868”, “#C44E52”, “#8172B3”]

Formation A is built from two normal distributions to represent a bimodal distribution which you might see when you have say sand and shale in one formation. Formation B is tight and unimodal, Formation C is right-skewed, and Formation D has only eighteen points.

Keep this mix of data distributions in mind, because the whole argument for the raincloud is that these four behave very differently and a boxplot will not tell you so.

Visualising Data on a Boxplot

Before building anything new, plot the boxplot you would normally reach for.

fig, ax = plt.subplots(figsize=(8, 5))
ax.boxplot(groups, vert=False, widths=0.5)
ax.set_yticks(range(1, len(formations) + 1))
ax.set_yticklabels(formations)
ax.set_xlabel(”Porosity (frac)”)

ax.set_ylabel(”Formation”)

If we look at formation A, it shows a wide box across the whole porosity range, its median sits in the middle, and nothing about it suggests anything unusual. As we will see shortly, the median is sitting in the empty gap between two populations in the bimodal distribution, where almost no data actually sits.

Building the Raincloud Plot

Layer one: the cloud

The cloud layer is a half-violin plot. This is kernel density estimate drawn on one side only. We compute the density with gaussian_kde, scale it to a fixed height so every formation is comparable, and fill it in above the formation’s baseline.

def add_cloud(ax, values, ypos, width=0.35, colour=”#4C72B0”):
    kde = gaussian_kde(values)
    xs = np.linspace(values.min(), values.max(), 200)
    dens = kde(xs)
    dens = dens / dens.max() * width
    ax.fill_between(xs, ypos, ypos + dens, color=colour, alpha=0.5, linewidth=0)

Building xs between values.min() and values.max() clips the density to the actual range of the data, so the cloud can’t drift past your real readings the way a default violin does.

Dividing by dens.max() normalises every cloud to the same peak height, so you are comparing shapes rather than raw densities between each of the formations.

That normalisation has a cost worth knowing about: it makes an 18-point dataset can look as substantial as a 300-point dataset, which is exactly why the raw data goes in underneath. Drawing the cloud on one side keeps it compact and leaves room for those points.

Plotting all four formations:

fig, ax = plt.subplots(figsize=(8, 5))
for i, (vals, c) in enumerate(zip(groups, colours)):
    add_cloud(ax, vals, i, colour=c)
ax.set_yticks(range(len(formations)))
ax.set_yticklabels(formations)
ax.set_xlabel(”Porosity (frac)”)
plt.show()

Formation A now shows its two humps plainly, which represents our bimodal distribution, C shows its left skew, and B is the narrow spike you would expect from a tight distribution.

Whilst we can see the distributions of the data a density curve has its own problem: it is a smooth estimate, and with few points it will invent a smoothness the data can’t support.

This is why we need to add the actual data points.

Layer two: the rain

The rain is the raw observations, scattered below the cloud with a little vertical jitter so points do not pile up and hide each other.

The jitter is purely cosmetic on the vertical axis and carries no meaning. The horizontal position is the real value.

Adding the rain under each cloud can be done using the following function

def add_rain(ax, values, ypos, spread=0.12, gap=0.06, colour=”#4C72B0”):
    jitter = rng.uniform(0, spread, size=len(values))
    ax.scatter(values, ypos - gap - jitter, s=8, color=colour, alpha=0.5,
               edgecolor=”none”)

and then recreating the plot:

fig, ax = plt.subplots(figsize=(8, 5))
for i, (vals, c) in enumerate(zip(groups, colours)):
    add_cloud(ax, vals, i, colour=c)
    add_rain(ax, vals, i, colour=c)
ax.set_yticks(range(len(formations)))
ax.set_yticklabels(formations)
ax.set_xlabel(”Porosity (frac)”)
plt.show()

Formation D shows a smooth, reasonable distribution, but the rain shows only a handful of points. This smoothness was a result of the KDE filling in gaps .

The rain layer shows the real observations, how they are distributed and how many of them there are.

Layer three: the box

The summary generated by the boxplot is still worth keeping and can be added a thin horizontal box between the cloud and the rain.

The trick with matplotlib is setting manage_ticks=False, which stops the boxplot from resetting the y axis we have carefully set up.

def add_box(ax, values, ypos):
    ax.boxplot([values], positions=[ypos], vert=False, widths=0.08,
               manage_ticks=False, showfliers=False,
               medianprops=dict(color=”black”))

With all three layers in place we can generate the final plot:

fig, ax = plt.subplots(figsize=(8, 5))
for i, (vals, c) in enumerate(zip(groups, colours)):
    add_cloud(ax, vals, i, colour=c)
    add_box(ax, vals, i)
    add_rain(ax, vals, i, colour=c)
ax.set_yticks(range(len(formations)))
ax.set_yticklabels(formations)
ax.set_xlabel(”Porosity (frac)”)
ax.set_ylabel(”Formation”)
ax.set_ylim(-0.4, len(formations) - 0.3)
plt.show()

For each formation we can now see the shape, the raw data, and the summary at once, and you can check them against each other.

Formation A’s whiskers still span the whole range, but now the bimodal distribution and the gap in data points between can be seen.

Formation D’s tidy little box sits over a handful of points, so you know to discount it.

The chart makes the gap between summary and data visible instead of hiding it.

The Shortcut to a Raincloud Plot

As mentioned at the start of the article, there is also a dedicated library called PtitPrince for generating this type of plot, which has different ways of displaying them. ptitprince (GitHub — pog87/PtitPrince: python version of raincloud · GitHub) wraps all of this into roughly one call:

import ptitprince as pt

pt.RainCloud(x=”formation”, y=”porosity”, data=data, orient=”h”)

It is a fine option, though it has occasionally lagged behind newer seaborn and matplotlib versions, so check it works in your environment. The reason to keep the hand-built version is control: orientation, spacing, colour per group, how much jitter, whether the box shows outliers. Once the three helpers exist, you own all of that.

Where The Raincloud Plot Can Fall Over

The raincloud shouldn’t be seen as a default for every situation. Beyond a handful of groups it starts to get busy, because the eye has three elements per row to track, and a ridgeline plot reads more cleanly when you only care about shape across many groups. With well log data running to tens of thousands of samples, the rain turns into a solid smear and you have to thin it with transparency or subsampling, at which point it can mean less. And as formation D shows, with very few points the cloud is mostly interpolation, so trust the rain and treat the cloud as decoration.

Used in the right place, though, the plot can be a useful tool in the toolbox.

The density cloud is what makes a raincloud a raincloud, but it isn’t always worth the extra layer. Sometimes keeping the boxplot and overlaying just the raw points tells you everything you need, especially in exploratory work when you only want to check whether the summary is hiding something. I’ve written that simpler version up on Substack, building it in seaborn with group means and sample-size labels, along with where it starts to break down.

→ Boxplots Are Hiding Your Data. Here’s a Simple Fix

Reference: Allen M, Poggiali D, Whitaker K, Marshall TR, van Langen J, Kievit RA. Raincloud plots: a multi-platform tool for robust data visualization. Wellcome Open Research 2021, 4:63. https://wellcomeopenresearch.org/articles/4-63

Understanding data completeness, is an essential part of any data analysis workflow. In the case of well log data, we often have similarly acquired datasets in multiple wells. One way I like to check this is by using a completeness heatmap.

At a glance the data completeness heatmap tells you which curves (or columns) you have across multiple measurement locations. In this example we are using different wells.

Form this we can also tell which curves are missing from which wells, and whether the gaps are concentrated in a particular formation or scattered across the well.

In my previous article, I have focused on how to improve the look of static matplotlib figures and there is a problem with them.

That problem is that static figures, even well-made ones, make your brain work to answer follow-up questions. In cases where we have a large amount of columns and rows in our heatmap, you simply can’t hover over a cell and see exactly which well (row) and curve (column) combination you’re looking at. Instead you have to look up which column and row you are in.

Plotly, another excellent Python plotting library fixes most of that.

In this article we will walk through building an interactive well log completeness heatmap from scratch, with hover tooltips, cell labels, white borders, and a multi-formation panel.

Although this example uses well log data, the same approach works anywhere you need to compare data completeness across repeated measurement locations, whether that means wells, stations, assets, samples, or sites.

The Data

First, we will create a dataset. This dataset is a synthetic set of 8 wells, 10 curves, and 4 formations. Ideally, you would need to load and compute these completeness values fo your own dataset.

import pandas as pd

wells  = [’Ardmore-1’, ‘Ardmore-4’, ‘Brindal-1’, ‘Brindal-3’,
          ‘Brindal-5’, ‘Brindal-8’, ‘Caxton-2’,  ‘Caxton-6’]

curves = [’GR’, ‘RDEP’, ‘RSHA’, ‘RHOB’, ‘NPHI’, ‘DTC’, ‘DTS’, ‘PEF’, ‘CALI’, ‘SP’]

completeness_data = {
    ‘Thornfield’: [
        [100.0, 100.0, 100.0, 26.2, 100.0, 40.8, 61.5, 49.5, 100.0, 54.0],  # Ardmore-1
        [100.0, 100.0, 100.0, 28.0, 100.0, 37.2, 60.8, 52.8, 100.0, 54.5],  # Ardmore-4
        [100.0, 100.0, 100.0, 31.5, 100.0, 38.8, 59.8, 50.5, 100.0, 54.0],  # Brindal-1
        [100.0, 100.0, 100.0, 29.8, 100.0, 41.5, 60.0, 46.5, 100.0, 52.8],  # Brindal-3
        [100.0, 100.0, 100.0, 30.2, 100.0, 37.5, 59.0, 52.2, 100.0, 56.5],  # Brindal-5
        [100.0, 100.0, 100.0, 31.0, 100.0, 40.2, 59.0, 53.5, 100.0, 55.8],  # Brindal-8
        [100.0, 100.0, 100.0, 35.2, 100.0, 37.5, 58.0, 50.7, 100.0, 54.8],  # Caxton-2
        [100.0, 100.0, 100.0, 31.0, 100.0, 43.2, 62.5, 46.5, 100.0, 54.2],  # Caxton-6
    ],
    ‘Veldwick’: [
        [100.0, 100.0, 100.0, 75.0, 100.0, 65.1, 59.8, 52.1, 100.0, 57.9],  # Ardmore-1
        [100.0, 100.0, 100.0, 87.0, 100.0, 73.8, 60.1, 47.8, 100.0, 53.2],  # Ardmore-4
        [100.0, 100.0, 100.0, 84.0, 100.0, 72.0, 62.0, 50.2, 100.0, 58.1],  # Brindal-1
        [100.0, 100.0, 100.0, 82.0, 100.0, 69.6, 62.6, 49.9, 100.0, 56.4],  # Brindal-3
        [100.0, 100.0, 100.0, 81.5, 100.0, 69.9, 59.6, 50.1, 100.0, 52.8],  # Brindal-5
        [100.0, 100.0, 100.0, 88.6, 100.0, 73.5, 59.4, 49.0, 100.0, 55.6],  # Brindal-8
        [100.0, 100.0, 100.0, 75.1, 100.0, 64.5, 59.6, 49.4, 100.0, 58.2],  # Caxton-2
        [100.0, 100.0, 100.0, 84.9, 100.0, 73.0, 61.0, 47.5, 100.0, 55.4],  # Caxton-6
    ],
    ‘Oskarby’: [
        [100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 80.4, 84.4, 100.0, 52.8],  # Ardmore-1
        [100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 85.5, 93.4, 100.0, 58.3],  # Ardmore-4
        [100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 84.1, 91.2, 100.0, 52.3],  # Brindal-1
        [100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 83.2, 90.9, 100.0, 56.1],  # Brindal-3
        [100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 84.0, 88.9, 100.0, 54.3],  # Brindal-5
        [100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 88.6, 93.4, 100.0, 53.7],  # Brindal-8
        [100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 80.3, 86.3, 100.0, 53.3],  # Caxton-2
        [100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 86.2, 91.3, 100.0, 55.0],  # Caxton-6
    ],
    ‘Dravenport’: [
        [100.0, 100.0, 100.0,  80.0,  80.0, 100.0, 100.0, 100.0, 100.0, 54.5],  # Ardmore-1
        [100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 56.5],  # Ardmore-4
        [100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 52.8],  # Brindal-1
        [100.0, 100.0, 100.0,  80.0,  80.0, 100.0, 100.0, 100.0, 100.0, 55.8],  # Brindal-3
        [100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 52.5],  # Brindal-5
        [100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 52.8],  # Brindal-8
        [100.0, 100.0, 100.0,  80.0,  80.0, 100.0, 100.0, 100.0, 100.0, 55.2],  # Caxton-2
        [100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 55.9],  # Caxton-6
    ],
}

One line to turn that into the results dict the rest of the article uses. In this dictionary we have a pandas dataframe per formation, wells as rows, curves as columns:

results = {
    formation: pd.DataFrame(matrix, index=wells, columns=curves)
    for formation, matrix in completeness_data.items()
}

formation_names = list(results.keys())

Displaying the Basic Heatmap with px.imshow()

To begin displaying our heatmap we can call upon px.imshow() .

px.imshow() is Plotly Express’s equivalent of matplotlib’s ax.imshow(). You pass it a 2D array or dataframe and it renders a colour-mapped grid.

import plotly.express as px

completeness_pivot = results[’Veldwick’]
fig = px.imshow(
    completeness_pivot,
    color_continuous_scale=”RdYlGn”,
    zmin=0,
    zmax=100,
    labels={”color”: “Completeness (%)”},
    title=”Well Log Completeness - Veldwick”,
)
fig.show()

Single-formation completeness heatmap for Veldwick using px.imshow().

The zmin and zmax arguments pin the colour scale to 0-100, which matters when you’re comparing formations.

Without these values being defined, Plotly autoscales to the data range.

Adding Hover Tooltips

By default px.imshow() shows the x, y, and z values on hover.

To control exactly what appears, and how it’s formatted we need to pass a hovertemplate via update_traces. The formatting can include regular HTML syntax, which makes it easy to style, and we can easily call upon the x and y values with %{y} , which is the well (row) and %{x} which is the curve (column).

fig = px.imshow(
    completeness_pivot,
    color_continuous_scale=”RdYlGn”,
    zmin=0,
    zmax=100,
    labels={”color”: “Completeness (%)”},
    title=”Well Log Completeness — Veldwick”,
)

fig.update_traces(
    hovertemplate=(
        “<b>Well:</b> %{y}<br>”
        “<b>Curve:</b> %{x}<br>”
        “<b>Completeness:</b> %{z:.1f}%<extra></extra>”
    ),
)
fig.show()

The <extra></extra> tag removes the trace name Plotly appends by default.

%{z:.1f} formats the data values to one decimal place, which is enough precision to be useful for this figure.

Cell Labels and Borders

The hover tooltip is useful when you’re exploring, but for a published figure it helps to have the percentages visible directly on the chart, in the same way the matplotlib version had annotations in each cell.

We can do this using texttemplate.

Also, by adding xgap and ygap we can add pixel-width gaps between cells, which Plotly fills with the background colour. Set the background to white and you get clean white borders.

fig = px.imshow(
    completeness_pivot,
    color_continuous_scale=”RdYlGn”,
    zmin=0,
    zmax=100,
    labels={”color”: “Completeness (%)”},
    title=”Well Log Completeness — Veldwick”,
)

fig.update_traces(
    xgap=2,
    ygap=2,
    texttemplate=”%{z:.0f}%”,
    textfont={”size”: 10},
    hovertemplate=(
        “<b>Well:</b> %{y}<br>”
        “<b>Curve:</b> %{x}<br>”
        “<b>Completeness:</b> %{z:.1f}%<extra></extra>”
    ),
)
fig.update_layout(
    xaxis_title=”Curve”,
    yaxis_title=”Well”,
    font={”family”: “monospace”},
    plot_bgcolor=”white”,
    paper_bgcolor=”white”,
    width=900,
    height=500,
)
fig.show()

%{z:.0f}% in the cell label rounds to a whole number, which can avoid unnecessary decimal places. The monospace font stops the percentage values shifting around as the digit count changes across cells.

Switching Formations with a Dropdown

In a Jupyter notebook, you can wire the figure up to a formation selector using ipywidgets.

The @interact decorator creates a dropdown from the list and re-runs the function every time the selection changes. The figure title updates automatically since the formation name is passed as a variable.

I find this a great way to switch between different formations or groups of data, especially when we are dealing with a large number of groups.

from ipywidgets import interact

@interact(formation=formation_names)
def plot_formation(formation=’Veldwick’):
    pivot = results[formation]
    fig = px.imshow(
        pivot,
        color_continuous_scale=”RdYlGn”,
        zmin=0,
        zmax=100,
        labels={”color”: “Completeness (%)”},
        title=f”Well Log Completeness - {formation}”,
    )
    fig.update_traces(
        xgap=2,
        ygap=2,
        texttemplate=”%{z:.0f}%”,
        textfont={”size”: 10},
        hovertemplate=(
            “<b>Well:</b> %{y}<br>”
            “<b>Curve:</b> %{x}<br>”
            “<b>Completeness:</b> %{z:.1f}%<extra></extra>”
        ),
    )
    fig.update_layout(
        xaxis_title=”Curve”,
        yaxis_title=”Well”,
        font={”family”: “monospace”},
        plot_bgcolor=”white”,
        paper_bgcolor=”white”,
        width=900,
        height=500,
    )
    fig.show()

This is a Jupyter-specific feature and it won’t work in a plain Python script or Streamlit app.

For Streamlit, the equivalent is st.selectbox, which feeds into a similar function. The multi-formation panel in the next section is a better fit if you want all four formations visible at once.

Faceting Across Formations

The matplotlib version handled multiple formations with a subplot grid, where we have one axis per formation, arranged in a column.

Although px.imshow() supports some faceting use cases, in this example I am going to use plotly.graph_objects with make_subplots, because we already have one 2D table per formation.

The results dict from the data section already has one pivot per formation, so no reshaping needed.

We can build the subplot figure using make_subplots, and then adding a Heatmap trace for each formation:

from plotly.subplots import make_subplots
import plotly.graph_objects as go

fig = make_subplots(
    rows=len(formation_names),
    cols=1,
    subplot_titles=list(formation_names),
    shared_xaxes=True,
    vertical_spacing=0.06,
)
for i, formation in enumerate(formation_names, start=1):
    pivot = results[formation]
    z_vals = pivot.values
    fig.add_trace(
        go.Heatmap(
            z=z_vals,
            x=pivot.columns.tolist(),
            y=pivot.index.tolist(),
            colorscale=”RdYlGn”,
            zmin=0,
            zmax=100,
            xgap=2,
            ygap=2,
            showscale=(i == 1),
            colorbar=dict(
                title=”Completeness (%)”,
                len=0.25,
                y=0.88,
            ) if i == 1 else {},
            text=[[f”{v:.0f}%” for v in row] for row in z_vals],
            texttemplate=”%{text}”,
            textfont={”size”: 9, “family”: “monospace”},
            hovertemplate=(
                “<b>Well:</b> %{y}<br>”
                “<b>Curve:</b> %{x}<br>”
                “<b>Formation:</b> “ + formation + “<br>”
                “<b>Completeness:</b> %{z:.1f}%<extra></extra>”
            ),
        ),
        row=i,
        col=1,
    )
fig.update_layout(
    height=290 * len(formation_names),
    title_text=”Well Log Completeness by Formation”,
    title_font_size=16,
    font={”family”: “monospace”},
    plot_bgcolor=”white”,
    paper_bgcolor=”white”,
)
fig.show()

Four-panel stacked heatmap (Thornfield, Veldwick, Oskarby, Dravenport) with white cell borders, percentage labels, shared x-axis, and formation-aware hover tooltips

The shared_xaxes=True argument keeps curve columns aligned across formations so that the same curve appears in the same column regardless of which panel you’re looking at, which makes vertical comparison straightforward.

The showscale logic avoids repeating the colour bar across every subplot.

Embedding in Streamlit

If you’re building a well review tool or a data QC dashboard, the Plotly figure drops into Streamlit with a single call:

import streamlit as st

st.title(”Well Log Completeness Review”)
st.plotly_chart(fig, use_container_width=True)

use_container_width=True makes the figure fill the column width, which looks better than Plotly’s fixed default.

From there you can wire up Streamlit widgets such as formation selectors, well filters, a date picker and the figure will rebuild.

This isn’t a Streamlit tutorial, so I’ll leave the wiring there. If you’re already working in Streamlit, the move from notebook figure to deployed tool is not much work.

What the Switch to Plotly Actually Changes

In a static heatmap, checking a specific cell means tracing the row and column yourself. That works for small plots, but it becomes slower as the number of wells, curves, or groups increases. With Plotly, the hover tooltip brings the well, curve, formation, and completeness value directly to you.

There is still a trade-off. Static matplotlib figures are often better for reports, papers, and slide decks. Plotly is better when the figure is part of an active review, notebook workflow, or dashboard.

Jupyter notebooks can become a chaotic mess after a few hours of work.

All of the focus is initially on getting the code to work, to display some pretty outputs or to visualise machine learning model metrics. Reworking the the code into something that others (or yourself in six months time) can follow is often forgotten.

The code might be correct, and the the analysis might hold up, but anyone picking it up cold is left to reconstruct your reasoning and thinking from scratch.

They start to ask questions like

• What was filtered out and why?

• What does that intermediate result means?

• Why one step leads to the next.

Even when the answers are technically somewhere in the code, they can take far too long to find. The problem is not usually the analysis itself, but rather it is the missing context around it.

Subscribe now

The gaps between cells

A notebook without narrative tends to look like this: imports, data load, some filtering, a plot, more filtering, another plot.

The reader doesn’t know why you did any of these, because you didn’t say. When you wrote it, you didn’t need to as it was obvious to you at the time.

A sentence or two between logical steps is usually enough to close the understanding gap, for example

That markdown cell took thirty seconds to write and it tells the reader what you saw, what you decided, and why the next cell exists. Without it, they would still be guessing and trying to follow what was happening based on the code.

Magic numbers are hidden reasoning

One of the more common sources of confusion in notebooks is the unnamed constant. A threshold, a cutoff, a multiplier chosen for a good reason at the time of writing now becomes invisible.

Where did 15 come from? What’s 120? Is 0.4 a standard cutoff or something specific to this well?

Compare that to:

Same values, same result. The second version takes maybe two extra minutes to write, but anyone reading it can see what the numbers represent and where they came from. If someone needs to change the cutoff, they know exactly where to look.

Burying the numbers in the equations and code makes the notebook harder to audit and harder to hand over.

Cells that do too many things

The overloaded cell is easy to miss when everything is working.

Imports, data loading, column renaming, filtering, and a plot all in one block. It’s fine until something breaks and you have to unpick the whole thing to find it. An example of this can be seen below.

Breaking it into logical units doesn’t mean every line gets its own cell. It means grouping by purpose and giving each group a brief header.

The code itself hasn’t changed, but it has become much easier to navigate, easier to debug when something breaks, and easier for someone else to work out what’s happening at each step.

Grouping cells by purpose also makes a notebook easier to return to months later. I’ve written about the naming and folder conventions that make that work in practice on Substack. Link at the end.

Unlabelled outputs are just as opaque

The same readability problem that affects code also affects outputs. A plot with no title, no axis labels, and no surrounding explanation is the visual equivalent of a magic number. The person who made it knows what it shows, but it can be difficult for anybody else to work out what the chart is trying to show.

For any figures that end up in reports or presentations, this becomes a real problem. An axis label that reads perm is not the same as one that reads Permeability (mD). Similarly, a title that says Formation scatter tells the reader almost nothing.

I wrote about this recently in Six Matplotlib Changes That Improve Your Figures for Reporting and Publication

The fix is the same as for magic numbers: make the context explicit.

• Label the axes

• Add a descriptive title

• If the figure is making a specific point, say so either in a markdown cell above it or as an annotation directly on the chart

A notebook full of readable code, but unlabelled plots is still half finished. The outputs are part of the document too and often times that is what you come back to the notebook for. For example, you may want to grab the figure for your technical paper or for passing onto a client.

A notebook is a document too

There’s a useful distinction between a notebook you’re using to think with and one you’re handing to someone else. The exploratory version: where you were trying things, backing up, dead-ending, isn’t the same thing as what a colleague needs to understand or verify your work.

The gap between the two is smaller than it looks. The things that make a notebook readable (a few markdown observations, named constants, cells grouped by purpose, labelled figures) can add up to less than an hour for a notebook that took days to build. It’s not glamorous work, but it’s the difference between something you’re comfortable handing over and something you need to walk someone through in person.

Share

Making individual notebooks readable is one half of the problem. The other half is what happens when you have dozens of them across different projects and you need to find something from six months ago. Over on Substack I’ve written about a lightweight system for naming, organising, and indexing notebooks so older work stays usable without turning maintenance into a side project.

Check out ot below

It’s no secret that matplotlib’s defaults are functional, but they don’t always do the data justice! I have talked about this multiple times on here and over on Substack.

You’ll get a chart with the right data, the right axes, and roughly the right shape. But functional and clear aren’t the same thing, and the distance between them is mostly a handful of lines of code and personal choices.

This article covers six of them. They are not clever hacks, instead they are small adjustments that can become automatic once you’ve made them a few times.

The example I am using here is synthetic porosity-permeability scatter for four lithology types: Sand A, Sand B, Shaly Sand, and Tight Zone. Whilst this article is geoscience focused, the same changes can be applied to any chart to take them from matplotlib basic to something that you can easily put in a report.

Subscribe now

Setting Up the Basic Matplotlib Plot

The starting point is a plain scatter with no styling decisions at all.

fig, ax = plt.subplots()

for formation, group in df.groupby(”Formation”):
    ax.scatter(group[”Porosity”], group[”Permeability”], label=formation)
ax.set_xlabel(”Porosity (v/v)”)
ax.set_ylabel(”Permeability (mD)”)
ax.set_title(”Porosity vs Permeability by Formation”)
ax.legend()

When we run the above, we get back the following, yet familiar matplotlib chart:

The raw matplotlib default — a working chart, but noisy and harder to read than it needs to b

It works. The data is there. But it’s also cluttered, visually heavy, and hard to focus on. The full box draws the eye away from the data. The colours fight each other. The axes don’t reflect the structure of the data. Five changes fix most of this.

1. Remove the box

The default figure comes with a full box. Four spines, equal weight, none of it called for by the data. Often, these extra elements are referred to as “Chart Junk”.

Removing the top and right spines immediately reduces the visual noise. Softening the remaining two drops their weight without losing the reference they give the reader.

ax.spines[”top”].set_visible(False)
ax.spines[”right”].set_visible(False)
ax.spines[”left”].set_color(”#aaaaaa”)
ax.spines[”bottom”].set_color(”#aaaaaa”)

Depending on the chart and what you are trying to get across to your readers, it can sometimes be beneficial to add gridlines, however, adding them should not detract the eye from the data.

In this data example, we are going to be dealing with a logarithmic Y-axis later on, so it will be helpful to have that horizontal reference line. I often find that the horizontal grid lines lets the eyes read permeability values without having to mentally extend a line across the full chart. I also feel that the the vertical grid mostly adds noise.

We can turn the grid on for the Y-axis as follows.

ax.yaxis.grid(True, color=”#ebebeb”, linewidth=0.8, zorder=0)
ax.xaxis.grid(False)
ax.set_axisbelow(True)

set_axisbelow(True) keeps the grid behind the data points. This may sound obvious, but by default matplotlib will draw the axis on top of the data, which can look messy.

Before and after removing the box and subduing the grid.

After applying the change, the difference is immediate and the data is easier to look at and interpret.

2. Use an intentional colour palette

Matplotlib’s default colour cycle isn’t bad. It wasn’t designed for any particular data structure though, so it assigns similar visual weight to every group regardless of what you’re trying to say.

Picking a palette deliberately forces a useful question: what do these groups mean relative to each other, and does the colour scheme reflect that?

Okabe-Ito is one solid starting point, but the right palette for your project depends on the data type, the audience, and how the chart will be reproduced. I’ve covered four tools on Substack that make that selection process significantly easier. Link at the end.

In this example, we have four different lithologies which have a rough quality order. Sand A and Sand B are the better reservoir rock. Shaly Sand and Tight Zone sit towards the the lower end.

A palette that groups the two sandy formations together and distinguishes the lower-quality ones makes that structure readable without any additional explanation.

The Okabe-Ito palette is a good starting point. It’s colour-blind safe and holds up across screens and print conditions. However, this is not the only palette available, and i would suggest that you explore other options to suit your needs.

PALETTE = {
    “Sand A”:     “#56B4E9”,   # sky blue
    “Sand B”:     “#CC79A7”,   # reddish purple
    “Shaly Sand”: “#E69F00”,   # golden amber
    “Tight Zone”: “#D55E00”,   # vermillion
}

Two blues for the sands, amber and vermillion for the lower-quality formations. This provides a distinct change without being garish, and it can also avoid the red-green pairing that disappears for about 8% of readers.

for formation, group in df.groupby(”Formation”):
    ax.scatter(
        group[”Porosity”], group[”Permeability”],
        label=formation,
        color=PALETTE[formation],
        s=45, alpha=0.80, edgecolors=”none”
    )

Default colour cycle versus the Okabe-Ito palette

Colour is one of the fastest channels the eye processes. A palette that does nothing to support the data structure is a missed opportunity.

This is not to say the above palette is the one everyone should use, but being aware of colour blindness is something that is key to creating accessible data visualisations.

3. Fix the axes

Permeability spans orders of magnitude in most subsurface datasets. The difference between a tight formation and a productive sand isn’t a linear step, and plotting it that way compresses the variation that actually matters into a small portion of the chart.

A log-scale y-axis is often more appropriate. For this kind of data, it’s usually much clearer, and can be done with a one liner:

ax.set_yscale(”log”)

Once the scale is right, the tick labels need attention. Matplotlib’s default log formatter tends to produce scientific notation or inconsistent formats depending on the range. For permeability in the 1 to a few thousand millidarcy range, plain whole numbers are easier to read.

ax.yaxis.set_major_locator(ticker.LogLocator(base=10, numticks=10))
ax.yaxis.set_major_formatter(ticker.FuncFormatter(lambda x, _: f”{int(x):,}”))

The y-axis limits are worth setting explicitly too. Matplotlib’s automatic limits leave outermost data points at the very edge of the plot area. Extending to the next power of 10 above the data maximum gives the chart some room without arbitrary padding.

import math

perm_max = df[”Permeability”].max()
top = 10 ** math.ceil(math.log10(perm_max))
ax.set_ylim(bottom=0.5, top=top)

Linear scale with default limits versus log scale with whole-number ticks and extended y-axis

This one tends to produce the biggest visible shift. On the linear scale, Sand A’s higher-permeability tail gets stretched out and the lower formations get squashed together. On log scale, the spread within each group is actually readable.

4. Add minor gridlines

Now, this may sound counter-intuitive to “removing the clutter” from a chart, However, when working with logarithmic scaled data I find that just having the major gridlines on the plot can cause issues for interpreting the values correctly.

Major gridlines on a log-scale axis sit at 1, 10, 100, 1,000. That’s useful, but it leaves a lot of unguided space in between. A value sitting between 10 and 100 could be anywhere from 11 to 99 without more reference. Minor gridlines fill those gaps at 2, 3, 4, 5, 6, 7, 8, and 9 within each decade.

The trick is keeping them light enough that they don’t compete with the data or the major gridlines. A dotted line in a near-white grey does the job.

ax.yaxis.set_minor_locator(ticker.LogLocator(base=10, subs=np.arange(2, 10) * 0.1, numticks=100))
ax.yaxis.grid(True, which=”minor”, color=”#f0f0f0”, linewidth=0.5, linestyle=”:”)

subs=np.arange(2, 10) * 0.1 places minor ticks at the nine intermediate positions within each decade. The dotted style and near-white colour keep them clearly subordinate to the solid major gridlines.

Log-scale axis without and with minor gridlines. You may need to expand the image to see the changes.

The minor gridlines don’t change what the chart says, they help make it easier to read a value off the y-axis without guessing.

5. Annotate the insight

If there’s something specific the figure is supposed to communicate, it’s worth saying it. Readers can draw their own conclusions from an unlabelled chart, but not always the same one.

Annotations don’t need to be elaborate, but a short label and an arrow pointing at the relevant part of the data usually does it.

ax.annotate(
    “Higher-quality reservoir\n(Sand A cluster)”,
    xy=(0.272, 18),
    xytext=(0.125, 150),
    fontsize=9.5,
    color=PALETTE[”Sand A”],
    arrowprops=dict(arrowstyle=”->”, color=PALETTE[”Sand A”], lw=1.3),
    va=”center”,
)

Matching the annotation colour to the relevant group ties the label to the data visually, without needing a separate callout or legend entry.

It’s easy to skip this one. The chart looks finished before the annotation goes in. But there’s a difference between a chart that shows data and one that says something about it, and this is usually where that gap gets closed.

6. Finish the figure before exporting

Two settings that are easy to overlook: layout and resolution.

constrained_layout=True handles spacing between plot elements automatically. It’s more reliable than tight_layout() and avoids the common situation where an axis label gets clipped or a legend drifts over the plot area.

fig, ax = plt.subplots(figsize=(8, 5.5), constrained_layout=True)

Resolution matters for anything going outside a Jupyter notebook. 150 dpi is fine for quick review. For a Medium article, a report, or a slide, 300 dpi is the sensible default. The file size is larger, but the output holds up at any zoom.

fig.savefig(”output_figure.png”, dpi=300, bbox_inches=”tight”)

These aren’t visual changes in the way the earlier steps are. They’re finishing work. The kind that becomes obvious in hindsight when you export a figure and notice the labels are cut off or the text looks soft.

Putting it together

Here’s the full figure with all changes applied.

The complete code is below, with a helper function to keep the styling consistent across figures.

import math
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib.ticker as ticker

# ── Synthetic dataset 
np.random.seed(42)
n = 80

formations_params = {
    “Sand A”:     {”phi_mean”: 0.26, “phi_std”: 0.025, “perm_mu”: 2.4, “perm_sigma”: 0.35},
    “Sand B”:     {”phi_mean”: 0.20, “phi_std”: 0.020, “perm_mu”: 1.8, “perm_sigma”: 0.30},
    “Shaly Sand”: {”phi_mean”: 0.18, “phi_std”: 0.020, “perm_mu”: 1.2, “perm_sigma”: 0.28},
    “Tight Zone”: {”phi_mean”: 0.10, “phi_std”: 0.015, “perm_mu”: 0.3, “perm_sigma”: 0.20},
}

frames = []
for name, p in formations_params.items():
    phi  = np.clip(np.random.normal(p[”phi_mean”], p[”phi_std”], n), 0.04, 0.35)
    perm = np.random.lognormal(p[”perm_mu”], p[”perm_sigma”], n)
    frames.append(pd.DataFrame({”Formation”: name, “Porosity”: phi, “Permeability”: perm}))
df = pd.concat(frames, ignore_index=True)

# ── Palette 
PALETTE = {
    “Sand A”:     “#56B4E9”,   # sky blue
    “Sand B”:     “#CC79A7”,   # reddish purple
    “Shaly Sand”: “#E69F00”,   # golden amber
    “Tight Zone”: “#D55E00”,   # vermillion
}

def apply_clean_style(ax, yscale=”linear”):
    ax.set_facecolor(”white”)
    ax.spines[”top”].set_visible(False)
    ax.spines[”right”].set_visible(False)
    ax.spines[”left”].set_color(”#aaaaaa”)
    ax.spines[”bottom”].set_color(”#aaaaaa”)
    ax.yaxis.grid(True, color=”#ebebeb”, linewidth=0.8, zorder=0)
    ax.xaxis.grid(False)
    ax.set_axisbelow(True)
    ax.tick_params(axis=”both”, colors=”#555555”, length=3, width=0.8)
    if yscale == “log”:
        ax.set_yscale(”log”)

fig, ax = plt.subplots(figsize=(8, 5.5), constrained_layout=True)
fig.patch.set_facecolor(”white”)

for formation, group in df.groupby(”Formation”):
    ax.scatter(
        group[”Porosity”], group[”Permeability”],
        label=formation,
        color=PALETTE[formation],
        s=50, alpha=0.82, edgecolors=”none”
    )

apply_clean_style(ax, yscale=”log”)

perm_max = df[”Permeability”].max()
top = 10 ** math.ceil(math.log10(perm_max))
ax.set_ylim(bottom=0.5, top=top)
ax.yaxis.set_major_locator(ticker.LogLocator(base=10, numticks=10))
ax.yaxis.set_major_formatter(ticker.FuncFormatter(lambda x, _: f”{int(x):,}”))

ax.yaxis.set_minor_locator(ticker.LogLocator(base=10, subs=np.arange(2, 10) * 0.1, numticks=100))
ax.yaxis.grid(True, which=”minor”, color=”#ebebeb”, linewidth=0.7, linestyle=”:”)

ax.set_xlabel(”Porosity (v/v)”, labelpad=10)
ax.set_ylabel(”Permeability (mD)”, labelpad=10)
ax.set_title(”Porosity vs Permeability by Formation”, pad=14)
ax.set_xlim(0.03, 0.35)

ax.annotate(
    “Reservoir quality trend”,
    xy=(0.265, 50),
    xytext=(0.068, 10),
    fontsize=9.5,
    color=”#555555”,
    rotation=0,
    arrowprops=dict(arrowstyle=”->”, color=”#555555”, lw=1.3),
    va=”center”,
    ha=”left”,
)

ax.legend(title=”Formation”, title_fontsize=9, fontsize=9,
          frameon=True, framealpha=0.95, edgecolor=”#dddddd”,
          loc=”lower right”)

fig.savefig(”porosity_permeability.png”, dpi=300, bbox_inches=”tight”)

None of these changes are complicated. Each one takes a few lines. The combined effect is a figure that’s easier to read, more honest about the data structure, and clearly finished rather than produced by default.

The defaults exist for good reasons. They get you to a working chart quickly without any configuration. But “working” is a low bar, and clearing it is not the same as making something that communicates well.

Share Subsurface Syntax

Choosing Okabe-Ito for this example wasn’t arbitrary. It came down to accessibility, contrast, and whether the colours would hold up across screens and print. For your own projects, those same questions apply, and the answer won’t always be the same palette. Check out the following article to find four tools that help when selecting a colour palette.

→ 4 Essential Tools to Help You Select a Colour Palette for Your Data Visualisation

I use AI tools every day. There is not a day that goes by where I’m not brainstorming ideas with ChatGPT or building prototypes with Claude Code.

If you have read my previous articles on the Jagged Frontier and automation complacency, then you will know that I have argued that if you can’t check the output then you are exposed.

So here’s the part that sounds like a contradiction.

When I’m building a lithology model that is going to be fed into a larger geomodel, I check every prediction against the geology, the cuttings, the wireline log character, and what I know about the basin. When I’m building a personal task manager for myself on a Saturday afternoon, I barely check anything. I accept what works, fix what breaks, and keep moving.

That probably sounds hypocritical, and I’ll explain why it isn’t. The short version: verification isn’t a fixed standard you either meet or fail. It’s a sliding scale, and where you sit on it should depend on what’s at stake and who’s downstream of your output.

Most of the conversation around AI verification misses this. It treats verification as binary. Either you check everything or you’re being reckless. In practice, that framing breaks down the moment you actually use these tools across different kinds of work.

Context changes everything

I have built several personal tools and prototypes with AI, including a time tracker and a personal task manager. Some of these were built the same way: fast iterations, accepting what works and fixing what breaks with minimal pausing to verify the code.

And that’s fine. Because the only person who relies on them is me.

If something obvious goes wrong, I usually notice. If something subtle slips through, the consequence is still limited because I’m the only person using it.

I then fix it, and the cost of failure is a minor inconvenience. There’s no impact to anyone else, and there are no commercial decisions hanging on the output or anyone relying on the output from the results from the app.

Now compare that to client work. When I’m building a machine learning model to predict lithology, estimate permeability, or identify facies, I verify everything. Model output gets checked against the geology, the cuttings descriptions, the wireline log character, and what I know about the basin and depositional environment.

Same person, same tools, completely different approach.

What sits between those two extremes is a spectrum, and where you sit on it should depend on the stakes.

If it’s a personal prototype or you’re trying to take a concept from idea to something visual, the right question is “does it work for me?” or “does this show the idea clearly?” Often that is good enough, and you can later expand it if it is needed for others or start again with more detailed guardrails in place.

If you’re building an internal team tool, others are now using it. It probably needs some resilience built in, and users can flag issues. The level of verification then scales with what the tool is going to be used for and what the impacts are if it gets something wrong.

If you’re building a client-facing deliverable, a facies interpretation, a custom coded app, anything that goes outside your own walls, then full verification is necessary. Edge cases need consideration. Output should be checked against independent data. Someone is going to make a decision based on what you produce, and that decision deserves more than “the code ran.”

Understanding what the stakes are, and noticing when they change, is the part that matters. The tools feel the same regardless of which mode you’re in. The stakes don’t.

Vibe coding has a boundary

OpenAI co-founder Andrej Karpathy’s phrase “vibe coding” captures something real about how these tools feel in practice. You describe what you want, the model generates something close, you nudge it, test it, accept the next suggestion, and keep moving.

There is also a distinction worth noting here, as the term is used loosely. Vibe coding, in the strictest sense means accepting AI generated code without reading it. If you have reviewed and understood what the model produced, then that is something different and it starts to move into AI coding assistant territory.

For personal tools, that can be incredibly effective. You’re not trying to produce perfect software. You’re trying to turn an idea into something usable.

But, vibe coding has a boundary.

The moment the output affects someone else, informs a decision, or becomes part of a client-facing workflow, the mode has to change. You're no longer just exploring. You're engineering, making claims, creating something someone else may trust. That's where verification stops being optional polish and becomes part of the work.

Where it gets dangerous

Here’s the part that concerns me.

When you’re prototyping with Copilot or building something quick with an LLM, there’s a feedback loop. You accept a suggestion. It works. You accept another. That works too. Over time, your threshold for scrutiny starts to drop. Not because you’ve made a conscious decision, but because the pattern of “accept and move on” becomes habitual.

That’s manageable when you’re building something for yourself. It’s not manageable when you’re producing something that someone else will rely on.

The danger isn’t in the prototype phase. It’s in the transition. The moment something moves from “quick tool I built for myself” to “a thing that informs a decision.” If your verification habits don’t shift with that transition, you’re carrying prototype-level scrutiny into production-level stakes.

And this isn’t a problem that only affects people new to a domain. It affects me too.

When I’m working in petrophysics, I have decades of knowledge and experience to fall back on. I can look at a predicted facies log and feel when something is off, even before I’ve checked it formally. But if I were to step outside the areas where I am comfortable, say into another domain or using AI to write code in a framework I haven’t used before then I’m in exactly the same position as anyone else.

The results may look clean and professional, meaning that no alarm bells are raised. This is because I don’t have an independent basis to easily notice when something is wrong.

That’s the real shape of the problem. It’s not “experienced people are safe and inexperienced people aren’t.” It’s that everyone has a set of domains where they can verify meaningfully, and a much larger set of domains where they can’t. The risk shows up wherever the gap between those two sets gets crossed without you noticing.

Generative AI tools can give a confidently wrong answer, and whether that goes unnoticed depends entirely on whether the person reviewing it has independent ground to stand on for that specific task.

What verification actually looks like

When I verify AI output on a client project, I’m not just checking whether the code runs.

I’m asking:

• Does this match the geology?

• Do these predicted facies make sense given what we know about the depositional environment?

• Does the porosity estimate align with the core data?

• Are the cuttings descriptions consistent with what the model is suggesting?

• Does the code look correct and follow standard practice?

That kind of verification requires:

• Domain knowledge

• Independent information, i.e data sources the model hasn’t seen

• Knowing what “wrong but plausible” looks like in your field

This is the part that worries me about broader AI adoption.

Not the people who have decades of experience and can spot when an output is geologically nonsensical. But, the people who don’t have that background, running the same models, getting clean-looking results, and having no independent basis to question them.

Subscribe now

The honest version

I think there’s a version of the AI verification conversation that’s more honest than what we usually hear.

It’s not: “always verify everything.”

It’s: “know what you’re verifying, why it matters, and whether you’re actually equipped to do it.”

I don’t verify my personal tools rigorously because the stakes don’t warrant it. I verify my client work carefully because the stakes do, and because I have the domain expertise to do it in a meaningful way.

The question everyone should be asking isn’t “are you using AI responsibly?” It’s:

Do you know which mode you’re in right now?

Have your verification habits kept pace with the stakes?

And if someone downstream is relying on your output, can you actually check it, or are you just assuming it’s fine because the model seemed confident?

The jagged frontier is personal too

I don’t believe the jagged frontier is just about what AI can and can’t do, I believe it should also be about what you can and can’t verify.

The boundary for that verification can shift depending on the task at hand, the stakes and your own knowledge and expertise.

The people who use AI well aren’t the ones who verify everything obsessively are the ones who skip verification entirely. They’re the ones who calibrate and match their level of scrutiny to the level of consequence.

That’s harder than it sounds. Because the tools can start to feel the same regardless of what you’re building. Copilot doesn’t know whether you’re prototyping a personal tool or shipping production code. The LLM doesn’t flag when the stakes have changed, that part is on you to realise and adapt accordingly.

Jupyter notebooks are a great tool for exploration. They can help you move quickly, test ideas, tweak parameters, and follow a line of thought without too much friction.

That flexibility is part of what makes them so useful, but it’s also one of the reasons they are easy to break.

A notebook can look finished, run perfectly in your current session, and still fail the moment someone else opens it from a clean start. Sometimes that someone else is you, six months later, trying to work out what past-you actually did.

Here are three simple habits I have tried to stick to over the years. They won’t get in the way while you are working, and they can save a lot of frustration later.

Subscribe now

1. Restart and Run All Before You Share

As you start working in a notebook the history of how the cells were run starts to build up.

These cells can end up getting run out of order, variables get redefined further down the page, and objects can stick around in memory. If you were to restart that notebook, you may find that it falls over and the results start to look different. This is a particular problem if someone is looking at your notebook for the first time.

To resolve this issue, and before you share a notebook, it is best to restart the kernel and run the whole thing from top to bottom.

If it breaks, that will tell you something useful. It usually means the notebook depends on hidden state that only existed in your session and not in the notebook itself.

If you run the notebook from a fresh start and it runs cleanly all the way to the end, then it is a good

2. Capture Your Environment at the Top

A notebook is often used to record an analysis, but it usually does a poor job of capturing the environment that made that analysis work. This is something I am often guilty of, but I have started to implement the following habit.

When you run your notebook it may run perfectly on your machine which is setup with one version of pandas or numpy, but when it runs on someone elses machine, it may behave differently. In others, there is a chance it will run just well enough to be misleading, especially if key functionality and behaviour in the libraries has changed.

The habit is to include a cell near the top of every notebook that captures the key environmental information: package versions, file paths, and any parameters that control the analysis. This doesn’t need to be sophisticated, and in fact sometimes keeping it simple is often to your advantage.

import sys
import pandas as pd
import numpy as np
import lasio

# Print environment info
print(f”Python:  {sys.version}”)
print(f”pandas:  {pd.__version__}”)
print(f”numpy:   {np.__version__}”)
print(f”lasio:   {lasio.__version__}”)

# Key parameters, change these here, not buried in the notebook
WELL_NAME = “15/9-F-11A”
FORMATION_TOP = 2100.0   # mMD
FORMATION_BASE = 2350.0  # mMD
GR_CUTOFF = 75.0         # API units

The parameters block is often the most useful part. It gives you a single place to see what the notebook assumes, and it makes rerunning the analysis with different inputs much easier.

It doesn’t need to be elaborate. The point isn’t to build a full configuration system inside the notebook, but rather to avoid the user making assumptions.

One small caveat though, is this only captures the environment at the moment the cell is run. If you reopen the notebook later and run it again after upgrading packages, the printed versions will update as well. That makes this useful for visibility, but not as a historical record.

If you need a more fixed record of the environment used at the time of the analysis, it is better to also save that information outside the notebook, for example in a requirements.txt, an environment file, or project documentation.

3. Treat Raw Data as Read-Only

This is the habit that can save the most trouble later.

It’s very easy in a notebook to load a dataset, start cleaning or filtering it, and then keep modifying that same object across multiple cells. After a while, it becomes hard to tell what has changed, what the original looked like, or whether a step has already been applied once before.

That becomes a problem the moment something goes wrong. Maybe a filter was too aggressive. Maybe a unit conversion was applied when it should not have been. Maybe a column was overwritten halfway through the notebook and nobody noticed. Once the raw data has been mixed up with the working copy, getting back to a clean starting point usually means starting again.

I find this especially painful when preparing data for machine learning modelling, where multiple transforms can build up over time and it becomes much harder to see exactly what has been done to the dataset.

A much safer habit is to treat raw data as read-only. Load it once, leave it alone, and do your work on a copy. Then save outputs to clearly named files in a separate location.

import pandas as pd
from pathlib import Path

# Paths -- keep these at the top
DATA_DIR = Path(”data/raw”)
OUTPUT_DIR = Path(”data/processed”)
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)

# Load raw data, do this once, never modify raw_df
raw_df = pd.read_csv(DATA_DIR / “well_logs_15_9_F11A.csv”)
print(f”Loaded {len(raw_df)} rows from raw data”)
print(raw_df.dtypes)

# Work on a copy
df = raw_df.copy()

# ... your processing steps here ...
# Save outputs with descriptive names
output_path = OUTPUT_DIR / “well_logs_15_9_F11A_processed.csv”
df.to_csv(output_path, index=False)
print(f”Saved processed data to {output_path}”)

Using Path objects also makes the notebook easier to move between directories or machines. Hard-coded absolute paths buried halfway through a notebook are one of the easiest ways to make it fragile.

This habit becomes even more useful when it is paired with a clear folder structure such as data/raw, data/processed, and outputs/figures. That way the flow of data is easy to follow, and someone opening the project for the first time can work out what is going on without having to inspect every cell.

See the following for a full project structure that I have started to implement to keep things organised: The Project Structure That Makes Good Jupyter Notebook Habits Automatic

You also don’t need to force all of this into a single cell.

In practice, it is often clearer to split it into a few small setup cells, for example one for imports, one for paths and configuration, and one for loading the raw data.

The Common Thread

None of these habits require significant effort to implement, but the hard part is doing them consistently. Even I still forget from time to time!

Reproducibility is not really about being tidy for the sake of it. It is about being able to rely on your own analysis when it matters. That might be when a result gets questioned, when you need to update the work months later, or when someone else has to pick it up and make sense of it.

These three habits will not make a notebook perfect, but they will make it much easier to rerun, trust, and hand over.

Boxplots are a great statistical tool for visualising data and are commonly used during the Exploratory Data Analysis (EDA) phase of data science projects. They provide us with a quick statistical summary of the data, help us understand how data is distributed and help identify anomalous data points (outliers).

They also make it easy to compare data distributions across groups, and they are familiar enough that most technical readers can scan them without too much effort.

However, one of their problems is that they can hide most of the data behind statistics.

A simple fix is to keep the boxplot and overlay the raw points. You still get the familiar summary, but you also get to see the observations that summary leaves behind.

This article walks through that combination in Python using seaborn and matplotlib to illustrate how we can see the individual data points that make up the statistical summary of a boxplot.

Subscribe now

What are Boxplots?

A boxplot reduces a distribution to a handful of summary statistics. That is useful, but it also means the shape of the distribution, the clustering, the gaps, and the number of observations behind each group mostly disappear.

Boxplots are a standardised way to display the distribution of data based on five key numbers:

• “minimum”

• 1st Quartile (25th percentile)

• median (2nd Quartile/ 50th Percentile)

• 3rd Quartile (75th percentile)

• “maximum”

The minimum and maximum values are defined as Q1–1.5 * IQR and Q3 + 1.5 * IQR respectively. Any points that fall outside of these limits are referred to as outliers.

Boxplots can be used to:

• Identify outliers or anomalous data points

• To determine if our data is skewed

• To understand the spread/range of the data

To construct a boxplot, we first start with the median value (50th percentile). This represents the middle value within our data.

A box is then formed between the 25th and 75th percentiles (Q1 and Q3 respectively). The range represented by this box is known as the interquartile range (IQR).

From this box extends two lines, which are also known as the whiskers. These extend to Q1–1.5 * IQR and Q3 + 1.5 * IQR or to the last data point if it is less than this value.

Any points that fall beyond the whisker limits are known as outliers.

I have previously written on Medium about creating boxplots, especially for well log data:

Creating Boxplots with the Seaborn Python Library

Why Adding Raw Data Points Can Help

A boxplot is good at showing the middle of the distribution and the overall spread in a compact way. It becomes less helpful when:

• group sizes differ and the chart gives no real sense of how much data supports each box

• values are clustered or split into sub-groups that the IQR spans but doesn’t reveal

• a few high or low values stretch the whiskers in ways that look more important than they really are

Overlaying the raw points helps with all three.

You can see how many values sit behind each group, whether the distribution looks smooth or broken into pockets, and whether extreme values are isolated or part of a broader tail.

In seaborn, the quickest way to do this is with stripplot, which adds a small amount of horizontal jitter so the points don’t sit directly on top of each other. It’s not the same as a true beeswarm plot, which places points more systematically, but it is simple, effective, and often good enough for exploratory work.

Setting Up the Dataset

To make the comparison worthwhile, the synthetic data needs to contain patterns that a plain boxplot doesn’t show very clearly.

We will use three formations:

• Aldwick: a fairly broad, high-porosity group

• Brentford: a bimodal group made from two clusters

• Castleton: a low-porosity group with a tight main cluster and a few higher values

The group sizes are also deliberately unequal.

import seaborn as sns
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches
import pandas as pd
import numpy as np

np.random.seed(42)
aldwick = np.random.normal(0.22, 0.04, 80)
brentford = np.concatenate([
    np.random.normal(0.12, 0.015, 30),
    np.random.normal(0.19, 0.015, 30)
])
castleton = np.concatenate([
    np.random.normal(0.075, 0.012, 45),
    np.random.normal(0.13, 0.01, 5)
])
data = pd.DataFrame({
    ‘Formation’: (
        [’Aldwick’] * len(aldwick) +
        [’Brentford’] * len(brentford) +
        [’Castleton’] * len(castleton)
    ),
    ‘Porosity’: np.concatenate([aldwick, brentford, castleton])
})
data[’Porosity’] = data[’Porosity’].clip(0, 0.35)

This gives us three groups with different internal structure, rather than three variations of the same general shape.

Step 1: Start with a Plain Boxplot

A standard boxplot gives a useful baseline and can be generated simply using:

fig, ax = plt.subplots(figsize=(8, 5))
sns.boxplot(data=data, x=’Formation’, y=’Porosity’, ax=ax)
plt.show()

At first glance, the comparison looks clean and straightforward. Aldwick has the highest median porosity, Castleton the lowest, and Brentford sits in between.

That is useful, but it doesn’t tell you much about what is happening inside each group. You can’t see that Brentford is made from two clusters, nor can you tell how many observations sit behind each box. Additionally, you can’t tell whether the values inside the IQR are evenly spread or tightly packed into parts of the range.

Step 2: Overlay the Raw Data Points

The main change is simply to add a strip plot on top of the boxplot.

One small detail matters here: set fliersize=0 on the boxplot first, otherwise the extreme values will appear twice.

fig, ax = plt.subplots(figsize=(8, 5))

sns.boxplot(
    data=data,
    x=’Formation’,
    y=’Porosity’,
    ax=ax,
    width=0.4,
    color=’lightgray’,
    fliersize=0
)
sns.stripplot(
    data=data,
    x=’Formation’,
    y=’Porosity’,
    hue=’Formation’,
    legend=False,
    ax=ax,
    jitter=True,
    size=4,
    alpha=0.5,
    palette=’Set2’
)
plt.show()

Now the differences are more informative.

Aldwick shows a broad cloud of values at the high-porosity end. Brentford is no longer just a mid-range box. You can see that the values gather into two bands. Castleton shows a tight low-porosity cluster with a small number of higher values above it, and a few outliers.

The summary statistics are still there, but they are no longer doing all the talking.

Share Subsurface Syntax

Step 3: Reduce Visual Clutter

With larger groups, the overlay can quickly become messy. Smaller markers and a little more transparency usually help.

fig, ax = plt.subplots(figsize=(8, 5))

sns.boxplot(
    data=data,
    x=’Formation’,
    y=’Porosity’,
    ax=ax,
    width=0.4,
    color=’lightgray’,
    fliersize=0
)
sns.stripplot(
    data=data,
    x=’Formation’,
    y=’Porosity’,
    hue=’Formation’,
    legend=False,
    ax=ax,
    jitter=True,
    size=3,
    alpha=0.4,
    palette=’Set2’
)
plt.show()

This version is easier to read. Dense areas begin to appear darker, which gives you a rough visual sense of where values are concentrated without needing a separate density plot.

It is still a simple chart, but it carries more of the distribution with it.

Step 4: Add Group Means

A boxplot already gives you the median. In skewed data, the mean can sit in a slightly different place, and that difference can be useful to see.

A short dashed line across each group keeps the mean marker distinct from the existing boxplot elements.

formations = [’Aldwick’, ‘Brentford’, ‘Castleton’]

fig, ax = plt.subplots(figsize=(8, 5))
sns.boxplot(
    data=data,
    x=’Formation’,
    y=’Porosity’,
    ax=ax,
    width=0.4,
    color=’lightgray’,
    fliersize=0
)
sns.stripplot(
    data=data,
    x=’Formation’,
    y=’Porosity’,
    hue=’Formation’,
    legend=False,
    ax=ax,
    jitter=True,
    size=3,
    alpha=0.4,
    palette=’Set2’
)
for i, formation in enumerate(formations):
    mean_val = data.loc[data[’Formation’] == formation, ‘Porosity’].mean()
    ax.hlines(
        mean_val,
        i - 0.25, i + 0.25,
        colors=’black’,
        linewidths=2,
        linestyles=’--’,
        zorder=5
    )
plt.show()

This adds one more layer of context without making the chart feel busy. In Castleton, for example, the small number of higher values nudges the mean above the main cluster. That is easier to appreciate when the points are visible as well.

Step 5: Final Polish

The final version adds a few useful presentation touches:

• consistent colours for each formation

• n= labels beneath each group

• light horizontal gridlines

• cleaner spines

• a clearer title and axis labels

formation_colours = {
    ‘Aldwick’: ‘#2ecc71’,
    ‘Brentford’: ‘#3498db’,
    ‘Castleton’: ‘#e74c3c’
}

fig, ax = plt.subplots(figsize=(9, 6), facecolor=’white’)
sns.boxplot(
    data=data,
    x=’Formation’,
    y=’Porosity’,
    ax=ax,
    width=0.4,
    color=’#ecf0f1’,
    fliersize=0,
    linewidth=1.2
)
sns.stripplot(
    data=data,
    x=’Formation’,
    y=’Porosity’,
    hue=’Formation’,
    legend=False,
    ax=ax,
    jitter=True,
    size=3.5,
    alpha=0.5,
    palette=list(formation_colours.values())
)
for i, formation in enumerate(formations):
    mean_val = data.loc[data[’Formation’] == formation, ‘Porosity’].mean()
    ax.hlines(
        mean_val,
        i - 0.25, i + 0.25,
        colors=’#2c3e50’,
        linewidths=2,
        linestyles=’--’,
        zorder=5
    )
    
    n = len(data.loc[data[’Formation’] == formation])
    ax.text(
        i, -0.015, f’n={n}’,
        ha=’center’, va=’top’,
        fontsize=9, color=’#555555’
    )
ax.set_xlabel(’Formation’, fontsize=12, fontweight=’bold’, labelpad=24)
ax.set_ylabel(’Porosity (v/v)’, fontsize=12, fontweight=’bold’, labelpad=10)
ax.set_title(’Porosity Distribution by Formation’, fontsize=14, fontweight=’bold’, pad=15)
ax.spines[[’top’, ‘right’]].set_visible(False)
ax.grid(axis=’y’, alpha=0.3, linestyle=’--’, linewidth=0.8)
ax.tick_params(axis=’both’, labelsize=10)
ax.set_ylim(0, 0.32)
mean_patch = mpatches.Patch(color=’#2c3e50’, label=’Group mean (dashed)’)
ax.legend(handles=[mean_patch], fontsize=9, framealpha=0.9)
plt.tight_layout()
plt.savefig(’porosity_overlay.png’, dpi=150, bbox_inches=’tight’, facecolor=’white’)
plt.show()

What the Overlay Reveals

The final chart shows things that a plain boxplot only hints at.

First, the group sizes are immediately more obvious. Aldwick has more observations than Castleton, and the chart no longer makes those boxes look equally well supported.

Second, Brentford is revealed as a split distribution rather than a single smooth middle range. A plain boxplot summarises that group neatly, but it doesn’t tell you that the values are gathering into two distinct bands.

Third, Castleton shows how a small number of higher values can affect the upper part of the summary without changing the fact that most observations remain tightly clustered at the low end.

This is the real strength of the combination. It lets you keep the summary while checking what kind of data is actually producing it.

That is particularly useful in exploratory work, where similar-looking boxplots can hide very different underlying patterns.

When This Is Most Useful

This combination works best when:

• you are comparing a small number of groups

• the number of observations per group is not too large

• you want to see whether the summary statistics are masking something important

• sample size, clustering, or outliers could change how you interpret the result

It’s especially helpful as a check before deciding what kind of final figure to publish.

A plain boxplot might still be the right choice for the finished chart. But, the overlay is often the thing that tells you whether the plain boxplot is leaving too much unsaid.

Drawbacks to Keep in Mind

This approach does have limits though.

Once each group contains several hundred observations, the raw points can start to blur into clutter, even with transparency. At that point, a violin plot, KDE, or another density-based chart may communicate the shape more clearly.

Jitter also needs a little care. The horizontal spread of the points is artificial. It is there only to stop overlap, not to encode any additional meaning. If a reader treats that width as data, the chart can be misread.

There is also a judgement call around audience. For exploratory work, the extra detail is often worth it. For presentation slides or reports, it depends whether the raw points genuinely help interpretation or just make the figure busier.

If this kind of chart interests you, ridgeline plots are worth adding to your toolkit. They work particularly well when you have more groups than a box plot can comfortably handle, and you want to compare distribution shapes rather than just summary statistics.

I wrote a full walkthrough on Substack covering how to build them in matplotlib with petrophysical data:

Ridgeline Plots in Matplotlib: An Alternative to Faceted Histograms

In the previous article, I looked at the project structure around a notebook: a consistent folder layout, a central config.py for paths, reusable code in src/, and a Cookiecutter template to generate the whole setup quickly.

That part solves a lot of the early mess. You stop rebuilding folders from scratch, copying imports from older notebooks, and writing file paths that only work on your machine.

But even with that in place, another problem shows up once the notebook count starts to grow.

When you only have a few notebooks, organisation hardly feels like a real issue. You know where things are. The filenames are still familiar. Even a slightly messy folder is manageable because the whole thing still fits in your head.

That stops working once the collection gets bigger.

Now you are dealing with exploratory notebooks, finished notebooks, old branches, repeated project folders, and the occasional file whose name made perfect sense at the time and now looks complete gibberish. At that point, the problem is not neatness. It is being able to find the right notebook quickly, know what state it’s in, and recover enough context to trust it.

This article picks up where the project structure article left off. The folder layout stays the same. What changes is the way the notebooks inside each project are organised.

The system is simple: organise by project, put status in the filename, keep a short README in each project, and maintain a lightweight index of finished notebooks. The goal is not to build a perfect archive. It is to make older work usable without turning notebook organisation into a side hobby.

Subscribe now

The Problem With Most Notebook Folders

Most notebook folders do not really organise work. They just collect it.

Working folders that contain notebooks tend to grow and become messy.

Files get created wherever the current task happens to live, then they get renamed inconsistently, and left behind once the immediate job is done.

The newest work stays visible. Older work gets buried. After enough time, finding something depends less on structure and more on whether you still remember what you happened to call it six months ago.

The other issue is that not all notebooks serve the same purpose. Some are rough exploration, whereas others represent the cleaned record of a finished analysis.

Then you have notebook templates or quick scratch files that were useful to test out an idea, but never looked at again.

There is also the context problem. Code can show what happened, but it often does not explain why a threshold was chosen, which notebook fed the final figure, or what changed between one version and another. Those details tend to live in memory. And when you try to refresh that memory months later it not only takes time, but can also be draining trying to figure out what you thought several months ago when you wrote it.

A useful notebook structure needs to do three things well:

• it should help you find notebooks quickly

• tell you what state they are in

• preserve enough context that older work still makes sense

Start With the Same Project Structure Every Time

This article assumes the project structure from the earlier piece is already in place. If not, you can check it out here:

The Project Structure That Makes Good Jupyter Notebook Habits Automatic

That means each project has a standard layout with: notebooks/, src/, data/raw/, data/processed/, outputs/, and a README.md.

Raw data stays separate from processed data. Reusable code moves into src/ once it starts getting copied between notebooks. Paths are handled centrally instead of being scattered through cells.

That matters because notebook organisation works best when it sits inside a predictable project layout. If every project stores things differently, neat notebook names will not save you.

This article is not a replacement for that earlier structure. It is the next layer of the same system.

Folder Structure: Organise by Project, Not by Topic

Once notebook collections get large, it is tempting to organise by topic. When I look back at old notebook folders I initially had the idea to structure them by their use case: one folder for QC, one for plotting, one for feature engineering and one for interpretation work.

It sounds sensible, however, in practice, it usually becomes awkward, especially as the number of notebooks start to multiply.

Most real notebooks don’t belong cleanly to one topic. A QC notebook might load data, validate curves, produce crossplots, summarise completeness by formation, and export something for later modelling. File it under one topic and you may miss it under another. File it under several and you create duplicates or shortcuts that quietly breed like rabbits.

I have found that project-based organisation is usually much easier to live with. Although it is far from perfect it has started to help with that organisation effort.

When someone asks about the Volve study from last quarter, there should be one project folder to open. When you come back to an older model build, there should be one place where the notebooks, data, outputs, and notes sit together.

So the structure still starts with projects:

projects/
    2025-11_volve-petrophysical-qc/
        notebooks/
        src/
        data/
            raw/
            processed/
        outputs/
            figures/
            exports/
        README.md
    2026-01_ml-lithology-classifier/
        notebooks/
        src/
        data/
            raw/
            processed/
        outputs/
            figures/
            exports/
        README.md
shared/
    templates/
    utils/

The date prefix helps keeps projects sorted naturally in time. The more important point is that every project has the same shape.

Once that is true, the next question is how the notebooks inside each project should be named.

Naming Conventions: Put Status in the Filename

Folder structure helps you find the right project. Filenames should tell you what each notebook is doing inside it.

The convention that I have started to use is:

{order}_{descriptor}_{status}.ipynb

So a notebooks folder might look like this:

notebooks/
    01_data-loading_explore.ipynb
    02_curve-qc_explore.ipynb
    03_curve-qc_record.ipynb
    04_formation-analysis_explore.ipynb
    05_formation-analysis_record.ipynb
    06_completeness-summary_record.ipynb

The order prefix gives the folder a readable sequence. The descriptor says what the notebook is actually about. The status suffix tells you whether it is scratch work or the finished version.

That suffix does most of the heavy lifting.

I tend to use four:

_explore for active working notebooks
_record for clean, rerunnable notebooks that represent the finished analysis
_archive for older notebooks kept for reference
_template for starter notebooks kept in a shared templates area

The important split is really just _explore versus _record.

That distinction stops a notebooks folder from becoming one flat pile where everything looks equally official. It also ties back neatly to the previous article. There, the structure separated notebooks from reusable code, raw data, and outputs. Here, the same idea is applied inside the notebooks layer itself.

The one catch is obvious: you have to update the suffix when a notebook changes status. If everything stays _explore, the system quietly loses its value.

Keep an Archive, but Keep It Tidy

An archive can be useful, however, it can easily and quickly become a messy dumping ground.

Sometimes we have a tendency to think that every little thing we did in a notebook matters and that we will come back to it. In reality, I rarely find myself going back to some of these notebooks as most of the important information and ideas have already been extracted.

If a notebook has been superseded, but you feel it still matters, move it into an archive/ folder or mark it clearly as _archive. That keeps the active notebooks folder readable without throwing away older work.

The aim isn’t to preserve every false start in amber. It is just to stop the live folder becoming a geological record of every passing thought.

The README: One File That Explains the Project

Every project folder should have a README.md.

This was already part of the earlier project structure, but it becomes even more useful once there are several notebooks involved. The README gives you the top-level context without making you open everything to work out what happened.

It does not need to be long. It just needs to answer the questions your future self will otherwise have to reconstruct:

• what the project is

• what data or wells it uses

• what it produced

• which notebooks matter

• what decisions or exclusions are important

A simple version might look like this:

# Volve Petrophysical QC

**Status:** Complete
**Period:** November 2025
**Wells:** 15/9-F-1, 15/9-F-4, 15/9-F-11
**Data source:** Volve open dataset, DISKOS export

## What This Is
GR, RHOB, and NPHI curve QC and completeness analysis across
three Volve producers.

## Key Outputs
- outputs/exports/curve_completeness_by_formation.csv
- outputs/figures/completeness_heatmap.png

## Notebooks
1. 01_data-loading_record
2. 02_curve-qc_record
3. 03_formation-analysis_record

## Key Decisions
- GR values < 0 excluded as noise
- Brent Group subdivisions collapsed to group level
- Intervals above casing shoe excluded

The most useful parts are usually the outputs and the decisions. Those are the first things that fade from memory and the things most likely to matter later.

If you’re required to produce reports of your findings, you could also add executive summaries of each notebook and key findings into the README file as you go along. That way when it comes to generating the final report you have already done half the work.

The Notebook Index: A Lightweight Cross-Project Search Layer

Once you have enough projects, even a good folder structure still depends a bit too much on memory.

You know the notebook exists. You know it dealt with completeness by formation or feature engineering or a specific well set. But which project was it in?

That is where a small notebook index helps.

At the top level of the workspace, keep a CSV like this:

notebooks_index.csv

Each row is one finished _record notebook:

project,notebook,status,purpose,last_run,notes
2025-11_volve-qc,03_formation-analysis_record,record,completeness by formation across three Volve producers,2025-11-28,used in ML feature matrix
2026-01_ml-classifier,02_feature-engineering_record,record,builds feature matrix for lithology classifier,2026-01-14,uses volve QC outputs

I wouldn’t bother with this at very small scale. But once the notebook count grows, it becomes useful surprisingly quickly.

The important thing is that the index only tracks _record notebooks. It is there to help you find finished work, not every temporary rabbit hole you disappeared into on a Tuesday afternoon.

The purpose column matters most. If every row says analysis notebook, the whole thing is pointless. If the descriptions are specific, it becomes genuinely searchable.

Alternatively, you could create this in markdown if you feel comfortable creating tables.

Shared Templates: Make Good Starts Easier

This part ties directly back to the earlier article.

If you already have starter templates, keep using them. A shared template_explore.ipynb and template_record.ipynb make it easier to begin in a consistent way without having to think about setup every time.

shared/
    templates/
        template_explore.ipynb
        template_record.ipynb

The explore template can stay minimal. The record template should be a bit more structured, with room for assumptions, exclusions, outputs, and notes.

This isn’t about forcing every notebook to look pretty. It’s about reducing friction and making notebooks easier to read later.

I have some prepopulated notebook templates in my cookiecutter template on github: andymcdgeo/jupyter_notebook_template

How This Fits With src/, config.py, and the Wider Structure

It’s worth being clear that this article is not arguing for a notebook-only system.

The earlier project structure still does the heavy lifting around code, paths, and outputs. If you’re copying functions between notebooks, they should still move into src/. If paths are fragile, fix them in config.py. If outputs are scattered, standardise where they are written.

If code is shared between multiple projects, then it is time to consider an alternative, such as a personal python library that you develop alongside your projects.

This article is solving a different layer of the same problem by making the notebook collection inside each project more legible.

What This Does Not Solve

This system doesn’t replace version control, and it doesn’t solve collaborative editing. Shared notebook development still has all the usual Git awkwardness.

It also doesn’t stop you creating too many notebooks in the first place. If every small variation becomes a new file, the collection will still grow faster than it should.

And yes, it needs a bit of maintenance. You have to rename notebooks when they move from exploration to record. You have to keep the README current enough to stay useful. You have to add finished notebooks to the index.

But those are small jobs. The alternative is usually much more annoying later.

Subscribe now

Closing Thoughts

The test of a notebook system isn’t whether it looks organised on a good day. It’s whether it still makes sense when you come back to to it months later or under pressure.

That is where most notebook folders start to fail. Not because the work was bad, but because the structure around it was too loose to hold its shape over time.

The project structure from the previous article solves part of that by giving each project a consistent layout. This layer builds on it by making the notebooks inside that structure easier to read, search, and trust.

If a notebook collection is small, you can get away with memory and loose naming. Once it grows, that stops being enough.

A little structure helps things go a long way.

Good organisation becomes more valuable when individual notebooks are also worth organising. The 6 Jupyter Notebook Habits article covers the practices that make notebooks traceable, rerunnable, and readable — which is what makes the system above worth building around them.

When working with matplotlib the default plots that are generated can be somewhat plain and difficult to read. I have discussed this several times in the past. However, with some effort and a few coding tweaks we can convert the default plots into something that is fit to be placed in a publication.

In this article we will be looking at the histogram.

A histogram can actually tell you quite a lot. It can show the overall shape of a distribution, reveal skew, hint at bimodality, help spot outliers, and make it easier to compare groups when used carefully.

The problem is that the default matplotlib version doesn’t make much of that clear. You get a set of blue bars, automatic binning, and very little else to help the reader interpret what they are looking at.

In this article, we will take a default plt.hist() chart and improve it step by step into something you would be more comfortable including in a report or publication. The example uses made-up porosity data that is a common measurement within petrophysics and geology, but the steps and techniques can equally be applied to any dataset.

Subscribe now

The Starting Point

To keep things simple, I will use some made-up porosity data. The values are designed to look realistic enough for demonstration purposes, with a slightly right-skewed distribution similar to what you might see in subsurface data.

import matplotlib.pyplot as plt
import numpy as np

np.random.seed(42)
porosity = np.random.beta(a=2.0, b=9.5, size=1000) * 0.45

fig, ax = plt.subplots()
ax.hist(porosity)
plt.show()

Before any styling, the default histogram looks like this:

What you get is the standard matplotlib histogram: blue bars, automatic binning, no axis labels, and very little context for the reader. It is not incorrect, but it does very little to help interpret the distribution.

Step 1: Label Your Axes

This sounds obvious, but it gets skipped more often than it should, especially in exploratory notebooks. An unlabelled histogram is not much use to anyone who did not make it.

fig, ax = plt.subplots(figsize=(8, 5))
ax.hist(porosity)
ax.set_xlabel(’Porosity (fraction)’, fontsize=12)
ax.set_ylabel(’Count’, fontsize=12)
ax.set_title(’Porosity Distribution’, fontsize=14)
plt.show()

This is a small change, but it immediately makes the chart easier to interpret. The reader can now see what is being plotted, what the bar heights represent, and what the figure is about.

Step 2: Choose Your Bin Count Deliberately

The number of bins has a big effect on how a histogram reads, so it is worth choosing it deliberately rather than leaving it entirely to the default behaviour.

There is no single correct answer.

Fewer bins emphasise the overall shape. More bins reveal more local variation, which might be useful structure or might just be noise. Too many bins and the histogram becomes very difficult to interpret. For many distributions, somewhere between 20 and 50 bins is often a reasonable place to start.

Working out the right number of bins may take some trial and error.

fig, ax = plt.subplots(figsize=(8, 5))
ax.hist(porosity, bins=40, color=’steelblue’, edgecolor=’white’, linewidth=0.5)
ax.set_xlabel(’Porosity (fraction)’, fontsize=12)
ax.set_ylabel(’Count’, fontsize=12)
ax.set_title(’Porosity Distribution’, fontsize=14)
plt.show()

The edgecolor='white' with a thin linewidth separates adjacent bars cleanly, which makes it much easier to read the boundaries between bins, especially where counts are similar.

Step 3: Add a KDE Overlay

A KDE (kernel density estimate) curve over the histogram can make the overall distribution shape easier to follow without getting too distracted by the individual bins. It is especially useful for seeing whether a distribution is broadly unimodal, bimodal, or slightly skewed in one direction.

To overlay a KDE on the histogram properly, the histogram needs to be normalised to density so that both sit on the same vertical scale.

import matplotlib.pyplot as plt
import numpy as np
from scipy.stats import gaussian_kde

fig, ax = plt.subplots(figsize=(8, 5))
ax.hist(
    porosity,
    bins=40,
    color=’steelblue’,
    edgecolor=’white’,
    linewidth=0.5,
    density=True,
    alpha=0.7
)

kde = gaussian_kde(porosity)
x_range = np.linspace(porosity.min(), porosity.max(), 300)
ax.plot(x_range, kde(x_range), color=’#1a3a5c’, linewidth=2)

ax.set_xlabel(’Porosity (fraction)’, fontsize=12)
ax.set_ylabel(’Density’, fontsize=12)
ax.set_title(’Porosity Distribution’, fontsize=14)
plt.show()

Switching to density=True changes the y-axis from count to density, so the axis label needs to change as well. That is a small detail, but an important one.

The KDE is useful as a visual guide, but it is still a smoothed estimate rather than the raw distribution itself. It helps show the overall shape, but it should not be treated as more “true” than the histogram underneath it.

A KDE curve on a single histogram is useful. When you need to compare distributions across several formations at once, stacking them as a ridgeline plot often gives a cleaner read. I’ve written about that approach on Medium. Link at the end.

Step 4: Annotate Key Statistics

The mean and median usually sit invisibly inside a histogram. Adding vertical lines for both can give the reader a couple of useful reference points straight away, especially when the distribution is skewed and the two values do not sit in the same place.

fig, ax = plt.subplots(figsize=(8, 5))
ax.hist(porosity, bins=40, color=’steelblue’, edgecolor=’white’,
        linewidth=0.5, density=True, alpha=0.7)

kde = gaussian_kde(porosity)
x_range = np.linspace(porosity.min(), porosity.max(), 300)
ax.plot(x_range, kde(x_range), color=’#1a3a5c’, linewidth=2)

mean_val = porosity.mean()
median_val = np.median(porosity)

ax.axvline(mean_val, color=’#e63946’, linewidth=1.8, linestyle=’--’,
           label=f’Mean: {mean_val:.3f}’)
ax.axvline(median_val, color=’#6a1b9a’, linewidth=2.0, linestyle=’-.’,
           label=f’Median: {median_val:.3f}’)

ax.legend(fontsize=11)
ax.set_xlabel(’Porosity (fraction)’, fontsize=12)
ax.set_ylabel(’Density’, fontsize=12)
ax.set_title(’Porosity Distribution’, fontsize=14)
plt.show()

For porosity data, knowing the mean and median in the same glance as the distribution shape is practically useful. If the distribution is right-skewed, as porosity data often is, the mean will sit noticeably to the right of the median.

Step 5: Clean Up the Visual Noise

When creating figures for publication it is often worth reducing unnecessary “chart junk”. These are elements of the chart that are not always needed. In this case we can remove the top and right spines to help reduce this clutter and keep the reader’s focus firmly on the data.

fig, ax = plt.subplots(figsize=(8, 5))
ax.hist(porosity, bins=40, color=’steelblue’, edgecolor=’white’,
        linewidth=0.5, density=True, alpha=0.7)

kde = gaussian_kde(porosity)
x_range = np.linspace(porosity.min(), porosity.max(), 300)
ax.plot(x_range, kde(x_range), color=’#1a3a5c’, linewidth=2)

mean_val = porosity.mean()
median_val = np.median(porosity)

ax.axvline(mean_val, color=’#e63946’, linewidth=1.8, linestyle=’--’,
           label=f’Mean: {mean_val:.3f}’)
ax.axvline(median_val, color=’#6a1b9a’, linewidth=2.0, linestyle=’-.’,
           label=f’Median: {median_val:.3f}’)

ax.spines[’top’].set_visible(False)
ax.spines[’right’].set_visible(False)
ax.tick_params(labelsize=11)
ax.legend(fontsize=11)

ax.set_xlabel(’Porosity (fraction)’, fontsize=12)
ax.set_ylabel(’Density’, fontsize=12)
ax.set_title(’Porosity Distribution’, fontsize=14)
plt.show()

This is a small change, but it shifts the visual weight onto the data rather than the frame around it.

Step 6: Add a Sample Count Annotation

One thing a histogram rarely includes is the number of observations behind it. For petrophysical data, where sample counts vary substantially depending on the well and the interval being examined, this matters. A smooth-looking distribution built on 40 samples tells a different story than one built on 4,000.

ax.annotate(f’n = {len(porosity):,}’,
            xy=(0.97, 0.95), xycoords=’axes fraction’,
            ha=’right’, va=’top’, fontsize=11,
            color=’#555555’)

Drop this in before plt.show().It’s a small detail, but it gives the reader a better sense of how much data the histogram represents.

Step 7: Final Polish

A few final changes can help bring everything together: a softer background to reduce contrast slightly, consistent font sizing across the figure, and a more informative title than the generic “Porosity Distribution”.

These are more general styling choices and come down to preference. For example, you may not want to change the background colour, especially if the figure is going into a journal article.

fig, ax = plt.subplots(figsize=(8, 5))
fig.patch.set_facecolor(’#fafafa’)
ax.set_facecolor(’#fafafa’)

ax.hist(porosity, bins=40, color=’steelblue’, edgecolor=’white’,
        linewidth=0.5, density=True, alpha=0.7)

kde = gaussian_kde(porosity)
x_range = np.linspace(porosity.min(), porosity.max(), 300)
ax.plot(x_range, kde(x_range), color=’#1a3a5c’, linewidth=2)

mean_val = porosity.mean()
median_val = np.median(porosity)

ax.axvline(mean_val, color=’#e63946’, linewidth=1.8, linestyle=’--’,
           label=f’Mean: {mean_val:.3f}’)
ax.axvline(median_val, color=’#6a1b9a’, linewidth=2.0, linestyle=’-.’,
           label=f’Median: {median_val:.3f}’)

ax.spines[’top’].set_visible(False)
ax.spines[’right’].set_visible(False)
ax.tick_params(labelsize=11)
ax.legend(fontsize=11, framealpha=0.5)

ax.annotate(f’n = {len(porosity):,}’,
            xy=(0.97, 0.95), xycoords=’axes fraction’,
            ha=’right’, va=’top’, fontsize=11, color=’#555555’)

ax.set_xlabel(’Porosity (fraction)’, fontsize=12)
ax.set_ylabel(’Density’, fontsize=12)
ax.set_title(’Histogram of Simulated Porosity Values’, fontsize=14)

plt.tight_layout()
plt.savefig(’porosity_histogram.png’, dpi=300, bbox_inches=’tight’)
plt.show()

Subscribe now

What Changed and Why It Matters

The final chart and the starting point contain exactly the same data. What changed is how much of that data is actually readable. A reader can now see the overall distribution shape, the central tendency, whether the mean and median diverge, and how many samples sit behind the histogram, all without any extra explanation.

Most of these improvements took only a few lines of code. None of them required anything especially advanced. The histogram was always capable of showing this information. The default settings just did not do much to bring it forward.

A histogram works well when you’re looking at one distribution at a time. When you need to compare porosity across several formations simultaneously, the overlapping bars and separate panels start to get in the way. In a separate article I look at ridgeline plots, which stack KDE curves vertically so every formation shares the same x-axis and differences in shape, skew, and spread are visible in a single glance.

-> Ridgeline Plots in Matplotlib: An Underused Way to Compare Distributions

Jupyter Notebooks stick around longer than they should.

That’s not really a criticism of the format. It’s a consequence of how well notebooks work at the start.

A notebook that started as a forty-cell exploration of some well log data becomes the thing everyone references, then the thing someone tries to run in a pipeline, then the thing nobody can quite reproduce reliably. It doesn’t go wrong suddenly. It drifts over time.

The problem isn’t that notebooks are bad tools. They’re excellent tools for a specific class of work: exploration, communication, teaching, one-off analysis where the narrative matters as much as the code.

The issue is that the same properties that make them good at that work can make them awkward for almost everything else.

Subscribe now

What Notebooks Are Actually Good For

The cell structure, mixed with explanations and inline outputs, is designed to make it easy to tell a story about a dataset or an analysis, to show your working, and to let someone else follow your reasoning.

That’s genuinely useful. If you’re exploring an unfamiliar dataset, a notebook lets you try things, see results immediately, and build up a picture incrementally. If you’re presenting findings, a notebook can serve as a written record of an analysis. If you’re teaching, the step-by-step format provides a great tool for anyone trying to replicate what you did.

All of those use cases have something in common.

They’re all modes where the notebook itself is the deliverable, or where the exploratory, document-like format is an asset rather than a liability. The code and the narrative belong together. You want someone to read both.

The Signals That It’s Time to Move

Most notebooks do not stay in the exploratory phase forever. At some point the analysis produces something useful: a data-cleaning function, a plotting helper, a QC routine. Then you, or someone else, wants to reuse it somewhere else. That is usually where the cracks begin to appear.

Here are some of the clearest signs that it is time to move logic out of the notebook and into modules or scripts.

Copying and pasting. If you find yourself copying a function from one notebook into another, you’ve already identified something that should be a module.

Keeping it in a notebook means every change needs to be replicated wherever you’ve copied it. That’s not a minor inconvenience. It’s how subtle inconsistencies creep in and how debugging becomes genuinely painful: you fix a bug in one place and it persists in three others you’ve forgotten about.

A common example is a data-loading or QC helper that starts life in one notebook cell because you only need it once. Then it gets copied into a second notebook, then a third, with small changes each time. Before long, you no longer have one version of the logic. You have several near-identical versions, each with slightly different assumptions, and none of them feels safe to change.

Execution order. Notebooks run in whatever order you choose. That flexibility is useful while exploring, but it means a notebook’s state at any given moment depends on its history.

Hidden state, variables that exist because of cells you ran earlier, or cells you deleted and re-ran, can produce results that look correct in the moment and break unpredictably later. If you’ve ever handed someone a notebook that works fine on your machine and had to say “make sure you run cell 4 before cell 7,” you’ve felt this problem. It doesn’t get easier as the notebook grows.

Testing. If a function matters enough that you want to verify it still works after a change, it usually belongs in a .py file where a test suite can reach it directly. You can test notebooks to a degree, but logic buried inside notebook cells is much harder to test systematically, review cleanly, and maintain over time.

Deployment. If the notebook is meant to run on a schedule, as part of a pipeline, or triggered by some external process rather than being opened and run manually by a human, you’re already in script territory.

Tools do exist to execute notebooks headlessly, and sometimes that’s the right call for a quick automation. But more often it’s a signal that the notebook format is being asked to do something it wasn’t designed for, and that friction tends to compound.

If the signals here feel familiar, the follow-up question is usually where the extracted logic should actually go. I’ve written a piece on Substack about the folder structure that makes this a straightforward decision on every project. Link at the end.

What Modules and Scripts Give You

Moving logic into modules and scripts doesn’t mean abandoning notebooks. It means separating two things that are better kept apart: the logic, and the narrative.

A module is a .py file that collects functions, classes, or constants. Once something lives there, any notebook in the project can import it. Change the function once, and every notebook that uses it gets the update. You can write tests against it. You can version-control it cleanly. You can read it in any text editor and immediately see the complete definition of a function without running anything first.

A script is a .py file designed to be run from the command line or triggered by a scheduler. It’s the right tool when the work is execution rather than exploration: clean this dataset, generate this set of figures, produce this output file. It has no hidden state, no cells to run in the wrong order, no reliance on a kernel that’s been running since Tuesday.

The combination isn’t complicated. Notebooks do exploration and communication. Modules hold reusable logic. Scripts handle execution. Each one does the job it’s actually suited to.

What you don’t want is a notebook that has quietly become the only place certain critical logic lives, that can only be run by opening it in a browser and clicking Run All, and that nobody is entirely sure produces the same results as it did six months ago. That situation is more common than it sounds.

Making the Move

The goal is not to empty the notebook. It is to leave in place the part notebooks are genuinely good at: exploration, comparison, interpretation, and recording decisions.

A better notebook is not one that does everything. It is one that keeps the thinking visible and moves the machinery elsewhere.

The practical question is when to do this. The answer is not “always from the start”. That tends to make early exploration cumbersome and over-engineer work that might be thrown away after two hours. Not every function you write in a notebook deserves promotion into the wider project. Some code is just scaffolding for thinking, and that is fine.

It is also not “never until everything breaks”.

A workable rule of thumb is simple: if you have used a piece of logic in more than one place, extract it. If it needs to run without a human at the keyboard, make it a script. If it matters enough to test, it belongs in a module.

The goal is not to eliminate notebooks. It is to separate narrative, reusable logic, and execution so each part lives in the right place.

The refactor is usually simpler than it feels. You move a function out of a notebook cell and into a .py file in your project’s src/ directory, then import it wherever you need it. The notebook becomes simpler. The logic becomes portable and testable. The project becomes easier to hand over.

What a Better Structure Looks Like

One reason notebook-heavy projects become messy is that there is often no obvious destination for the logic once it outgrows the notebook. Functions stay where they were first written because there is nowhere else that feels natural. That is where a simple project structure helps.

You do not need anything elaborate. You just need clear separation: notebooks for exploration and explanation, a src/ directory for reusable logic, a data/ area that distinguishes raw inputs from processed outputs, and a consistent place for figures and reports. Once those boundaries exist, the refactor becomes much easier because the question is no longer whether something should move, but where it belongs.

A structure like this will not do the refactoring for you, but it removes a lot of hesitation. The notebook can stay focused on the analysis itself, while the logic that needs to be reused, tested, or run elsewhere has a proper home.

project_name/
├── notebooks/
│   ├── 01_exploration.ipynb       # scratch pad
│   ├── 02_analysis.ipynb          # clean story
│   └── archive/                   # old versions
├── src/
│   ├── __init__.py
│   ├── config.py                  # all paths
│   ├── data_loaders.py            # reusable loaders
│   └── plotting.py                # plot styling and saving
├── data/
│   ├── raw/                       # never modify these
│   └── processed/                 # outputs from notebooks
├── outputs/
│   └── figures/
├── .gitignore
├── requirements.txt
├── CHANGELOG.md
└── README.md

The project structure article in this series covered this setup in more detail: a src/ directory with an __init__.py, a consistent place for reusable helpers, and notebooks that stay lean because the heavy lifting happens in importable code. The structure will not do the refactoring for you, but it removes the question of where things should go when you do.

Subscribe now

The Cost of Waiting

Notebooks don’t become the wrong tool through any single decision. It usually happens through gradual accumulation: one more function added to an already cluttered cell, one more place where the same logic gets copied, one more attempt to turn a piece of exploratory analysis into a repeatable process. The format accommodates all of it until, at some point, it doesn’t.

If you recognise the signals early: copy-paste, execution-order fragility, the need for testing, the need for deployment, the move is usually manageable.

Wait long enough and you’re not refactoring a notebook. You’re untangling a system.

Knowing when to move logic out of a notebook is one thing. Having a project structure that makes it obvious where it should go is another. On Substack I’ve written a guide to the folder layout, config.py pattern, and reusable src/ modules that turn this from a judgement call into a habit -- including a Cookiecutter template that generates the whole setup in a single command.

→ The Project Structure That Makes Good Notebook Habits Automatic

When you need to compare distributions across more than four or five groups, many chart types become harder to read effectively. Faceted histograms force the eye to jump between panels, box plots reduce each group to a compact summary, and violin plots can start to feel crowded as the number of groups increases.

A ridgeline plot solves this by stacking KDE curves vertically, one per group, with each row slightly offset from the one below. Every distribution shares the same x-axis. You can scan across all groups at once and comparisons between adjacent rows are immediate.

In this article, we will build ridgeline plots two ways: first from scratch in Matplotlib, then with joypy for a quicker route.

Subscribe now

Setting Up the Dataset

We will use synthetic well log data representing porosity distributions across seven geological formations.

import matplotlib.pyplot as plt
import matplotlib.cm as cm
import pandas as pd
import numpy as np
from scipy.stats import gaussian_kde

np.random.seed(99)
formations = {
    ‘Vardan’:   np.random.normal(0.32, 0.04, 120),
    ‘Kelvik’:   np.random.normal(0.22, 0.05, 150),
    ‘Stenby’:   np.random.normal(0.18, 0.04, 130),
    ‘Morvik’:   np.random.normal(0.14, 0.03, 100),
    ‘Halvard’:  np.random.beta(2, 8, 90) * 0.5,    # Skewed - mixed facies
    ‘Torven’:   np.random.normal(0.20, 0.06, 140),
    ‘Elgfjord’: np.random.normal(0.16, 0.03, 110),
}
# Clip to realistic porosity range
formations = {k: np.clip(v, 0, 0.5) for k, v in formations.items()}

The Halvard formation uses a beta distribution to simulate a right-skewed dataset. This is a synthetic representation of a shale-dominated interval in which lower porosity values dominate, with a smaller number of higher-porosity interbeds.

Step 1: The Default: Faceted Histograms

First, we will start with separate histograms for each formation. This keeps the distributions isolated and avoids overlap, but allows us to use the same axes.

n = len(formations)
fig, axes = plt.subplots(n, 1, figsize=(8, 10))

for ax, (name, values) in zip(axes, formations.items()):
    ax.hist(values, bins=20)
    ax.set_ylabel(name)
plt.tight_layout()
plt.show()

This works well enough as a first pass. You can see the rough centre and spread of each formation, and any obviously unusual group will still stand out as well as any outliers.

The problem is that comparison becomes hard as soon as you want to judge the distributions against one another. Your eye has to move up and down the full figure to compare one formation with the next. Shape is also harder to read once the histograms are separated into individual panels, especially if you are looking for subtler differences such as skew or multi-modality.

That is the point where a ridgeline plot becomes useful. It keeps the same one-group-per-row structure, but turns the figure into something you can scan much more quickly.

Step 2: Building the Ridgeline From Scratch

The ridgeline version keeps the same basic idea as the faceted histograms: one distribution per formation, stacked vertically. The difference is that instead of showing counts in bins, it uses a smooth density curve for each group and offsets each one slightly above the next.

That does two useful things. First, it makes the overall shape of each distribution easier to read. Second, because every curve shares the same horizontal axis, it becomes much easier to compare neighbouring formations at a glance.

The mechanics are straightforward. For each formation, we estimate a KDE over a fixed x-range, then scale it so the peak height is comparable across groups, and then add a vertical offset before filling the area under the curve.

formations_list = list(formations.keys())
n_groups = len(formations_list)
overlap = 0.8

fig, ax = plt.subplots(figsize=(10, 8))
x_range = np.linspace(0, 0.5, 300)
colours = cm.Blues(np.linspace(0.35, 0.85, n_groups))

for i, (name, colour) in enumerate(zip(formations_list, colours)):
    kde = gaussian_kde(formations[name], bw_method=0.25)
    y = kde(x_range)
    y_norm = y / y.max()
    offset = (n_groups - 1 - i) * overlap
    ax.fill_between(x_range, offset, y_norm + offset,
                    alpha=0.8, color=colour, zorder=i)
    ax.plot(x_range, y_norm + offset,
            color=’white’, linewidth=1, zorder=i)
    ax.text(-0.01, offset + 0.05, name,
            ha=’right’, va=’bottom’,
            fontsize=9, fontweight=’bold’,
            fontfamily=’monospace’, color=’#2c3e50’)
plt.show()

The overlap value controls how tightly the ridges sit together. Push it higher and the curves begin to visually blend into one another. Pull it back and the figure starts to behave more like a set of separated mini-plots. Somewhere in the middle usually works best.

Normalising each KDE is also worth noting. Without that step, formations with different sample sizes or density peaks can dominate the figure visually. Normalisation shifts the emphasis away from absolute density height and onto comparative shape, which is usually the more useful story in this kind of plot.

Ridgeline plots are one of three underused chart types I think every Python data practitioner should know. I’ve covered all three over on Substack, including when to reach for each one. Link at the end.

Step 3: Choosing the Right Bandwidth

The default KDE bandwidth often over-smooths the data. This matters when you want to see real structure, bimodality, clustering, or a skewed tail.

# Default bandwidth - tends to over-smooth
kde_default = gaussian_kde(formations[’Halvard’])

# Narrower bandwidth - preserves more structure
kde_narrow = gaussian_kde(formations[’Halvard’], bw_method=0.2)

For the Halvard formation, the default bandwidth would flatten the right skew. A value of 0.2 to 0.3 preserves it.

Step 4: Consistent X-Axis Range

Every group must share the same x-axis range. If you allow each KDE to choose its own range automatically, the shapes may look comparable even when they are not. One formation might appear broader or more tightly clustered simply because it has been plotted over a different span of values.

That breaks the main advantage of a ridgeline plot, which is side-by-side comparison on a common scale.

In this example, porosity is clipped between 0 and 0.5, so the KDE should also be evaluated over that same fixed interval:

x_range = np.linspace(0, 0.5, 300)  # Fixed range across all formations

Defining the range once and reusing it for every formation keeps the visual comparison honest. The peak positions, the spread, and the tails all sit on the same reference frame.

This is especially important when some groups have narrower distributions than others. If each curve is allowed to autoscale, a tight formation and a broad formation can end up looking more similar than they really are. Fixing the x-axis removes that distortion and lets the differences in shape speak for themselves.

Step 5: Final Polish

Once the structure of the plot is working, the last step is presentation. Ridgeline plots can become visually noisy quite quickly, especially when several curves overlap. A few small design choices make a big difference to readability.

First, the filled areas are given a slightly transparent blue gradient so the stacked layers remain distinct without feeling heavy. The white outline on each curve helps separate one ridge from the next, particularly where neighbouring distributions overlap.

The formation labels are placed directly beside each ridge rather than in a legend. This makes the chart easier to scan because the reader does not have to keep jumping back and forth between the figure and a separate key. Removing the y-axis ticks also helps, since the labels already tell us what each row represents.

Finally, most of the spines are stripped away so the emphasis stays on the distributions themselves. The only axis that really matters here is the shared porosity scale along the bottom.

fig, ax = plt.subplots(figsize=(10, 8), facecolor=’white’)

x_range = np.linspace(0, 0.5, 300)
colours = cm.Blues(np.linspace(0.35, 0.85, n_groups))
overlap = 0.8

for i, (name, colour) in enumerate(zip(formations_list, colours)):
    kde = gaussian_kde(formations[name], bw_method=0.25)
    y = kde(x_range)
    y_norm = y / y.max()
    offset = (n_groups - 1 - i) * overlap
    ax.fill_between(x_range, offset, y_norm + offset,
                    alpha=0.85, color=colour, zorder=i)
    ax.plot(x_range, y_norm + offset,
            color=’white’, linewidth=1.2, zorder=i)
    ax.text(-0.005, offset + 0.06, name,
            ha=’right’, va=’bottom’,
            fontsize=9, fontweight=’bold’,
            fontfamily=’monospace’, color=’#2c3e50’)

ax.set_xlabel(’Porosity (v/v)’, fontsize=12,
              fontweight=’bold’, labelpad=10)
ax.set_title(’Porosity Distribution by Formation’,
             fontsize=14, fontweight=’bold’, pad=15)
ax.set_xlim(0, 0.5)
ax.set_yticks([])
for spine in [’top’, ‘right’, ‘left’]:
    ax.spines[spine].set_visible(False)
ax.spines[’bottom’].set_visible(True)
ax.tick_params(axis=’x’, labelsize=10)
plt.tight_layout()
plt.savefig(’ridgeline_porosity.png’, dpi=150,
            bbox_inches=’tight’, facecolor=’white’)
plt.show()

The result is a chart that still carries a fair amount of information, but feels much cleaner and easier to read than the default version.

What the Ridgeline Reveals

The Halvard formation immediately stands out. Where every other formation has a roughly symmetric bell curve, Halvard has a right-skewed distribution with a long tail towards higher porosity values. In a box plot this would appear as a positively skewed distribution. In the ridgeline plot you can see exactly where that tail sits and how it relates to the main cluster.

Its also easier to see multi-modal distributions for each distribution compared to the earlier multi-histogram plot.

None of this requires reading exact values. The shapes communicate the story directly.

Using joypy

If you want the ridgeline effect without building each layer manually, joypy gives you a much quicker route. It wraps the KDE calculation, stacking, and offsetting into a single function call, which makes it useful for exploratory work or for getting to a first draft quickly.

To use it, reshape the data into a simple long-form dataframe with one column for the group labels and another for the values being plotted:

import joypy

records = []
for name, values in formations.items():
    for v in values:
        records.append({’Formation’: name, ‘Porosity’: v})
df = pd.DataFrame(records)

fig, axes = joypy.joyplot(
    df,
    by=’Formation’,
    column=’Porosity’,
    figsize=(10, 7),
    colormap=cm.Blues,
    alpha=0.8,
    linecolor=’white’,
    linewidth=1.2,
    title=’Porosity Distribution by Formation’,
    x_range=[0, 0.5]
)

That is enough to get a readable ridgeline plot with far less code.

The trade-off is control. A library approach is convenient, but the from-scratch version gives you much more say over bandwidth choice, label placement, colour handling, spacing, and the overall composition of the figure. If you are exploring ideas, joypy is handy. If you are aiming for a chart you would actually publish, the manual version is usually easier to tune properly.

When to Use a Ridgeline Plot

Ridgeline plots work best when:

• You have 4 to 20 groups. Fewer and a simple overlapping KDE or violin plot is cleaner. More and the labels become unreadable.

• Each group has at least 50 observations. KDE is unreliable on smaller samples.

• The x-axis is continuous and shared across all groups.

• You want to show shape, skewness, bimodality, spread rather than just summary statistics.

Full Code

import matplotlib.pyplot as plt
import matplotlib.cm as cm
import numpy as np
from scipy.stats import gaussian_kde

np.random.seed(99)
formations = {
    ‘Vardan’:   np.random.normal(0.32, 0.04, 120),
    ‘Kelvik’:   np.random.normal(0.22, 0.05, 150),
    ‘Stenby’:   np.random.normal(0.18, 0.04, 130),
    ‘Morvik’:   np.random.normal(0.14, 0.03, 100),
    ‘Halvard’:  np.random.beta(2, 8, 90) * 0.5,
    ‘Torven’:   np.random.normal(0.20, 0.06, 140),
    ‘Elgfjord’: np.random.normal(0.16, 0.03, 110),
}
formations = {k: np.clip(v, 0, 0.5) for k, v in formations.items()}
formations_list = list(formations.keys())
n_groups = len(formations_list)
overlap = 0.8

fig, ax = plt.subplots(figsize=(10, 8), facecolor=’white’)
x_range = np.linspace(0, 0.5, 300)
colours = cm.Blues(np.linspace(0.35, 0.85, n_groups))

for i, (name, colour) in enumerate(zip(formations_list, colours)):
    kde = gaussian_kde(formations[name], bw_method=0.25)
    y = kde(x_range)
    y_norm = y / y.max()
    offset = (n_groups - 1 - i) * overlap
    ax.fill_between(x_range, offset, y_norm + offset,
                    alpha=0.85, color=colour, zorder=i)
    ax.plot(x_range, y_norm + offset,
            color=’white’, linewidth=1.2, zorder=i)
    ax.text(-0.005, offset + 0.06, name,
            ha=’right’, va=’bottom’,
            fontsize=9, fontweight=’bold’,
            fontfamily=’monospace’, color=’#2c3e50’)

ax.set_xlabel(’Porosity (v/v)’, fontsize=12, fontweight=’bold’, labelpad=10)
ax.set_title(’Porosity Distribution by Formation’,
             fontsize=14, fontweight=’bold’, pad=15)
ax.set_xlim(0, 0.5)
ax.set_yticks([])
for spine in [’top’, ‘right’, ‘left’]:
    ax.spines[spine].set_visible(False)
ax.spines[’bottom’].set_visible(True)
ax.tick_params(axis=’x’, labelsize=10)
plt.tight_layout()
plt.savefig(’ridgeline_porosity.png’, dpi=150,
            bbox_inches=’tight’, facecolor=’white’)
plt.show()

Building a ridgeline plot from scratch gives you full control over the output, but it also raises an obvious question: what other chart types are worth adding to your toolkit? Ridgeline plots sit alongside bump charts and beeswarm plots as visualisation types that most people never reach for, even when they would tell the story far better than a histogram or a box plot. Over on Substack I’ve put together a piece covering all three, with before-and-after examples using well log data.

→ 3 Python Charts You’re Probably Not Using (But Should Be)

Part one of this series covered six habits: treating the notebook as a decision log rather than a script, writing a header cell before any code runs, separating exploration from the record, naming variables so they carry meaning across stages, the kernel restart test before closing, and the cold open test to check whether the notebook stands on its own without you.

Those habits make notebooks much easier to return to, hand over, or audit months later.

But habits without structure only get you so far. Every project still starts from scratch. You still spend a few minutes creating folders. You still copy an imports cell from an old notebook. You still hardcode a file path and then wonder why it breaks on a different machine or a colleague’s laptop.

This article is about the structure that makes those habits automatic. A folder layout that stays consistent across every project. A config.py that removes fragile relative paths. And a Cookiecutter template that generates everything in one command, including notebooks that open with your header and imports already in place.

The example focuses on geoscience project, but the structure applies to any Python data project.

The Folder Structure

Start every project with the same layout. The exact folder names matter less than the consistency. When every project looks the same, you stop thinking about where things live and start thinking about the work.

project_name/
├── notebooks/
│   ├── 01_exploration.ipynb       # scratch pad
│   ├── 02_analysis.ipynb          # clean story
│   └── archive/                   # old versions
├── src/
│   ├── __init__.py
│   ├── config.py                  # all paths
│   ├── data_loaders.py            # reusable loaders
│   └── plotting.py                # plot styling and saving
├── data/
│   ├── raw/                       # never modify these
│   └── processed/                 # outputs from notebooks
├── outputs/
│   └── figures/
├── .gitignore
├── requirements.txt
├── CHANGELOG.md
└── README.md

Three rules make this structure work.

data/raw/ is read only.

Notebooks read from it, but never write to it. That means you can rerun the entire project from scratch without worrying about overwriting source data.

All outputs stay with the project.

Processed data goes to data/processed/. Figures go to outputs/figures/. Nothing ends up scattered across a desktop or Downloads folder.

src/ is for reusable code.

If you have copied something between notebooks more than once, it belongs here. This is especially useful if you have re-usable code functions that you have developed over time, and don’t want to rewrite from scratch each time. You can place these functions in here. Who knows, eventually these reusable functions may turn into well developed apps in the future!

The Quick Profile

Not every task needs the full structure shown above. Sometimes you just want to answer a quick question, check something in a dataset, or run a one-off calculation.

For those sessions a stripped-down layout is more appropriate:

project_name/
├── notebooks/
│   └── 01_exploration.ipynb
├── data/
│   └── raw/
├── outputs/
└── README.md

No src/, no CHANGELOG, no archive. One notebook and minimal overhead.

You can always promote a quick project to a full one later if the work grows. Retrofitting structure onto a project that has already started and sprawled is much harder.

Both layouts can be generated automatically from the same Cookiecutter template (which we will cover later). When creating a new project using Cookiecutter you simply choose between a full or quick profile.

config.py: One File for All Your Paths

Hardcoded file paths are one of the most common reasons notebooks break. You write something like ../../data/raw/volve.csv, move the notebook to another folder, and suddenly nothing runs.

A simple fix is a central config.py in src/ that defines paths once for the entire project.

# src/config.py
from pathlib import Path

PROJECT_ROOT = Path(__file__).parent.parent
LOCAL_RAW    = PROJECT_ROOT / “data” / “raw”
PROCESSED    = PROJECT_ROOT / “data” / “processed”
FIGURES      = PROJECT_ROOT / “outputs” / “figures”
CENTRAL_DATA = Path.home() / “geoscience_data”

def find_data(filename: str) -> Path:
    local = LOCAL_RAW / filename
    if local.exists():
        return local
    central = CENTRAL_DATA / filename
    if central.exists():
        return central
    raise FileNotFoundError(
        f”{filename} not found in:\\n  {LOCAL_RAW}\\n  {CENTRAL_DATA}”
    )

This solves two problems that appear frequently in geoscience work.

1. Many teams maintain large reference datasets that get reused across projects: the Volve dataset, FORCE well logs, shared LAS libraries. Duplicating these inside every project folder wastes space and often leads to version drift.

2. It makes project handovers easier. If a colleague runs the notebook and the data is missing, find_data() raises a clear error showing exactly where the notebook expected to find it.

In a notebook the usage becomes simple:

import sys
sys.path.append(’../src’)

from config import find_data, PROCESSED, FIGURES
well_data = pd.read_csv(find_data(”volve_well_logs.csv”))
well_data.to_csv(PROCESSED / “well_data_clean.csv”, index=False)

The sys.path.append('../src') line assumes the notebook sits inside the notebooks/ folder. If you reorganise the project structure you may need to adjust this path.

No fragile relative paths scattered through the notebook.

src/plotting.py: Consistent Styles and Automatic Figure Naming

Two small annoyances appear on almost every project: inconsistent plot styling across notebooks, and figure files named figure1.png that become impossible to identify later.

Both issues can be solved with a small helper module called plotting.py. In here you can define the key style for your figures, and modify it later to suit your changing needs. Perhaps you change preference from light mode to dark mode, and need to make all your output figures with a dark background.

# src/plotting.py
import matplotlib.pyplot as plt
import datetime
from config import FIGURES

plt.rcParams.update({
    ‘figure.figsize’: (10, 6),
    ‘axes.spines.top’: False,
    ‘axes.spines.right’: False,
    ‘font.family’: ‘Arial’,
    ‘axes.titlesize’: 14,
    ‘axes.labelsize’: 12,
})

def save_fig(fig, name: str, dpi: int = 150):
    date = datetime.date.today().strftime(”%Y%m%d”)
    filepath = FIGURES / f”{date}_{name}.png”
    fig.savefig(filepath, dpi=dpi, bbox_inches=’tight’)
    print(f”Saved: {filepath}”)

In notebooks, you would call the following:

from plotting import save_fig

fig, ax = plt.subplots()
ax.plot(df[’DEPT’], df[’GR’])
save_fig(fig, ‘gr_curve_well_15_9_F1’)
# -> Saved: outputs/figures/20260226_gr_curve_well_15_9_F1.png

Every figure is automatically dated and saved into the project structure. The figures folder becomes a useful record rather than a mystery archive of unnamed files.

CHANGELOG.md and the Archive Convention

Two small additions that take very little effort but often prove useful later.

CHANGELOG.md records important changes made during the project. It does not replace git history. It simply captures decisions that might otherwise be forgotten.

## 2026-02-26
- Switched GR normalisation from standard scaler to min-max
- Added formation depth filter

## 2026-02-10
- Initial analysis complete
- Found three wells with incomplete NPHI data

Writing a quick entry at the end of a session often saves a lot of archaeology later. And if you are stuck for time, you could always use an LLM to pull the git history for the latest changes and write the changelog entry for you.

The archive convention works the same way. Before restructuring a notebook, move the previous version into notebooks/archive/ with a date prefix. The archive folder stays readable instead of becoming a dumping ground.

The Cookiecutter Template

Once you have the structure setup the way you need it to work, the final step is to automate its creation.

This can be done with Cookiecutter. It is a great tool for quickly starting new projects without having to recreate the structure every time.

Cookiecutter generates a complete project structure from a template with a single command. You configure the template once and reuse it for every new project.

And if you find yourself missing functionality or extra structure in your cookiecutter template, you can always expand it and apply it to any new projects.

Installing Cookiecutter

To install cookiecutter, open a terminal and type:

pip install cookiecutter

Cookiecutter Template structure

Create a template folder somewhere permanent on your machine. Keep it outside your project directories since this is a reusable tool.

geoscience-project-template/
├── cookiecutter.json
├── hooks/
│   └── post_gen_project.py
└── {{cookiecutter.project_slug}}/
    ├── notebooks/
    │   ├── 01_exploration.ipynb
    │   ├── 02_analysis.ipynb
    │   └── archive/
    │       └── .gitkeep
    ├── src/
    │   ├── __init__.py
    │   ├── config.py
    │   ├── data_loaders.py
    │   └── plotting.py
    ├── data/
    │   ├── raw/
    │   │   └── .gitkeep
    │   └── processed/
    │       └── .gitkeep
    ├── outputs/
    │   └── figures/
    │       └── .gitkeep
    ├── .gitignore
    ├── requirements.txt
    ├── CHANGELOG.md
    └── README.md

One thing I have found helpful, is asking Copilot/Claude to create the above structure for you.

cookiecutter.json

The cookiecutter.json file defines the prompts that appear when generating a project from the command line / terminal.

{
  “project_name”: “My Geoscience Project”,
  “project_slug”: “{{ cookiecutter.project_name.lower().replace(’ ‘, ‘_’) }}”,
  “author”: “Andy McDonald”,
  “description”: “A short description of the project”,
  “python_version”: “3.11”,
  “profile”: [”full”, “quick”]
}

The post-generation hook

After Cookiecutter creates the project, this small script runs automatically after. It removes files that are not needed for the selected profile (full vs quick).

import os
import shutil

profile = “{{cookiecutter.profile}}”
FULL_ONLY = [
    “src”,
    “notebooks/archive”,
    “notebooks/02_analysis.ipynb”,
    “data/processed”,
    “CHANGELOG.md”,
    “.gitignore”,
]
def remove(path):
    if os.path.isdir(path):
        shutil.rmtree(path)
    elif os.path.isfile(path):
        os.remove(path)
if profile == “quick”:
    for item in FULL_ONLY:
        remove(item)
    print(”\\nQuick project ready. Start with notebooks/01_exploration.ipynb”)
else:
    print(”\\nFull project ready.”)
    print(”Explore: notebooks/01_exploration.ipynb”)
    print(”Analyse: notebooks/02_analysis.ipynb”)

Starter notebooks

The template also includes starter notebooks with your usual header and imports already in place.

Because Jupyter notebooks are JSON files, Cookiecutter can substitute variables inside them automatically. The project name, description, and author fields appear directly in the first markdown cell.

Instead of starting from a blank notebook every time, you open one that already contains the structure you normally add manually.

The first notebook is the scratch pad or the exploration notebook:

The second is a more in depth notebook that allows you keep and prepare the full story.

The structure here only works if the notebooks inside it are written well. In part one of this series on Substack I cover the six habits that make notebooks readable and rerunnable months later. Link at the end.

Using the Cookiecutter template

Creating a new project becomes a single command from your terminal:

cookiecutter /path/to/geoscience-project-template/

You will be prompted for a few details:

The full project structure appears immediately, ready to open in Jupyter.

Keep the Template Up to Date

Over time you will add utilities to src/, improve your default imports, or refine the structure slightly.

When that happens, update the template as well.

The template becomes a record of your current working practices. Every new project then begins from that baseline instead of from scratch.

Putting It All Together

The goal of everything in this article is to remove the friction that stops good habits from sticking and to also speed up the process of creating your geoscience Jupyter Notebook projects.

A consistent folder structure means you never spend five minutes deciding where things go. A central config.py means you never write a broken relative path again. A shared plotting.py means your figures are consistent without thinking about it. And a Cookiecutter template means all of it is in place before you write a single line of analysis code.

None of these pieces is complicated on its own. The value comes from using them together, consistently, on every project.

If you want to get started without building the template from scratch, the full geoscience project template used in this article is available on GitHub:

→ Jupyter Notebook Project Template on GitHub

This is part two of a two-part series. Part one covers the six habits that make notebooks easier to maintain and understand over time: treating the notebook as a decision log, writing a header cell before any code runs, separating exploration from the record, naming variables to carry meaning across stages, the kernel restart test, and the cold open test.

If the structure described here feels like too much right now, the habits from part one will still get you most of the way there. Once the structure is in place though, those habits become much easier to maintain because the environment nudges you toward them automatically.

→ Part 1: 6 Jupyter Notebook Habits That Make Your Work Easy to Return To

Here is a situation working in Jupyter Notebooks that many people will recognise.

A colleague messages you asking how a particular calculation was carried out in a study that was worked on six months ago. You know the answer is somewhere in your jupyter notebook.

You open it, confident the logic and information is in there.

Forty minutes later, you are still not sure.

The code ran. The outputs exist. But the reasoning behind the key decisions is gone. Which Clay Volume cutoff did you use and why? Was normalisation applied to the gamma ray data before or after the formation tops were picked? There is a commented-out cell that might be relevant, or might be an abandoned experiment. You can’t tell.

Then you start to notice that a cell further down the page has errored as a variable is not present. After much head scratching you find out it was calculated 5 cells further down.

This is not a messy notebook problem. Messy exploration is normal and expected. This is a documentation problem: the analysis produced a result, but it did not produce a record.

In subsurface work, that distinction carries real weight. Notebooks often sit at the base of interpretations that can feed future decisions, reserve estimates, and machine learning model training pipelines. When someone asks why a decision was made, “I think that was the approach” is not an audit trail. It is a gap.

These six habits don’t slow down exploration. They ensure that what comes out of it is traceable, rerunnable, and readable by someone who was not in the room when it was written, including you, six months later.

This article does have a geoscience twist to it, but these habits can be applied to any data science project.

1. Treat the Notebook as a Decision Log, Not a Script

This is the shift that changes everything else.

Most notebooks are written as scripts: a sequence of code cells that produces an output. The problem with a script framing is that it records what happened, but not why. Code tells you that Gamma Ray values below zero were removed. It doesn’t tell you whether that was a deliberate interpretation choice, a quality control step, or a temporary workaround that was never revisited.

A notebook written as a decision log treats every significant step as something that needs to be justified, not just executed.

The practical difference is in how you use markdown cells. Not as labels for the code below them, but as the reasoning that precedes it.

Compare these two approaches to the same step.

Label (not enough):

## GR Cleaning
Removing negative values and normalising.

Decision log (what you actually need):

## GR Curve: Cleaning Decisions
Negative GR values present in 3 wells (n=14 samples total). Spike pattern
and depth distribution consistent with tool noise rather than formation signal.
Excluded prior to any formation-level analysis.

Normalisation: min-max applied rather than z-score. Reason: preserves the
shape of the GR distribution across formations, which matters for the
cross-plot in section 4. Z-score would compress the tails where the
lithological contrast is most visible.

The code that follows executes the decision. The markdown records it.

When you return to this notebook in four months, you don’t need to reverse-engineer the reasoning from the code. It is written down. When a colleague picks it up, they know not just what was done but why, and under what assumptions.

A useful test: if you cannot write a defensible markdown cell for a step, that is a signal the decision itself needs more thought before you commit it to code.

2. Write the Header Before You Write Any Code

At the start of any notebook, you can save yourself time later on by spending two minutes writing a header cell before writing a single line of code.

# Project: Volve Field -- Petrophysical QC
## Notebook: 03_formation_analysis -- Curve Completeness Assessment

**Purpose:**
Assess GR and RHOB curve completeness by formation across the Volve dataset.
Output used to determine viable curve inputs for the lithology classification model.

**Input:**
`data/raw/volve_well_logs.csv`
Source: Volve open dataset, DISKOS export, retrieved 2026-01-15.
Wells in scope: 15/9-F-1, 15/9-F-4, 15/9-F-11 (producers only).

**Assumptions and exclusions:**
Intervals above casing shoe depth excluded (varies by well, see formation tops file).
GR values < 0 treated as noise and removed prior to analysis.
Brent Group subdivisions collapsed to group level due to sparse tops coverage.

**Output:**
`data/processed/curve_completeness_by_formation.csv`
`outputs/figures/completeness_heatmap.png`

**Last updated:** 2026-02-26 | **Status:** Complete

The two sections that matter most in geoscience or data science work are data provenance and assumptions.

Data provenance answers: where did this data come from, when was it obtained, and exactly which subset are you working with. A file path tells you nothing about whether the data came from a live database pull last week or a spreadsheet that was emailed across six months ago. Source and scope belong in the header because they are invisible everywhere else.

Additionally, a file path that resolves on your machine and nowhere else is not provenance. It is a dead end. If the data lives anywhere other than a shared location the next person is guaranteed to have access to, the header needs to explain how to get it:

**Input:**
`data/raw/volve_well_logs.csv`
Source: Volve open dataset, DISKOS export, retrieved 2026-01-15.

Wells in scope: 15/9-F-1, 15/9-F-4, 15/9-F-11 (producers only).

Access: Shared project drive at \\server\projects\volve_qc\data\raw\
If unavailable, contact data management or re-export from DISKOS using
the query parameters documented in `docs/data_export_notes.md`.

The fields that matter when access is not guaranteed: where the file physically lives or can be retrieved from, who to contact if access is unavailable, how it was originally obtained so it can be re-obtained the same way, and any query parameters or export settings used. A re-export with different settings is not the same dataset, and that distinction will not be obvious to the person picking up the notebook six months later.

Assumptions and exclusions are where the interpretive choices live. Collapsing Brent Group subdivisions is a meaningful geological decision. Excluding above-casing intervals is a meaningful QC decision. Both are defensible. Neither shows up in the code without a comment, and comments are the first thing to disappear during a cleanup. The header cell is permanent.

A simple status message can inform you and other notebook users whether this is a complete notebook or if it is still a work in progress.

3. Separate Exploration From the Record

Exploration and documentation are not the same activity, and trying to do both in one notebook produces something that serves neither purpose well.

Exploration is nonlinear by nature. You are testing approaches, hitting dead ends, trying variations you will ultimately discard. That process should be unconstrained. A notebook full of experimental cells, abandoned branches, and intermediate print statements is not a problem. It’s how analysis actually happens.

The problem comes when that exploratory notebook is also treated as the record. It looks like an analysis. It produces outputs. But the path through it is not a story. It is a session log.

The habit is to maintain two notebooks per project.

The working notebook is where exploration lives. It has no cleanliness requirement. Dead ends stay in. Commented-out experiments stay in. Variable names that only made sense at 2am stay in. It is never shared and never needs to be.

The record notebook is written once you understand the shape of the answer. It tells one story: problem, approach, result. It runs cleanly from top to bottom. Every significant decision has a markdown cell. Outputs are saved to predictable locations.

A practical signal for when to make the transition: when you can describe the approach out loud, in sequence, without referring to the notebook. If you can’t do that yet, you are still likely in exploration territory.

The record notebook is not just a cleaned-up version of the working notebook. It should be written from scratch, with the benefit of already knowing what the analysis is and what functions or code snippets matter from your exploration notebook. That distinction matters. Cleaning up an exploratory notebook tends to preserve its structure while hiding its confusion. Writing the record from scratch produces something that actually reads as intended and can also highlight any issues in your initial analysis.

The record notebook matters even more as AI starts to enter the analysis workflow. A model can help generate interpretations, but someone still has to verify them. And that verification depends on having a clear record of what decisions were made and why. I’ve written about why verification becomes the hardest problem in technical AI work over on Substack. Link at the end.

4. Name Variables for the Person Who Comes Back

Variable naming is documentation that can’t be separated from the code. Unlike comments, it can’t be deleted. Unlike markdown cells, it is present on every line that references the object.

The goal isn’t descriptive names for their own sake. It’s names that carry two pieces of information: what the object contains, and what stage of the workflow it represents.

A stage-prefix convention makes both visible at a glance:

raw_logs            # loaded directly from file, unmodified
clean_logs          # nulls removed, depth range trimmed, curves validated
logs_with_vcl       # Vcl appended from a clay volume calculation
train_logs          # subset used for model fitting
test_logs           # held-out subset, not seen during training

For model objects, include the algorithm and the target variable:

model_rf_lithology        # random forest classifier, predicts lithology class
model_lgbm_permeability   # LightGBM regressor, predicts permeability (mD)

The rubric is simple: a name should tell you what it is and where it sits in the workflow. If you can’t construct a name that does both, the object itself may not be clearly defined yet. That ambiguity is worth resolving before writing code that depends on it.

One habit that avoids the creeping df2, df3, df_final problem: name the object at creation, not at use. If you plan to rename it later, you will very unlikely rename it later.

5. The Kernel Restart Is Not Optional

Before closing any record notebook, restart the kernel and run all cells from top to bottom. If it doesn’t complete cleanly, it is not finished.

The most common notebook failure mode is hidden state: a variable that was defined in an earlier session and no longer exists in the code, a file loaded from a path that no longer resolves, an import that worked once because another library happened to already be in memory. In the current session, everything appears correct. On a different machine, or six months later, it fails silently or not so silently at a cell with no obvious explanation.

The restart-and-run check is the only reliable way to surface these problems before they surface for someone else.

The practical objection is compute time. Some steps are expensive and re-running a full model training pass or reloading a large dataset on every review pass isn’t realistic. The pattern that handles this cleanly is to cache intermediate outputs to data/processed/ and load from cache in subsequent steps. The record notebook reads the cached output rather than re-executing the expensive step. The reproducibility standard still holds: the notebook runs top-to-bottom, deterministically, it just does not re-derive everything from raw data in a single pass.

The distinction worth being clear on: a notebook that produces the same outputs given the same inputs is reproducible, regardless of whether it uses cached intermediates. A notebook that only works when run in a specific order, with specific variables already in memory from a previous session, is neither reproducible nor fast. It is a future debugging problem waiting to surface at the worst possible moment.

6. Apply the Cold Open Test Before Closing

The first five habits are about writing the notebook. This one is about checking whether it actually works without you.

Before treating any record notebook as complete, try this: close it, open it fresh, and ask whether a colleague could follow it without asking you a single question. Not a full peer review. Just a five-minute scan. Can they identify the purpose from the header? Can they locate the key decisions in the markdown? Do they know what to do with the outputs?

If the answer to any of those is no, the notebook is not finished regardless of whether it runs cleanly.

In subsurface work this test has particular weight because notebooks routinely outlive the context in which they were written. A petrophysicist moves to another asset, goes on leave, or leaves the organisation. Six months later someone needs to understand what was done and why, and the only record is the notebook. It has to stand on its own.

A practical checklist to run before marking any record notebook complete:

## Notebook Sign-Off Checklist
[ ] Purpose is clear from the header without opening any code cells
[ ] Every significant decision has a markdown explanation, not just code
[ ] Data source and date of extraction are recorded in the header
[ ] Assumptions and exclusions are explicitly stated
[ ] Output file names make their contents obvious without opening them
[ ] Header accurately reflects what the notebook actually does
[ ] Kernel restarted and all cells run top-to-bottom without error

The last point on the header is easy to overlook. A header written at the start of a project often doesn’t reflect what the notebook became by the end of it. A notebook titled “exploratory curve QC” that ended up producing the formation completeness dataset used in model training is misleading to anyone who encounters it later. Update the header when the scope changes. It takes thirty seconds and it is the first thing anyone reads.

The cold open test is the difference between a notebook that is technically complete and one that is actually usable. In a field where analysis is regularly revisited, audited, or handed over, that gap matters more than it might seem.

One Habit at a Time

These habits compound, but they don’t all need to start at once.

If applying all six on your next project feels like too much overhead, start with the header cell. Write it before you write any code. It takes two minutes, it forces you to state the purpose and the assumptions before you have committed to any of them, and it is still there when you come back to the notebook in three months wondering what it was for.

The goal is not notebooks that look rigorous. It’s notebooks that can be explained, re-run, and handed over without a conversation first. A header cell, honest markdown at each decision point, a clean top-to-bottom run, and sixty seconds asking whether a colleague could follow it cold gets you most of the way there without slowing down the exploration that precedes it.

These habits matter even more as AI starts to enter the analysis workflow. When a model generates an interpretation or a processed output, someone still needs to verify it. That verification depends on having a clear record: what data went in, what decisions shaped it, and what the output actually represents. A notebook without that record doesn’t just make your own work harder to return to. It makes AI-assisted work harder to trust. I explored this problem directly in a recent piece on Substack — specifically why verification becomes the binding constraint in technical domains, and what that means for how you build and use AI tools.

→ The Jagged Frontier: Why AI Amazes and Frustrates in Equal Measure