(This is a complete article, but if you are a Medium member you can also read it here)

Most data you come across in publications or on the web is in the human-friendly wide data format; tables in reports, spreadsheets and CSV files are mostly like this. But with plotting tools like Seaborn or Plotly Express and for certain Pandas operations, narrow-format data is more useful or even essential. Different tools need different data formats to work with.

To be able to use these data formats appropriately, we aim to answer the following questions:

• What are narrow and wide data formats?

• How to plot wide data with Matplotlib and Pandas plots

• How to convert wide to narrow data with Pandas melt()

• How to plot narrow data with Seaborn and Plotly Express

Wide Format

A wide data table is the most familiar. It stores different variables in separate columns. For example, here’s a summary of monthly fruit sales:

There is one column for months, each fruit has its own column, and the entry in it represents the number of items sold in that month.

Spreadsheet users will often organise data this way, and if you are dealing with CSV files, the data is often in this format. It’s easy to read: focus on the fruit of interest, run your finger down to the right month, and there is the sales figure. We are all used to this sort of thing.

Narrow Format

Narrow, or long format, is rather different: Instead of one column per fruit, we store the fruit names as values in a single Fruit column. And there is a single Sales column.

This is not as easy to read as the wide format unless you are Seaborn or Plotly Express!

Instead of the fruit types being column names, they are now data values.

Which One Is Better?

Neither. Each is better for different tasks.

Wide format is best for…

We would typically find wide data in:

• spreadsheets

• reports

• simple Pandas plots

We’ll see examples of how to use the different formats with different tools soon. As a sneak preview, below is an example of how easy it is to use wide data with the built-in plotting capabilities of Pandas:

wide.plot(x=”Month”)

Pandas automatically plots each numeric column and labels them with the column name. Very convenient.

Pandas plotting is great for simple, quick plots, but more sophisticated libraries like Seaborn and Plotly have different needs.

Long format is best for…

For more sophisticated tools, long format is often preferred, for example in the Seaborn and Plotly Express plotting libraries.

Indeed, most modern Python visualisation libraries prefer long data, and we will explore these shortly.

Share

Creating the DataFrames

Before we can demonstrate the usefulness of narrow data, let’s create the data we saw above.

Here is the code to create a wide dataframe:

import pandas as pd

wide = pd.DataFrame({
    “Month”:[”Jan”,”Feb”,”Mar”],
    “Apples”:[120,150,170],
    “Oranges”:[90,95,110],
    “Bananas”:[70,65,80]
})

This will give you the wide table we saw above.

This is a quick way of creating data to play around with, but in a practical project you are more likely to be importing data from an external source, you might do something like:

wide = pd.read_csv(”datapath/fruit_sales.csv)”)

Plotting with Matplotlib

The old grandaddy of plotting is comfortable with wide data. Here, plot the individual lines separately.

import matplotlib.pyplot as plt

plt.plot(wide[”Month”], wide[”Apples”], label=”Apples”)
plt.plot(wide[”Month”], wide[”Oranges”], label=”Oranges”)
plt.plot(wide[”Month”], wide[”Bananas”], label=”Bananas”)
plt.legend()
plt.show()

A slightly neater method giving the same result using a loop:

for fruit in [”Apples”,”Oranges”, “Bananas”]:
    plt.plot(wide[”Month”], wide[fruit], label=fruit)
plt.legend()
plt.show()

You have to plot each column individually, and the legend is created by labelling the plots and calling plt.legend().

Plotting with Pandas

Using the built-in plotting capabilities of Pandas is very easy with wide format data. If you simply define the x-axis, then Pandas will plot the remaining columns.

wide.plot(x=”Month”)

Three lines are plotted automatically: one line per numeric column, and a legend is displayed automatically.

Pandas plots are based on Matplotlib, but the coding is simpler.

Convert Wide to Long

It is quite likely you will be provided with wide format data, but converting to narrow format is easily achieved with the Pandas melt() function:

long = wide.melt(
    id_vars=”Month”,
    var_name=”Fruit”,
    value_name=”Sales”
)

melt() is a very useful Pandas function as it transforms a table from wide format (many measurement columns) into long format (one measurement column plus a variable column).

Let’s go through it step by step.

In the wide data, there are four columns.

• Month

• Apples

• Oranges

• Bananas

The Month column identifies each row. Imagine that we are going to plot a series of line plots as we did above; Months is on the x-axis, and the values for each fruit column would be plotted against that.

We want Months to serve the same purpose when we convert it to narrow data, so we want to keep it as it is.

This is what id_vars means.

id_vars=”Month”

However, we want to meltall the others. The three columns are going to disappear as columns, and instead, their names become values in a new column.

We can use melt only specifying the id column like this:

long = wide.melt(id_vars=”Month”)

The result is this:

Two new columns were created automatically: variable and value. These are not very meaningful names, so it’s better to specify those names ourselves.

long = wide.melt(id_vars=”Month”, var_name=”Fruit”, value_name=”Sales”)

This gives a better result:

This table is much easier to understand.

So let’s see how we use the long data in a practical setting.

Plotting with Seaborn

As you may well know, Seaborn is based on Matplotlib but works with long format.

import seaborn as sns

sns.lineplot(
    data=long,
    x=”Month”,
    y=”Sales”,
    hue=”Fruit”
)

Seaborn plots the sales numbers against the month, and by setting the hue to Fruit understands this as a category, plots a separate line for each and assigns a different colour to each plotted line.

Consequently, we get one line per fruit, and the legend is automatically generated.

Plotting with Plotly Express

This is very similar to Seaborn, but instead of the hue parameter we need to set color. (I’ve set the width here simply to fit the page better.)

import plotly.express as px

fig = px.line(    
    long,
    x=”Month”,
    y=”Sales”,
    color=”Fruit”,
    width=800
)

So what’s good about long data

Suppose you add another fruit. With the wide format you would need to add another column so the structure of the data changes. But using long format we just add more rows with a different value for the new fruit.

This makes little difference to most of the code above. For the Matplotlib version, we would need to add a new line in the Matplotlib code (or a new entry in the list for the loop version), and Pandas plots will pick up a new column automatically.

The libraries that require narrow data would need no modification at all, as new rows with any data values are catered for automatically.

When Should You Use Each?

If you are comfortable using Matplotlib or want the simplicity of Pandas plots, then you can use wide data.

However, if you want to use the sophisticated statistical plots tħat Seaborn offers or the interactivity and flexibility of Plotly Express, then narrow data is pretty much essential.

When displaying a table, wide data is preferable for human readability, so it’s a good idea to keep a copy of both forms, narrow for plotting and wide for displaying tables.

This has been a very quick and simple look at wide versus long data; there are more sophisticated uses of melt(), Seaborn and Plotly which we might look at another day. Thanks for reading; I hope you have found it useful.

Share

Curated book titles from Bookshop.org (sponsored links)

Some of the most useful books I’ve read and can recommend.

US

UK

Support your local bookshop, not billionaires

MCP and Function Calling are both excellent methods of giving an AI Agent access to local or global resources - essential for most useful applications. We need AI to be able to look at our documents, our calendars, our emails; to trawl the web for information, be it stock movements, travel information or news.

Without tools, the LLM would be stuck in the world when it was trained, useful, but not useful enough for real-world applications.

If you’ve been programming AI agents for a while, then you might well turn to function calling to implement a link to an external source.

But isn’t MCP supposed to be the new silver bullet? It seems like the whole world and its dog have implemented an MCP library of tools that lets your agent access their API. So should we follow suit?

To answer that question, we need to ask a few more questions. But first, let’s take a quick look at the two technologies…

Read more for free...

Subscribe now

I analysed 78 years of London weather data and produced a comprehensive report with charts, without writing a single line of code. I did this with Claude Code for free.

So, today, there are links to two articles that you can read on Medium for free that cover the two topics.

Subscribe now

Data Analysis with Claude Code — it’s not just for programmers

Claude Code has a good reputation as a programmer’s tool. It can also do a fine job of data analysis.

I used it to analyse decades of London weather records, got a comprehensive report with detailed charts (like those above) and found real evidence of climate change in London. Claude wrote and executed the code needed to load the data and create the charts, but it all ran in the background — I didn’t have to look at the code at all.

All I had to do was to describe my data, explain what I wanted to know, and Claude Code did the rest — the analysis, the statistics, the charts, the written report. All of it. The skill it needed from me wasn’t programming in any conventional sense — everything I asked was in English.

Read for free...

How to Use Claude Code for Free with Ollama

Claude Code is a mega-popular vibe-coding environment, and Ollama is a great tool for making LLMs easy to use locally, for free.

Combining the power of Claude Code with the freedom of Ollama has got to result in something good.

To run Claude Code conventionally, you need to subscribe to the tune of $20 a month (less if you pay a year up front). Frankly, I don’t think that’s anything to complain about, given the productivity gains you could achieve.

But free is better!

(Since I wrote this article, I have had problems running Claude code - it says a subscription is needed. This is not true, but it may be that Ollama restricts access for certain models at certain times, and it is translated as an error. Using a different model might work better - I have not yet had time to get to the bottom of the problem.)

Read for free...

See my curated list of Data Visualization books on Bookshop.org US UK

Matplotlib has been a mainstay of data visualisation for decades — it is a powerful package. But because it’s been with us for a long time and it is rooted in visualisation for print media, its default style can produce charts that look like relics from 20 years ago.

More modern styles use less ‘ink’ and are not afraid of white space. They also tend to use fewer primary colours and are generally more subtle.

There are stylesheets that will give Matplotlib a more modern look, and they are easy to use, but we are going to look at how we can, step by step, with a little thought and a few lines of code, very simply bring a more modern look to our classically styled charts.

We will also show how we can translate that code into an easily used custom stylesheet.

Continue to read for free...

This is an occasional newsletter that links to articles about Data Vizualisation and AI. My articles are usually published on Medium, and the links here will be to a free version.

You can follow me on Medium here

So, if you are a paid subscriber, please cancel your payment and subscribe for free instead.

If you are a free subscriber, then please don’t cancel your subscription - I hope to get back to posting in the future.

You might have seen AI-driven data visualization applications before, but other tools that I have used have a kind of DIY feel.

By contrast, Plotly Studio is a complete desktop app that takes you smoothly and professionally from raw data to a visualization application.

I have written about how to use AI for data analysis and visualisation using Google Colab’s Data Science Agent and also using Gemini CLI. Plotly Studio has similar aims, but it is quite a different beast. Whereas the other approaches are extensions of a general-purpose tool, Plotly Studio is a desktop app dedicated to transforming data into data visualisation implemented in Dash.

Plotly Studio is a recent product from Plotly that is being heavily promoted, but unlike the Plotly chart library, it is not free. This is unsurprising as it incorporates a significant AI element that does not rely on the user’s own subscriptions. However, there is a range of reasonable pricing options, including a free one that lets you get a feel of the product…

Continue reading the full article for free here.

Subscribe now

Matplotlib is powerful but can be daunting as it can get verbose when you move beyond basic charts. If you don’t need that level of granular control, there’s a faster path to great visuals: the built-in plotting tools in Pandas. They deliver clean, effective charts with a fraction of the code, letting you focus on insights instead of syntax.

As you might expect, the Pandas plotting library is designed to plot data provided as Pandas dataframes; indeed, the plotting methods are methods from the dataframe class. In the examples below, we will use a dataframe named weather that records the weather in London in the year 2024. To draw a simple line chart of the maximum monthly temperatures, you would write the code below:

weather.plot(x=’Month’, y=’Tmax’)

This is about as simple as it gets. Unless we specify otherwise, the default plot is a line chart. In the code, the plot method from weather is given values for the y and x parameters and will draw the appropriate line chart.

The rest of this article covers the basic charts available from Pandas; future articles will look at statistical charts and customisation.

Carry on reading for free here…

Share

Matplotlib and Plotly are both data visualization powerhouses, but they were created for different purposes. Which you use depends on the purpose of your Python application.

Matplotlib was created back in the early 2000s and is a very flexible, low-level library for creating high-quality, print-ready, static plots.

Plotly, on the other hand, is newer and was built to be interactive and for the web.

Plotly uses its own plotly.js JavaScript library to render plots — a Python application produces a JSON specification, which is passed to the JavaScript library to be rendered. Whereas Matplotlib uses different backends depending on the output format required.

So, while there is overlap between the two libraries (you can display Matplotlib figures in a webpage, and you can download Plotly figures as static printable charts), they are different beasts and built for different purposes.

To help you choose the right solution, we’ll look at three aspects of the libraries: interactivity, style and ease of use.

Continue reading the full article for free here

Of course, you’ve used Plotly Express to create quick and beautiful charts, and you know how easy it makes data visualisation in Python. But if your needs grow beyond the standard offerings, you might find yourself tangled up in complex Plotly Graph Objects (GO) code.

That’s where px_xtras comes in.

What is px_xtras?

px_xtras is a Python library that mimics the simplicity of Plotly Express to hide the complexity of Plotly GO. It offers a clean, high-level interface to create new visualisations with just a few lines of code.

And, of course, it plays nicely with popular frameworks like Streamlit or Taipy and anywhere else you can use Plotly charts.

The current range of charts includes:

• Waterfall charts

• Waffle charts

• Dumbbell charts

• Lollipop charts

• Slope charts

• Metric — a text-based visualisation of changing data

Subscribe now

Here is an example of the code for a Waterfall Chart:

# Waterfall Chart
wf_data = pd.DataFrame({
“Category”: [”Revenue”, “COGS”, “Operating Expenses”, “Taxes”, “Net Profit”],
“Value”: [100000, -50000, -20000, -10000, 0],
})
fig_wf = pxx.waterfall(df=wf_data,
categories_col=’Category’,
values_col=’Value’,
title=”Company Profit Waterfall”,
)
st.plotly_chart(fig_wf)

And here is the result:

You can read the full article here:

px_xtras: Beyond Plotly Express, More Beautiful Charts

(Medium - no paywall)

And here are some more example charts:

Share

You have a new data set, a CSV file, so you spin up a Jupyter Notebook, import pandas, numpy, matplotlib, whatever… create a dataframe and start wrangling the data, building pretty much the same old charts that you made last time.

It’s fine, it works, you’ve done it all before. It’s not difficult, but isn’t it just a bit… tedious?

How about a different approach? How about, like the guy in the pointy hat above, you command the visualisations to appear, and they obediently do so?

I’ve been experimenting with the Gemini CLI for data analysis and the automatic production of interactive Streamlit dashboards from that data.

The following non-paywalled articles show my approach…

Vibe Analysis: Conversational Data Exploration with Streamlit and LLMs

One Command to Turn Any CSV into a Streamlit App with Gemini CLI

Subscribe now

This was a remarkably quick and easy way to create data visualisations directly from the data. You can read the full story here (No paywall):

I created a documented Streamlit Dashboard in less than 5 Minutes with Gemini CLI

Share

Apart from my own work, I’ve been reading some interesting articles in Data Science and AI. Here are a couple that are definitely worth your time.

Agents aren’t always the solution

While reviewing my own recent offering, Build Multi-Agent Apps with OpenAI’s Agent SDK, I came across a very interesting take on the use of AI agents, or indeed, when not to use them.

A Developer’s Guide to Building Scalable AI: Workflows vs Agents shows how a workflow model that uses direct calls to an LLM is often more appropriate than an agentic approach. It does a good job of explaining the pros and cons of the agentic apps and, is a warning that, even if they are fun and sexy, you shouldn’t just grab at the next shiny thing but think a bit more deeply about the nature of your project before diving in. If you are thinking of getting involved in AI app development, this is well worth a read.

A shorter and equally authoritative piece from Anthropic, Building effective agents, also sets out the difference between workflow-based apps and those using agents.

Data Visualization with ChatGPT

is excellent in finding interesting data and presenting it in novel ways. His recent Substack, Is GPT-4o Now Any Better With Data Visualization Report Creation? explores how to use ChatGPT to create PDF reports which include automatically generated charts. His starting point is the Harvard Comparative Death Penalty Database and, as he says, “Bringing death penalty data to life shows us where the world stands on one of the most controversial practices in the legal system.” And he does this by walking us through how to use GPT-4o to analyse the data, produce data visualization and produce a complete report in PDF format.

All links are to free articles unless otherwise indicated with a $ after the title.

Subscribe now

Support your local bookshop, not billionaires

I’ve created a set of recommended books, mainly about data visualization. They are available on my shopfront on bookshop.org. Books can be delivered from a real shop of your choice. I have a shopfront in the US and the UK. Purchases generate a small commission for me.

Some agent framework devs seem to believe that complexity is a virtue. They revel in building complex layers of abstraction on top of fundamentally simple ideas. However, I tend to go along with Einstein’s maxim, “Everything should be made as simple as possible, but not simpler”.

OpenAI takes a refreshingly different approach to other framework developers : they don’t try to be clever, they try to be clear.

I walk through how the SDK makes building multi-agent apps refreshingly simple. If you’ve been struggling with bloated frameworks, this might be just what you need.

The post is freely available via this link:

Build Multi-Agent Apps with OpenAI’s Agent SDK

Share

I’ve been reading Cole Nussbaumer Knaflic’s book Storytelling with Data (see Notes) - it’s quite illuminating. However, the author does not attempt to provide any implementation details, so I’m trying to remedy that.

When working with time series data, we often blend the certainty of historical records with the uncertainty of future projections. This is particularly true when dealing with financial data, where you tell a story of how current events might influence future trends. But if current and future data are rendered similarly in a chart, your audience might miss the difference or misinterpret your insights.

In this article, we’ll use Python and Plotly to visualize a time series where the past and future are clearly distinguished using thoughtful design choices. Whether you’re presenting forecasts, planning reports, or just want cleaner charts, this guide will help you level up your data storytelling.

Why Visual Distinction Matters

When visualizing time series data, it’s easy to fall into the trap of treating all data equally. But there’s clearly a huge difference between what has already happened and what we expect to happen. If we don’t make that distinction visually clear, we might risk confusing the audience.

Say we’re presenting a forecast of sales growth. If our projected values are rendered with the same styling as historical data, are we implying that the future is just as certain as the past?

Visual design is our opportunity to signal uncertainty. A dashed line can indicate something is estimated. A shaded background might subtly cue that we’re entering a new phase. An annotation helps clarify meaning right on the chart, reducing the need for footnotes or explanations in your text.

Good design doesn’t just make better-looking charts — it makes them more honest and more effective.

Share

Setting Up Your Data

We’ll start with some fictional data and build a chart step by step. We will use Plotly Graphic Objects, as this library is straightforward to use and gives us more flexibility than Plotly Express.

I’ll present the code as cells from a Jupyter Notebook (the code will be available in my GitHub repo shortly after publication — see the link at the end of the article). If you prefer a plain old Python program, then just concatenate the code cells and it should work fine.

Before we define the data, we should import the libraries.

import plotly.graph_objects as go
import pandas as pd

import plotly.io as pio
pio.templates.default = "plotly_white"

Now, I must admit to having a thing about the default Plotly theme: I don’t much like it, so I almost always set the default theme to ‘plotly_white’, which I find to have a much cleaner look. Also, for this exercise, we want the plain white background that this theme gives us. The last two lines of code ensure that this theme is used throughout.

The data is just a set of integers and dates. We can imagine that they are sales figures of some sort, although this is not important. The important thing is that they are split into historical figures and projected ones. The data covers 15 years from 2015 to 2029, and I’m assuming that you are reading this in 2025, so the first 9 years are historical data and the final 6 years are predicted values.

# Sample data
data = {
    "Date": pd.date_range(start="2015-01-01", periods=15, freq="YE"),
    "Value": [10, 15, 20, 25, 30, 20, 35, 40, 38, 45, 50, 55, 60, 62, 65],
}

df = pd.DataFrame(data)

# Split into historical and projected data
historical = df.iloc[:10]  # First 10 years
projection = df.iloc[9:]   # Last 6 years

The sample data is created as a range of 15 datetimes with a frequency of 1 year and a list of 15 integer values. We create a data frame from this and then split the original dataframe into the first 10 and last 5 years. The result is two dataframes historical and projection which contain the historical data and the predicted data, respectively.

3. Plotting with Plotly Graphic Objects

To create a chart with Plotly Graphic Objects (GO), we first create a GO object and then add traces (i.e. plots).

To plot a line chart, we use the method go.scatter() to produce a trace and add that to the figure. That’s right, a scatter plot. But we set the mode parameter to ‘lines’, which tells the method to use lines instead of dots as the marker for plotting — a scatter plot that uses lines to plot each point is, of course, a line plot.

The code to plot the historical data is shown below where you can see that we set the x and y parameters to the Date and Value fields of the dataframe and specified the line colour to be blue.

# Create the figure
fig = go.Figure()

# Add historical data
fig.add_trace(go.Scatter(
    x=historical["Date"], 
    y=historical["Value"], 
    mode="lines", 
    name="Historical",
    line=dict(color="blue")
))

fig.show()

The resulting plot is shown below.

That is the first 10 years of data.

Now let’s add in the projected data.

# Add projected data with a dashed line
fig.add_trace(go.Scatter(
    x=projection["Date"], y=projection["Value"], mode="lines", name="Projection",
    line=dict(color="blue", dash="dash")
))

fig.show()

This is pretty similar to the previous code, but it uses the second dataframe and sets the line style to be dashed. Adding this trace creates a new plot that builds on the old one.

Plotly GO has automatically added a legend so that we can see what the different traces mean. However, even without that textual clue, it is very clear from the visual look of the plots that the data are different.

Enhancing with Graph Objects

It is hard to envisage a graph that does not need some sort of textual description to set the context. So, let’s add some basic text: a title and labels for the axes.

# Update layout
fig.update_layout(
    title="Historical Sales Data and Projected Sales",
    xaxis_title="Year",
    yaxis_title="Units Sold",
    xaxis=dict(showgrid=False),
    yaxis=dict(showgrid=True),
    showlegend=False
)

# Show plot
fig.show()

The update_layout method does exactly what it says, it allows us to change aspects of the plot. Here we have added the text fields as mentioned and set the grid to horizontal lines only. We also removed the legend.

Here’s the new version.

That is looking good, already, but we are going to further distinguish between the two types of data.

The following code cell adds a shaded area to the projected data.

# Add a shaded background for the projection period
fig.add_shape(
    type="rect",
    x0=projection["Date"].iloc[0], x1=projection["Date"].iloc[-1],
    y0=0, y1=1,  # Covers full y-axis
    xref="x", yref="paper",
    fillcolor="lightblue", opacity=0.3, layer="below"
)
fig.show()

Next, we add a textual annotation to describe that area.

# Add annotation for projection period
fig.add_annotation(
    x=projection["Date"].iloc[2],  # Middle of the projection period
    y=max(df["Value"]) * 1.1,  # Slightly above max value
    text="Projected data",
    showarrow=False,
)
fig.show()

The chart now looks like this.

The projections are now easily distinguished as they are contained in that shaded box.

For consistency, I have written code to similarly contain the historical data. Frankly, I’m not sure that this is absolutely necessary, but here it is anyway.

# Add a shaded background for the historical period
fig.add_shape(
    type="rect",
    x0=historical["Date"].iloc[0], x1=historical["Date"].iloc[-1],
    y0=0, y1=1,  # Covers full y-axis
    xref="x", yref="paper",
    fillcolor="lightgrey", opacity=0.3, 
    layer="below"
)

# Add annotation for historical period
fig.add_annotation(
    x=historical["Date"].iloc[2],  # Middle of the projection period
    y=max(df["Value"]) * 1.1,  # Slightly above max value
    text="Historical Data",
    showarrow=False,
)

fig.show()

This treats the historical data in the same way as the projections but the shading is a different colour.

The resulting chart clearly shows the purpose of each section of the data and so the chart does not require a legend.

Annotating specific events

If you had been hiding under a rock for the last few years, or have a very short memory, then you might be confused by the slump in sales in the middle of the historical period. To allay the fears of sales executives who are not paying attention, we can add text to the graphs to explain such anomalies.

Plotly GO allows us to add text, as we saw above, but we can also add nice little arrows to highlight a point in the graph.

# Add annotation for COVID in 2020
fig.add_annotation(
    x=pd.Timestamp("2020-12-31"),  # Year 2020
    y=25,  # Position above the value for 2020
    text="COVID Slump",
    showarrow=True,
    arrowhead=2,
    arrowsize=1,
    arrowcolor="red",
    ax=0, ay=-60,  # Arrow pointing down
    font=dict(size=12, color="red")
)
# Show plot
fig.show()

The code above positions a text annotation and defines an arrow that points to a particular part in the graph.

That is my final version of the graph.

Final Thoughts

We’ve demonstrated how Plotly Graphic Objects can be used to create clear and easy-to-interpret graphs.

Different line styles help distinguish between various data types, and grouping them within shaded boxes further enhances clarity.

Textual annotations and labels are crucial for providing context and can also be used to draw attention to specific areas of interest.

Data visualization is fundamentally about communication, and I hope the examples shown highlight how well-designed, visually appealing charts can enhance that communication.

While the data used here is simple and fictional, the techniques are applicable to your own real-world datasets.

Share Hands on Data Visualization, Data Science, AI and Python

As ever, thanks for reading and I hope that this has been useful. To see more of my articles, follow me on Medium or subscribe to my Substack.

The code and data for this article can be found in my GitHub repo. I will place a link to it here shortly after publication.

Notes

1. I have been very much influenced by Cole Nussbaumer Knaflic’s book Storytelling with Data: A Data Visualization Guide for Business Professionals. It’s well worth a read if data visualisation is your thing. It’s available here (affiliate link).

2. I messed up the presentation of the date of the Covid pandemic, but I’d already created all the code and images and practically finished the article before I noticed. And I’m too darned lazy to do it over. Sorry!

Currently, although paid subscriptions are open, you won’t get anything extra - this may change in the future. But if you feel inclined to support my work regularly, you could do this. If you don’t want to, that’s fine, just take out a free subscription. Or, if you don’t want to get stuff in your inbox, just follow me.

Or…

Ask an LLM a well-put question and the likelihood is that you will get a sensible answer. Ask the same question again, though, and you might get an equally sensible but slightly different answer.

For example, I was trying to get an API to write executable code, and it turned out to be a challenge because no matter how many times and in how many places I told it not to, once in a while, it would put Markdown-style fencing around the code, making it syntactically wrong. (OK, it's not too much of a problem filtering them out, but still…)

The responses from an LLM vary because there is always some randomness in processing the result. When using an API, you can reduce this with the ‘temperature’ setting, but, nevertheless, there will always be some variation.

If you are a human talking to ChatGPT, Gemini, Claude, etc., then this won’t matter. If the wording of the response is a bit different or the layout has changed, you still understand.

But if you are a computer program tuned to process one type of response, then even a slight variation may cause an error.

The solution is to use structured outputs to guarantee a consistent result.

We are going to look at how to create structured outputs using PydanticAI and apply them to a couple of use cases. First, we’ll see how structured outputs work, and then we’ll process unstructured data into a structured format that can be used programmatically. We’ll also see how we can generate runnable code reliably.

Why PydanticAI?

First, it’s open source and easy to use. And second, it is model-neutral: you can use it with several different LLMs (unlike OpenAI’s structured outputs, for example, that only work with their API).

Share

Why Use Structured Outputs from an LLM

There are several benefits in using structured outputs over conventional responses, for example:

• Reliability is improved— enforcing a predictable format will ensure the LLM response is consistent, and no complex parsing of the LLM response is required

• Integration is easier — elements of a structured response can be used directly in an application or more easily processed into a required format

• Development is faster — less code is required, and testing is less complex

Altogether, if you aim to use LLM responses in an application, structured outputs make life more straightforward.

What does a structured output look like

Typically, we are talking about JSON-like responses with specified fields and types. Such a structure can be easily used directly in a Python program as an array or dictionary, or may be converted into a Pandas dataframe or other data structure.

Take a look at the following message from a customer and the prompt.

complaint = """Hi, this is Jane. My internet has been down for two days 
               and I'm very upset."""

prompt = f"""
Extract the following information from the customer complaint:

Customer Name, Issue Category, Sentiment, Urgency.

Return your answer as a JSON object.
{complaint}
"""

We’re asking the LLM to analyse the customer complaint and produce a JSON response that will look something like the following.

{
  "customer_name": "Jane",
  "issue_category": "Connectivity",
  "sentiment": "Frustrated",
  "urgency": "High"
}

In this experiment, we will specify the structured output format a little more formally than this, and since the framework we are using is PydanticAI, it won’t be any surprise that we use Pydantic to define the output structure.

PydanticAI

PydanticAI is a fairly new framework for constructing AI agents. It’s MIT-licensed and pretty easy to use.

The following code is in the form of Jupyter Notebook cells, but you ought to be able to construct a plain old Python file from it, too. I’ll pop a link to the source code at the end of the article.

First things first, we need to import the Pydantic library, and so that it doesn’t interfere with the Jupyter runtime, we need to import nest_asyncio and run the apply method. You won’t need this in a plain Python program.

from pydantic_ai import Agent
import nest_asyncio
nest_asyncio.apply()

Simple structured output

This example comes from the PydanticAI documentation. We ask a simple question, and the answer should be a city. The response should be the name of the city and the country that it is in.

You can see in the code below that we have a very straightforward Pydantic declaration of a class CityLocation. It consists of two strings, one for the city name and the other for the country name.

The agent is defined with the model type and the result_type set to CityLocation , meaning that the response must be of that type.

We then run the agent with a query ‘Where were the olympics held in 2012?’ and print the result. This can be seen in the panel below the code.

from pydantic import BaseModel

class CityLocation(BaseModel):
    city: str
    country: str

agent = Agent('openai:gpt-4o-mini', result_type=CityLocation)
result = agent.run_sync('Where were the olympics held in 2012?')
print(result.data)

city='London' country='United Kingdom'

We could ask any question where the result is a city and receive a response in the same format. But what if the question does not produce a valid output? This could be due to an error in the LLM processing or because it is simply impossible to provide a suitable response to the question given.

In the case of an LLM failure, you probably need to validate the response against the Pydantic model and retry the query. There is a whole section on validation in the PydanticAI documents that deals with this, and it would take another article to deal with it, so I won’t go into it here.

However, when a valid answer isn’t possible, a simple strategy is to ask the LLM to fill in fields in the response that indicate whether a response is valid or not, e.g.

from pydantic import BaseModel

class CityLocation(BaseModel):
    city: str
    country: str
    valid: bool = True
    error: str = None

prompt = """If the answer is not a city, 
            return empty strings for the city and country, 
            return valid=False and an error message."""
agent = Agent('openai:gpt-4o-mini', 
              result_type=CityLocation, 
              system_prompt=prompt)

Here are a couple of queries, one bad, the other good:

result = agent.run_sync('What is a fish?')
print(result.data)

result = agent.run_sync('In what city was the soccer world cup final last played?')
print(result.data)

And the results:

city='' country='' valid=False error='The answer provided is not a city.'
city='Lusail' country='Qatar' valid=True error=None

So, that’s how to get structured outputs from an LLM. Let’s take a look at a couple of practical examples that might form part of an application.

We’ll first look at how we can transform unstructured customer feedback into a form that we can load into a Pandas dataframe for processing, and then we’ll see how we can generate runnable Python code.

Structured customer feedback

I’m going to use fictitious customer feedback similar to my AI for BI article Transform Customer Feedback into Actionable Insights with CrewAI and Streamlit. It is a set of ratings and comments about products from a made-up online shoe store.

I’ve stored the comments in a Markdown file called shoes.md, and so we can read it like this:

fb_raw = "shoes.md"
with open(fb_raw, 'r') as f:
    fb = f.read()

print(fb)

When printed, it looks like the following: a star rating and a comment about the product.

# Customer feedback

### CloudStrider Sneakers

1. Stars: 2  
   "The shoes were super comfortable initially, but after just two months, the sole started to peel off! Really disappointed with the durability."  
2. Stars: 1  
   "I ordered my usual size, but these sneakers were way too tight. Had to return them. The sizing is totally off."  
3. Stars: 3  
   "Great for short runs, but they hold onto odors even after washing. Not ideal for heavy use."  
4. Stars: 5  
   "Extremely lightweight and breathable. Perfect for casual jogging. No issues so far!"
### PeakTrek Hikers
1. Stars: 1  
   "These boots claim to be waterproof, but my feet were soaked after just walking through damp grass. Not worth it."  
2. Stars: 2  
   "I slipped multiple times on rocky trails because the soles don’t grip well on wet surfaces. Not safe for serious hikes."  
3. Stars: 4  
   "The boots are sturdy and provide decent ankle support, but I wish the padding was thicker for longer treks."  
4. Stars: 5  
   "Used these for a weekend hike in the mountains, and they held up really well. No blisters, and they felt solid."

I want to transform this into a structured format, and I use the following prompt:

description=f""" 
            Analyse '{fb}' the data provided - it is in Markdown format. 
            The data is about the range of shoes in an online shop each with 
            set of messages from customers giving feedback about the shoes that they have purchased.
            """

This is used along with two Pydantic models: the first is the format of an individual comment, while the second defines a list of those comments. An Enum is used to define the possible sentiment values.

The agent is then called using the list as the structured output definition.

from pydantic import BaseModel
from enum import Enum

class Sentiment(str, Enum):
    negative = 'negative'
    neutral = 'neutral'
    positive = 'positive'

class feedback_report_shoe(BaseModel):
    product: str
    overall_rating: int  
    issue: str 
    review: str
    sentiment: Sentiment

class feedback_report(BaseModel):
    feedback: list[feedback_report_shoe]

agent = Agent('openai:gpt-4o-mini', result_type=feedback_report)
result = agent.run_sync(description)

The result, in JSON format, can be seen using the following print statement.

print(result.data.model_dump_json(indent=2))

And here is a sample of that output:

{
  "feedback": [
    {
      "product": "CloudStrider Sneakers",
      "overall_rating": 2,
      "issue": "Durability",
      "review": "The shoes were super comfortable initially, but after just two months, the sole started to peel off! Really disappointed with the durability.",
      "sentiment": "negative"
    },
    {
      "product": "CloudStrider Sneakers",
      "overall_rating": 1,
      "issue": "Sizing",
      "review": "I ordered my usual size, but these sneakers were way too tight. Had to return them. The sizing is totally off.",
      "sentiment": "negative"
    }
...

  ]
}

Now that we have a nicely structured output, we can easily convert it into a dataframe for further processing.

import pandas as pd
df = pd.DataFrame([dict(f) for f in result.data.feedback])
df['sentiment'] = df['sentiment'].apply(lambda x: x.value)
df

This code will give us the dataframe illustrated below.

We can process the data further, for example, ranking the products according to their star rating.

average_ratings = df.groupby('product')['overall_rating'].mean()
average_ratings = average_ratings.sort_values(ascending=False)
print(average_ratings)

We first grouped the products, calculated the mean of the rating and then sorted them.

BounceMax Runners        4.75
EcoStep Slip-Ons         4.50
LuxeLace Oxfords         4.50
PeakTrek Hikers          3.00
CloudStrider Sneakers    2.75
SwiftShine Heels         2.50

In this example, getting the LLM to structure the data makes it easy for us to process it. Now, let’s look at my code generation problem.

Share

Unstructured code generation

LLMs are quite proficient at producing code, but, as I discussed earlier, getting them to produce something consistent and usable is slightly trickier.

Here is a simple example of code generation. We simply asked the agent to produce a Python function to add two integers.

from pydantic_ai import Agent

agent = Agent(  
    'openai:gpt-4o-mini',
)

result = agent.run_sync("Write a Python function to add two integers"
                        "together and give an example call")  
print(result.data)

Here is a simple Python function that takes two integers as input, 
adds them together, and returns the result:

```python
def add_two_integers(a, b):
    """
    Add two integers together.

    Parameters:
    a (int): The first integer.
    b (int): The second integer.

    Returns:
    int: The sum of the two integers.
    """
    return a + b

# Example call
result = add_two_integers(5, 3)
print(result)  # Output: 8
```

In this example, the function `add_two_integers` is defined to take two 
parameters, `a` and `b`. It calculates the sum of these two integers and 
returns the result. The example call at the bottom demonstrates how to use 
the function by adding the integers 5 and 3, resulting in an output of 8.

The result is typical of a chat-like response and contains exactly the right information. But I cannot use the output to directly run the code because the response is targeted at a human not a computer program.

Structured code generation

Here is a different version with a Pydantic model defined to split the output into the code section and a commentary.

class Code_Output(BaseModel):
    code: str
    commentary: str

agent = Agent('openai:gpt-4o-mini', result_type=Code_Output)

result = agent.run_sync("Write a Python function to add two integers" 
                        "together and give an example call")  

The code part of the response contains a runnable function — no additional information or Markdown fencing, just the code.

print(result.data.code)

def add_two_integers(a, b):
    """
    This function takes two integers and returns their sum.
    """
    return a + b

And the commentary contains the rest of the output.

print(result.data.commentary)

# Example call
result = add_two_integers(5, 3)
print(result)  # Output: 8

The function 'add_two_integers' is defined to take two integers as parameters 
and return their sum. An example call is provided where the function is 
called with the integers 5 and 3, and the result is printed.

We can run the code easily. Here is a Notebook cell that does just that. The functional part of the cell is just exec(result.data.code). The rest of the code ensures that the user can review the code and is warned that running it could cause problems. It is a bad idea to run LLM-generated code unseen unless it is in a sandbox environment such as Docker. LLMs make errors and while it is unlikely that a function to add two integers will cause any harm, we should still check.

import ipywidgets as widgets

text = widgets.HTML("<h3>Running the code is potentially dangerous</h3>") 
button = widgets.Button(description="Run it, anyway",
                         button_style='danger')
output = widgets.Output(layout={'border': '1px solid black'})
with output:
    print("Code result:")

display(widgets.VBox([text,button, output]))

def run(b):
    with output:
        exec(result.data.code)

button.on_click(run)

VBox(children=(HTML(value='<h3>Running the code is potentially dangerous</h3>'), Button(button_style='danger',…

Conclusion

Structured outputs are a useful way of ensuring consistency in an LLM’s response, as I hope these simple examples have illustrated.

They are easily implemented in PydanticAI: simply define a Pydantic model and use that in the call to the AI agent. The result will be a data structure that is easily used in further coding.

I haven’t properly covered what happens if the LLM does not respond as required, which is always a possibility, as this could easily be a lengthy discussion. The method that I showed above only copes with the simplest of errors, and more sophisticated error detection may well be necessary for a particular application, for example, validation against the model and retrying the query - PydanticAI discusses this in their docs.

As ever, thanks for reading and I hope that this has been useful. To see more of my articles, follow me on Medium or subscribe to my Substack.

The code and data for this article can be found in my GitHub repo. I will place a link to it here shortly after publication.

Currently, although paid subscriptions are open, you won’t get anything extra - this may change in the future. But if you feel inclined to support my work regularly, you could do this. If you don’t want to, that’s fine, just take out a free subscription. Or, if you don’t want to get stuff in your inbox, just follow me.

I’ve been away for a little while, wandering around the woods of rural England, doing very little and not being online. Before I disappeared, I discovered that Google has integrated its Gemini LLM into its online Notebook environment Colab. Here is the result of my experiments in getting Gemini to automatically analyse and visualize data using a public data set about the popularity of various programming languages.

Ok, forget everything I’ve said in the past about using AI to create data visualizations. This could make everything else redundant.

Just open a Google Colab notebook, upload your data and ask its Data Science Agent to do all the work for you.

Google’s Data Science Agent is based on Gemini, its LLM offering. It is incorporated into Colab and can simplify data analysis by automating the creation of complete Jupyter notebooks. It has two modes: at the cell level, it lets you prompt Gemini to generate code there in the cell, but it can also generate entire notebooks from data files and a user prompt. Additionally, Gemini can also explain errors in your code and fix them.

According to Gemini itself, the agent's benefits include:

Fully functional Colab notebooks: Not just code snippets, but complete, executable notebooks.

Modifiable solutions: Easily customize and extend the generated code to fit your specific needs.

Sharable results: Collaborate with teammates using standard Colab sharing features.

Time savings: Focus on deriving insights from your data instead of wrestling with setup and boilerplate code.

So, let’s give it a whirl and see how it works.

We’ll give it a dataset that tracks the popularity of 30 programming languages over time. Then we’ll ask it to analyse the data. Gemini will come up with a plan which you can accept (or you can ask for modifications) and it will then execute that plan, creating code cells and executing them to complete the tasks it defined in its plan.

Later, we’ll look at what happens when we prompt the agent to produce a more directed plan that specifies our particular interests.

First, however, we will be lazy, accept a default plan, and let Gemini go its own way.

You’ll need a Google account to run Colab. If you are not familiar with it, it is essentially a Jupyter Notebook running on a Google server but with extras. To quote Google’s Gemini:

“Colab provides free access to computing resources, including GPUs and TPUs, making it suitable for machine learning, data science, and education”

It's free to use, but the availability of resources may vary depending on demand. You can find it here:

https://colab.research.google.com/

The data and the notebooks for this exercise will be available for download from my Github repo shortly after publication. See the Notes below for details.

Share

Colab with Gemini

When you first log in to Colab, you need to create a new notebook, which will look very much like a standard Jupyter Notebook. However, below the initial empty code cell, you will be presented with a button that invites you to “Analyse files with Gemini”.

Click on that and you will be prompted to upload a file. The file I uploaded looks like the screenshot below.

It is a simple table of the popularity of programming languages from July 2004 to the end of 2024. The numbers in the language columns are percentages and represent the proportion of users of a language (see Notes).

You should be aware that the data you upload will disappear when you finish your Colab session. (In the notebooks I provide for download, I have substituted the reference to a local file with an external source so the data will be persistent across sessions. You can upload these notebooks to Colab or run them in a local instance of Jupyter.)

A generic plan

After I uploaded the data, I was invited to ‘Analyse Files’. Having done that, Gemini responded thus:

The response will be tailored to whatever file you upload, of course.

You can ask for changes, although for this exercise I accepted the default plan and executed it.

Time to make a cup of tea.

Actually, executing the plan doesn’t take that long with this fairly small file, but with a large dataset, I can imagine you might have to wait a while for Gemini to work through the various tasks.

Gemini keeps you posted about its progress, gradually ticking off the tasks as they are completed.

And, hey presto, we have a complete notebook with code cells for each task. Sometimes, Gemini will create coding errors. When this happens, it will attempt to retry the code in a new cell. When Gemini runs the notebook, it will run all the cells even though there was an error, but if you try to re-run it, it will stop at that point. To get a fault-free run, you need to manually remove the guilty cell.

Since we have given no direction on how to proceed, Gemini’s plan is fairly generic and not necessarily suitable for our purposes. But let’s see what it has done for us.

Each task in the plan is coded in a separate Jupyter cell, and this is preceded by a text introduction. We’ll go through each one.

• Data Loading: This simply loads a local CSV file into a dataframe and prints out the head.

• Data exploration: This stage prints out the shape of the dataframe and the datatypes. It checks for missing values (there are none) and then prints statistics for each column (i.e. language).

• Data cleaning: Here, the Date column is converted to a datetime format.

• Data analysis: This task analyses the trends in programming language popularity over time. It calculates the average popularity for each programming language, determines the peak popularity and corresponding date, identifies significant changes in popularity, and calculates the average popularity per year for each language. The first attempt at doing this failed as Gemini tried to operate on a Year column that it had created, which was not a numerical field. It automatically made a second (successful) attempt to correct the original mistake.

• Data visualization: Gemini produces the code to plot various graphs in Matplotlib — popularity over time, average popularity and so on.

Here is a sample of one of the visualizations; it tracks the popularity of programming languages over time using the average for each year.

Following these operations, Gemini comes up with a Summary telling us what it has done and suggesting further steps.

You have to bear in mind that Gemini was given no instructions other than to analyse the data. So, it has produced a generic plan and then executed it. If we had wanted something different, we could have asked for it (see below).

Generating new cells.

That needn’t be the end of the story. Gemini is active inside each code cell. Each time you create a new code cell, you will see this:

Click on ‘generate’ and you will be shown a dialog box in which you enter a prompt. Gemini already knows about the Notebook and will happily generate the code for a new cell. For example, I asked it to

“Create a plot from the original data that plots the average popularity of the top 10 most popular languages for 2024. The result should be a bar plot.”

It dutifully produced the code to draw the bar chart, but there was a bug. Here is the code that it generated:

# prompt: Create a plot from the original data that plots the average popularity of the top 10 most popular languages for 2024. The result should be a bar plot.

import matplotlib.pyplot as plt

# Assuming 'df' is your DataFrame from the previous code

# Get the latest popularity data (2024)
latest_data = df[df['Date'] == df['Date'].max()]

# Drop the 'Date' and 'Year' columns
latest_popularity = latest_data.drop(['Date', 'Year'], axis=1).iloc[0]

# Sort by popularity and get the top 10
top_10 = latest_popularity.sort_values(ascending=False).head(10)

# Create the bar plot
plt.figure(figsize=(10, 6))
plt.bar(top_10.index, top_10.values)
plt.xlabel("Programming Language")
plt.ylabel("Popularity")
plt.title("Average Popularity of Top 10 Languages (2024)")
plt.xticks(rotation=45, ha='right')  # Rotate x-axis labels for better readability
plt.tight_layout()
plt.show()

It got a little confused about the column name ‘Year’. This is something that Gemini itself created and is not in the original data. It is easily fixed manually, but it is useful to know that Gemini is capable of fixing its own errors. After an error message you are presented with an ‘Explain error’ button. Click this and the Gemini window displays the problem, a hypothesis (what it thinks caused the error), and a fix.

It correctly diagnosed the error:

It then went on to produce the correct code (instead of dropping the ‘Date’ and ‘Year’ columns, it just dropped ‘Date’), which drew the chart below.

I tried a different prompt in the next cell.

Compare how the popularity of Python compares with a combination of Javascript and Typescript.

JavaScript and TypeScript do much the same job, and I would think that at least some of the decline in the use of JavaScript can be attributed to developers moving to TypeScript. So this prompt asks the agent to combine the two languages and compare them with Python.

Gemini produced the following code.

import matplotlib.pyplot as plt
# Calculate the combined popularity of JavaScript and TypeScript.
df['JavaScript_TypeScript'] = df['JavaScript'] + df['TypeScript']

# Calculate the average popularity of Python and the combined JavaScript/TypeScript.
avg_python_popularity = df['Python'].mean()
avg_js_ts_popularity = df['JavaScript_TypeScript'].mean()

print(f"Average popularity of Python: {avg_python_popularity}")
print(f"Average popularity of JavaScript and TypeScript combined: {avg_js_ts_popularity}")

# Find the peak popularity and corresponding date for Python and combined JavaScript/TypeScript
peak_python_popularity = df['Python'].max()
peak_python_date = df.loc[df['Python'].idxmax(), 'Date']

peak_js_ts_popularity = df['JavaScript_TypeScript'].max()
peak_js_ts_date = df.loc[df['JavaScript_TypeScript'].idxmax(), 'Date']

print(f"\nPeak popularity of Python: {peak_python_popularity} on {peak_python_date}")
print(f"Peak popularity of JavaScript and TypeScript combined: {peak_js_ts_popularity} on {peak_js_ts_date}")

# Plotting
plt.figure(figsize=(12, 6))
plt.plot(df['Date'], df['Python'], label='Python')
plt.plot(df['Date'], df['JavaScript_TypeScript'], label='JavaScript + TypeScript')
plt.xlabel('Date')
plt.ylabel('Popularity')
plt.title('Popularity of Python vs. JavaScript + TypeScript')
plt.legend()
plt.grid(True)
plt.xticks(rotation=45)
plt.tight_layout()
plt.show()

Which comes up with the following text output as well as the chart that follows.

Average popularity of Python: 14.725691056910566
Average popularity of JavaScript and TypeScript combined: 8.893536585365853

Peak popularity of Python: 31.41 on 2020-07-01 00:00:00
Peak popularity of JavaScript and TypeScript combined: 12.469999999999999 on 2023-06-01 00:00:00

Gemini has responded to the prompt correctly. And we can see that both lines in the graph show a general increase in popularity, but with Python surging ahead from around 2016 and JavaScript/TypeScript declining a little recently.

A more directed prompt

That’s one way of using the agent, but probably a more productive method is to tell the agent what you are interested in.

I went through the same exercise, but this time, instead of simply asking it to analyse the data, I gave it this prompt:

We are interested in how Python has increased in popularity as a language for AI and how Typescript might be replacing Javascript as a front-end language.

And this was the plan that the agent proposed.

As you can see, the plan is much more focused than the previous generic one and concentrates on the languages that we were asking about.

The data analysis and visualisations concentrated on the growth in popularity of the three languages and below you can see a chart that highlights Python’s periods of significant growth in popularity.

The full analysis and the charts generated can be seen in the notebook. Below, you can see that the summary that Gemini produced was very relevant, and it provided sensible suggestions for further investigation.

Conclusion

Google’s Data Science Agent is a very easy-to-use feature in Colab, just upload the data file(s) and write a suitable prompt to guide the agent’s analysis, and it will generate a plan, execute it and produce an entire notebook for you based on the data provided.

If you only give it a vague prompt, then the agent will produce a generic analysis. So it is worth putting in some effort to think about what you need from the analysis and writing a prompt that explains this. The better the prompt, the more useful the result.

I have only given you a brief look at the analysis, visualizations and code that Gemini created. I would encourage you to look at the full notebooks in the repo where you will see the entire analysis, all of the charts, the code and the commentary that Gemini produced.

I have found the combination of Gemini and Colab to be quite satisfying to use — easier than some other AI-supported tools, and I suspect I may continue to use it in future. Take a look at it and, if you feel like it, write a comment about your experiences below.

As ever, thanks for reading and I hope that this has been useful. To see more of my articles, follow me on Medium or subscribe to my Substack.

Notes

1. The data comes from the Kaggle dataset Most Popular Programming Languages Since 2004 by Muhammad Khalid, accessed 29th April 2025. A copy of the version used here can be found in my GitHub repo, here.

2. All the code and data can be found in the repository, here.

Subscribe now

Using LLMs to analyse data and produce visualizations is not new - well, it's fairly new, I suppose, but it's nothing spectacular these days.

But getting an AI to go through a complete dataset to do this is inefficient. And, if your data set is large, you could end up with an unexpected bill for your API usage, have your request refused, or even break the context window.

So here's an alternative.

A bit of easily generated metadata can be used by an LLM to create the code for creating a visualization without needing to read a single cell of the data. If you have a dataset, it's likely that you know something about it - so you are one up on an LLM already - and using that knowledge, it becomes easy to instruct the LLM with a little descriptive data rather than the data itself.

I'm going to describe the code involved as cells in a Jupyter Notebook and then show you a Streamlit app that allows the user to upload a file and specify a chart to be drawn.

Subscribe now

Don't rely on AI for everything - it's tempting but not efficient.

I have a CSV file that records CO2 emissions for many countries since the mid 19th century. It has only 5 columns but over 25,000 rows. It's less than a megabyte of data, but that is enough for my preferred LLM setup to refuse to deal with it.

But if I give an LLM a little information, it can do the work I want without looking at the data at all.

Here is the information I give it.

• File name: 'data/co2_total.csv'

• Data columns:

  • unnamed: index

  • 'Entity': a string containing a country or area name

  • 'Code': a short code for 'Entity'

  • 'Year': an integer representing the year

  • 'Annual CO2 emissions': a real number representing the CO2 Emissions in billions of tonnes

With this, I can ask the LLM to write Python code to analyse the data and produce charts.

For example, I could ask it to provide code to plot the yearly emissions of a set of countries. I could ask it to work out the top 5 emitters and draw a bar graph of their emissions. I show how these two examples work below.

The code to do this is not difficult. I'll go through a set of Jupyter Notebook cells step by step for this particular dataset.

Code inspection or sandboxing

You can't trust them! LLMs are notorious for hallucinating, and while it might be unlikely, it is possible that the code produced could be destructive. There are two ways of preventing this: check the code before executing it or execute it in a sandboxed environment where it cannot do any harm.

If we want non-programmers to use an app like this, then the only option is sandboxing, but to introduce the technique, we'll assume the user will understand the code generated and let him/her review it before running it.

When it comes to the Streamlit app, the code can also be reviewed and edited before running it, but if the app is published, then it is running in a container and thus isolated from anything that it could harm. If you run it locally, you might be wise to use Docker to execute the program.

The Notebook

I'm going to specify a particular CSV file that I derived from Our World in Data and a simple metadata file that contains the column information (as we saw above).

You could use pretty much any AI framework, or none, but I like PydanticAI, so I'm going to use that. It's open source and easy to understand.

A good thing about PydanticAI is that it doesn't care which model you use; you simply tell it the model name, and it will deal with it. The code below uses OpenAI since I already have an account with them. But any mainstream LLM would suffice.

For plotting, I've tried the code with Matplotlib, Altair and Plotly - all work, but my preference is to use Plotly. Other plotting libraries would probably be fine as long as the LLM is familiar with them.

If you intend to use this code, you will need an OpenAI account and an API key. And they will charge you a small amount for usage (I've run the code many times, and it has cost me a few 10s of cents). Other LLM providers might be cheaper (or not).

The Data

The data looks like the table below. It is just a set of carbon emissions over time for a list of countries.

I wrote the metadata that the LLM is going to use in a variable metadata, as you can see below.

metadata = """
File name: 'data/co2_total.csv'
Data columns:
     unnamed: index
    'Entity': a string containing a country or area name
    'Code'  : a short code for 'Entity'
    'Year'  : an integer representing the year 
    'Annual CO2 emissions': a real number representing the CO2 Emissions in billions of tonnes
"""

I then decided on what charts I wanted the LLM to produce and put this into the variable request.

# Change this for the chart you want to create

request = """
The chart should display the emissions for France, United States, 
China and Russia over time.
"""

This expects the data to be filtered on the countries given and a chart drawn from that.

Both of these are variable data (in the interactive application, they are set by the user). If you'd like to try it out with a different dataset, these are the cells you need to change.

It's all in the prompt

In the next cell, I have defined three things. The first is the plotting library I want to use; as I mentioned, I've successfully used Matplotlib or Altair, but here it is set to ''Plotly'' with some extra instructions added in brackets.

The next string is the user prompt that will be sent ot the LLM. This is constructed with the variables that were set above.

Next is the system prompt, which is used to set up the Agent.

plotting_lib  = "plotly (use the plotly_white template and a figure size of 1200 x 800)"

prompt = f"""
A description of data the follows...

<description>
{metadata}
</description>

Your task is as follows...
<task>
{request}
</task>

The expected output is valid Python code for a {plotting_lib} chart
"""

# System prompt

sys_prompt= f"""
You are an expert in writing Python code for data analysis and visualization.
Your aim is to read and analyse data and create the code for a chart as instructed.
Do not read the data directly. Use the metadata provided to create the a {plotting_lib} chart 
from the data described there. Do not use Markdown fencing around the code.
"""

Frankly, this is where most of the work is done. These prompts ensure that the LLM does not try reading the data but only uses the metadata to create a program that responds to the user request and uses the specified plotting library.

The Agent

The agent will be set up with the prompts above and then send the request to the LLM.

According to the PydanticAI docs, running PydanticAI in Jupyter will cause this error:

RuntimeError: This event loop is already running

The error is caused by conflicts between the event loops in Jupyter notebook and PydanticAI's. ' One way to manage these conflicts is by using nest-asyncio. Namely, before you execute any agent runs, do the following:

from pydantic_ai import Agent
import nest_asyncio
nest_asyncio.apply()

Having done that, we can get on to the useful code. Below, we create the agent with an OpenAI model and a system prompt, run it with the user prompt and print the result for the user to review.

agent = Agent(  
    'openai:gpt-4o-mini',
    system_prompt=sys_prompt,  
)

result = agent.run_sync(prompt)  

print(result.data)

The code that it produced with this setup is reproduced below. It's pretty much what I would have written had I done it by hand, so it must be quite good, right?.

# Agent-generated code
import pandas as pd
import plotly.express as px

# Load the data
df = pd.read_csv('data/co2_total.csv')

# Filter the data for specific countries
countries = ['France', 'United States', 'China', 'Russia']
filtered_df = df[df['Entity'].isin(countries)]

# Create the plot
fig = px.line(filtered_df, x='Year', y='Annual CO2 emissions', color='Entity',
              title='Annual CO2 Emissions for Selected Countries Over Time',
              template='plotly_white')

# Update the figure size
fig.update_layout(width=800, height=600)

# Show the figure
fig.show()

To be crystal clear, the code above is the output of the code so far - don't include this in a Notebook cell!

The next cell just asks the user whether the code should be run. And if the answer is yes, then the code is executed.

execute_code = input("Do you want to execute this code? (yes/no): ").strip().lower() == 'yes'

if execute_code:
    exec(result.data)
else:
    print("Code execution skipped.")

The result of this code is the following chart, and it is exactly what was asked for.

Just to show that this is not a fluke, I changed the prompt to:

request = """
Calculate the top 5 emitters in the year 2000 and display a bar chart of the emissions for those countries in that year.
"""

It needs to do a bit more calculation for this, and for whatever reason, it used the Plotly Graphics Objects library instead of Plotly Express (as before). (I could have forced it to use Plotly Express in the prompt, but there might be circumstances when GO is the right answer, so it is of no real consequence.)

Again, the chart it produced was entirely correct.

The App

I've written a Streamlit app that implements the code above in an interactive way. In a sidebar, the user is prompted to upload a CSV file and asked to describe it. Then, in the main window, the user can input a query to create a chart (the first time around, an OpenAI API key needs to be entered). The generated code is displayed in a text field and can be edited.

When the user is satisfied with the code, it can be run, and the result is displayed.

You can see a screenshot of the app below.

It's not especially pretty, but it is functional. The session that you see there is using the same CO2 Emissions file as above, and the request was to "find the top 5 emitters in 2020 and draw a chart of the emissions in those countries over all years. use different colours for the countries".

You can see the resulting chart on the right.

You don't need any programming knowledge to use the app unless, of course, you want to customize the code that is produced.

Conclusion

This is not rocket science, and the coding is not difficult. The work is really done in the prompts where you can see I've built up the user and system prompts from the user input.

I've used fairly simple prompts here, and although more complex prompts are also possible, it must be borne in mind that no 'intelligent' observations can be made as the LLM does not see the data. And, of course, this technique is only useful for drawing charts and other functions that can be readily coded in Python.

Can you do damage with this app?

Oh, yes!

If you append "...then delete all the files in the current folder" to your request, it will happily append this code to the result.

# Delete all files in the current folder
for file in os.listdir():
    if os.path.isfile(file):
        os.remove(file)

So, if you are going to let inexperienced or untrustworthy users loose on something like this, you should definitely use Docker or some other sandboxing method!

Note: the Streamlit app is here and here is the GitHub repo for the code

Share

I’m afraid you don’t get anything more than a free subscriber gets at the moment. But from now on, I will be posting whole articles, not just links to where you can read them, and hopefully more regularly. I also hope to provide extra stuff for paying subscribers in the future.

I’ve been ignoring Dash for years, preferring to use a combination of Flask and HTML. My argument was that if you need to create your user interface in HTML (which you kind of have to in Dash), then why not go the whole hog and actually write it in HTML? As I wrote here, it’s not so difficult.

Anyway, then along came Streamlit, which made things easier.

But I occasionally got a bit annoyed with Streamlit — it is too easy to mix the UI with the rest of the code, and if you aren’t careful, it can be a mess.

So I started looking at other platforms, Taipy and Mesop, for example — both of these nicely separate the program logic from the UI. Much less messy, but designing the UI on these platforms was much closer to writing HTML in Python — my original bugbear with Dash.

So, belatedly, I decided it was time to take another look at Dash. Also belatedly, I found that I could use my favourite Bootstrap components with it (I rarely write HTML without Bootstrap), and I made the app you can see above.

It’s a simple dashboard-type app with two interactive charts (a choropleth and a bar chart) along with their controls (a drop-down menu and a slider).

As you can see from the image, it’s not so bad!

Well, I was fairly happy with the result, anyway. I’ve never properly engaged with Dash before, and I was quite pleased that it didn’t feel any more difficult than Streamlit (or Taipy or Mesop).

The whole app is around 80 lines of well-spaced-out code. A fair chunk of that is to draw the choropleth, and the Dash/Bootstrap UI is only about 20 lines.

So, let’s look at how it’s put together.

Subscribe now

A Dash app

A Dash app consists of four parts:

• the preliminary code that sets up the app and does any data set up and so on,

• the page layout, which is defined as Dash and Bootstrap components,

• callbacks, which are the functions that are called in response to user interaction,

• the app run command.

We’ll look at all of these and how they are put together shortly, and you will find all of the code in my GitHub repo.

But first, let’s look at the way we define the page.

Page layout

When deployed, a Dash app consists of two parts: the web page displayed in the browser and the code that supports it, which remains on the server. This separation is hidden; the HTML page layout, which will become the web page, is defined in the app code.

Both Dash and Bootstrap provide us with components that get converted to HTML. Bootstrap also incorporates styles we can adopt to make our page look good and stylistically consistent.

I have defined a default style using Bootstrap that gives every component a common look: a border, a light background, margins around the component and padding inside it. This is a single line of code (that we will see soon) that is applied to each component. The layout of the page looks like the image below. You might find the design a little fussy, but one of my aims is to demonstrate the use of Bootstrap styles, and this does the job, as we will see later.

There are four main components to the UI: the outer container that wraps the whole page; a header that is styled a little differently to make it stand out; and two columns that contain the charts, the user controls and some text. At the bottom of the outer container, there is a footer — just a string of text.

In HTML, we would write this page as a set of tags, mostly <div> tags, that are nested inside each other. With Dash, we use the components provided and code the UI in Python.

A first Dash app with Bootstrap styling

Let’s have a quick blast through a simple Dash app that uses Bootstrap components.

Below is the code for a very simple Dash app that uses Bootstrap styling, and below that is a screenshot of the result. We will use this as the basis of the full app.

from dash import Dash, html
import dash_bootstrap_components as dbc

# Initialize the Dash app with Bootstrap stylesheet
app = Dash(external_stylesheets=[dbc.themes.BOOTSTRAP])

# Define a default CSS class for styling
DEFAULT_CLASS = 'border border-primary rounded bg-light p-2 m-2'

# Define the layout of the app
app.layout = dbc.Container(
    [
        html.H1("A Bootstrap 'primary' styled header",
        className="text-primary"),
        html.Div("Some normal text in a 'secondary' style",
        className='text-secondary'),
    ], className=DEFAULT_CLASS
)

# Run the app
if __name__ == '__main__':
    app.run(debug=True)

The first couple of lines import the required libraries. If you want to follow along with the coding, you’ll probably want ot create a virtual environment and pip install these (or use whatever your favourite installation method is). Later, you will also need plotly, pandas and numpy.

After that, the Dash app itself is created, and here we need to tell Dash that we will be using the Bootstrap themes.

There are two aspects of the Bootstrap components: one is the component themselves (e.g. the rows and columns we’ll see later). Styles are defined as classes by Bootstrap, and we can string them together to make quite complex styles. For example, there are classes that define borders, ones that define margins and padding, and those that determine how the text and the background should look.

As I mentioned before, I set up a default style, and that is the next line of code that you see.

DEFAULT_CLASS = 'border border-primary rounded bg-light p-2 m-2'

The meaning of each part is as follows:

• border - there is a border around the component

• border-primary - that is the colour of the border

• rounded - the border has rounded corners

• bg-light - the background colour

• p-2 - two pixels of padding

• m-2 - a two-pixel margin

There are many, many ways of setting up the way that Bootstrap components look. See the [Bootstrap docs](Get started with Bootstrap · Bootstrap v5.3) for the full set of components and styles and the Dash Bootstrap Components for how to use them in Dash.

We can see the usage of this default style in the first component in the layout. Because that is the next thing we come across. The app.layout defines how the app will look, and here we assign it to a component called dbc.container. This is a general-purpose container provided by Bootstrap, which contains a list of elements or other containers. At the end of that list, we can specify the style of the container, and it is here that we use the default style.

Inside the container, there are two more elements: a header and a div container. The purpose of a header is self-explanatory, and a div is another general-purpose container (that will create an HTML tag).

Each of these elements contains a string, which will be displayed and a classname which defines the style (in this case, the colour).

The last couple of lines of code run the app, and in this case, we set the debug flag to True. This is convenient for development, as it will reload the app whenever it detects a change.

The thing we don’t see in this app is the dynamic part — the callback functions. These are invoked when a UI component changes, and we need something to be done. We’ll get on to those later.

Share

Callbacks — adding charts and controls

The final app displays charts of CO₂ emissions that come from a local CSV file.

Our next app gets part of the way there by reading data and building a chart from it. Furthermore, it incorporates a user control (a slider) that lets the user choose the particular aspect of the data to be displayed (the year).

This illustrates how Dash deals with user input via callback functions. These functions are linked to values in the user interface: the input that has changed and an output value that the callback will calculate from that changed input.

Here is what our data looks like.

It is a simple table that records CO₂ emissions for a country (‘Entity’) in a particular year and is sourced from Our World in Data.

The application allows users to explore CO₂ emissions data over time by selecting a specific year from a slider. A choropleth map updates dynamically, displaying emissions for each country using a color scale.

First, we import the necessary libraries — there are a couple more this time:

from dash import Dash, html, dcc, callback, Output, Input
import plotly.express as px
import pandas as pd
import dash_bootstrap_components as dbc

• Dash and Dash Bootstrap Components (dbc) as before.

• Plotly Express (px) is used to generate the choropleth map.

• Pandas loads and manipulates the CO₂ emissions dataset.

As before, we create an instance of Dash and apply an external Bootstrap stylesheet for styling:

app = Dash(external_stylesheets=[dbc.themes.BOOTSTRAP])

The dataset, stored in a CSV file, is loaded using Pandas. We record the minimum, maximum, and midpoint years for configuring the slider later:

df = pd.read_csv("data/co2_total.csv")
marks = {int(i): str(i) for i in df.Year.unique() if i % 50 == 0}

min_year = df.Year.min()
max_year = df.Year.max()
mid_year = (max_year + min_year) // 2

Again, we define the default layout.

DEFAULT_CLASS = 'border border-primary rounded p-2 bg-light m-2'

Defining the Layout

As we have seen above, Dash uses a declarative layout structure and here, we organize components using Bootstrap’s Container and Row:

app.layout = dbc.Container([
    html.H1('Dash App with Bootstrap Components - CO₂ Emissions', 
             className="{DEFAULT_CLASS} text-center text-dark bg-dark-subtle"),
    dbc.Row([
        html.Div('Global CO₂ Emissions', className='h3 text-danger'),
        html.Div("Move the slider to display CO₂ emissions by country for a specific year", className='text-secondary'),
        dcc.Slider(min=min_year, max=max_year, value=mid_year, step=1, marks=marks, id='slider-value'),
        dcc.Graph(id='graph2-content'),
        dcc.Markdown("""
            The data used in this article is derived from Our World in Data.
            OWD publishes articles and data about the most pressing problems that the world faces.
            All its content is open source and its data is downloadable.
            See [Our World in Data](https://ourworldindata.org/) for details.
        """, className='text-secondary')
    ], className='border border-primary rounded p-2 bg-light m-2')
], className="container-fluid")

Here are the components in detail:

• Container: This is a a general purpose container for the whole app

• Title (H1): The main heading is styled with Bootstrap classes — the text and background style from the default style are overridden.

• Row: by itself, a row doesn’t do anything except act as another container. The reason for using a row here is because the final app will create two columns in the row for the two graphs

• Divs: These are text containers that provide user guidance and context — the first is an H3 header that is coloured red (the ‘text-danger’ class).

• Slider: Allows users to select a year to update the map.

• Graph: Displays CO₂ emissions using a choropleth map.

• Markdown: A markdown container can contain any valid markdown text and here provides data attribution.

Creating the Callback for Interactivity

Dash uses a callback function to update the map when the slider value changes. A callback function is preceded by the @callback decorator, which defines the input and output for the function. The input and output consist of two parts: the ID of a UI element and a value type. In this case, the input comes from the slider, and it is a 'value' (the year), while the output goes to the graph container and is a 'figure'.

@callback(
    Output('graph2-content', 'figure'),
    Input('slider-value', 'value')
)
def update_graph(value):
    col = 'Annual CO₂ emissions'
    max_val = df[col].max()
    min_val = df[col].min()

    fig = px.choropleth(df[df['Year'] == value],
        locations="Code",  # Country ISO codes
        color=col,  # CO₂ emissions data
        hover_name="Entity",  # Country name
        range_color=(min_val, max_val),
        scope='world',
        projection='equirectangular',
        title=f"CO₂ Emissions by Country in {value}",
        color_continuous_scale=px.colors.sequential.Reds
    )

    fig.update_layout(
        margin=dict(l=0, r=0, t=60, b=0),
        coloraxis_colorbar=dict(title="Tonnes"),
    )

    return fig

The function is quite simple: it listens for changes in the slider value and draws a Plotly choropleth with the dataframe filtered by value which is the year selected by the slider. The updated figure is returned to the graph component.

Finally, we start the Dash server to launch the app.

if __name__ == '__main__':
    app.run(debug=True)

The result can be seen in the screenshot below.

That is all you need to know to create the complete app, so I won’t go into great detail more here (the complete code is in my [GitHub](public_code/dash-bootstrap at main · DataVizandAI/public_code · GitHub) repo and the README file will give you details of which file is which, so please visit that).

Below is the code that changes in the UI. You can see that we add two columns to the original row, and the second column contains code very similar to the first. In this column, a bar chart will be drawn, and the input is from a drop-down menu that contains a list of countries.

dbc.Row([
            dbc.Col( # ...the original row content
            ),
            dbc.Col([
                html.Div('Total CO₂ Emissions', className='h3 text-danger'),
                html.Div("Choose a country to display the CO₂ emissions over time", className='text-secondary'),
                dcc.Dropdown(df.Entity.unique(), 'Canada', id='dropdown-selection'),
                dcc.Graph(id='graph-content'),
                ],
                className= DEFAULT_CLASS
            ),
        ]),

To cater for this new chart, we need another callback function.

@callback(
    Output('graph-content', 'figure'),
    Input('dropdown-selection', 'value')
)
def update_graph(value):
    dff = df[df.Entity==value]
    return px.bar(dff, x='Year', y='Annual CO₂ emissions', 
                  template='plotly_white')

This simple function takes the value from the drop-down menu (the country) and plots a bar chart of emissions over time and returns the updated figure to the graph container with the ID ‘graph-content’.

Conclusion

In this tutorial, we built a Dash web application that visualizes CO₂ emissions using a choropleth map. We integrated Bootstrap components to enhance the layout.

Is this easier than Streamlit? You’ll probably think it is more in line with the way that Taipy or Mesop work, but which is the easiest method? This, at least to some extent, is down to personal preference.

I’ve used the open-source version of Dash here — there is an Enterprise version which provides you with tools to make developing and deploying sophisticated apps easier. I’ve no idea how much it costs.

This was, in the main, an exercise to help me make my mind up which data visualisation framework I prefer, and I have to report that I still don’t think I am any the wiser. Perhaps you are. I’d very much like to hear what you think.

Thanks for reading, and I hope this was useful. If you would like to read more of my stuff, most of it is on Medium and also on my website (if you aren’t a Medium member). I also intend to post more complete article here on Substack.

Here are some useful links:

• Dash (open source docs)

• Dash Bootstrap Components

• Our World in Data

• [The repo for this code](public_code/dash-bootstrap at main · DataVizandAI/public_code · GitHub).

All images are by me, the author, unless otherwise indicated

I am not connected with any of the companies mentioned in this article except as a user of their products

Is this the final feature that makes Jupyter Lab a complete IDE? Jupyter already has many features that make it a great IDE, but now it also has GitHub Copilot integration.

We’re going to explore Notebook Intelligence (NBI), a Jupyter Lab extension that brings the power of GitHub Copilot to Jupyter Lab. The author, Mehmet Bektas, has already written an introductory article, on Medium, which is well worth reading (Introducing Notebook Intelligence!). It shows you NBI’s main features and how to install the extension. Bektas also points out that, despite his close relationship with the Jupyter Project, NBI is not an official Jupyter extension. But I would like to point out that as a Jupyter Distinguished Contributor, this guy definitely knows his stuff.

Here I’m going to look at three ways we can get NBI to quickly create new code — both Notebooks and a Streamit program. I’ve published all of the code, including Notebooks that go step-by-step in an associated GitHub repo.

We can use NBI to create a new Notebook in two ways — by giving prompts from within an existing Notebook or by using the /newNotebook command. We can also use NBI to create a new Python program using the /newPythonFile command, which we will use to create a Streamlit app.

Subscribe now

The NBI UI

After installation, NBI can be seen in the toolbar on the left of the Jupyter Lab window. But before you can use it you need to log in to a GitHub account to use Copilot. This is straightforward, to log in NBI will provide you with a code you can copy and a link to click on. The link will take you to GitHub where you can enter the code and log in.

Now NBI is available. You may notice that code completions pop up as you type — just like Copilot in VSCode — just hit <tab> to accept them or carry on typing to ignore them.

You can also hit <ctrl-G> to bring up an NBI prompt in a Notebook. You can ask it to modify the code in that cell or create new code. A nice feature is that it shows you a diff of the original and the suggested code before you accept it. You can also edit the suggested code — the screenshot below illustrates this.

That is very similar to the way Copilot works in VSCode. And I used this feature to create a Notebook where I wrote almost no code myself. I simply wrote the prompts and accepted the suggestions. Below, you can see the result of a code suggestion from a blank cell. NBI creates the code and adds the prompt as a comment.

(Note the circled icon in a Notebook cell where you can click instead of hitting <ctrl-G> to bring up the prompt.)

The step-by-step approach to creating a new Notebook

The Notebook that I created analyses climate change data since 1850. The raw data shows how temperatures have changed over that period and gives figures for each month, in each year. I wanted to see how the summer temperatures had changed over time so needed to reformat the data and average the summer months into a single figure. Then I wanted to draw a bar chart of the result.

Instead of writing the code to do this, I let NBI do all the work.

Well, most of it. I wrote a line to read in the raw data.

import pandas as pd
df = pd.read_csv('HadCRUT-global-monthly.csv')

This gave me a dataframe like this:

You can see that the date combines the year and the month but I want to separate those and create a table of temperatures in monthly columns with a row for each year. So I opened a new cell and prompted NBI to create new columns for year and month. You can see the result below — the comment is the prompt that I gave.

# nbi-prompt: in df there is a column 'Time' that contains a date in 
# YYYY-MM format. Split this into two columns, one for YYYY and one for MM 
df['Year'] = df['Time'].str.split('-').str[0]
df['Month'] = df['Time'].str.split('-').str[1]
df.head()

This gave me the extra columns I needed so the next thing was to reformat the table.

# nbi-prompt: create a new dataframe that uses the data from df. 
# The new dataframe should have columns for each of the months and rows for each year. 
#The cells should contain the Anomaly (deg C) values
new_df = df.pivot(index='Year', columns='Month', values='Anomaly (deg C)')
new_df.head()

And then I wanted to create the ‘summer’ column.

# nbi-prompt: Create a new column in new_df called summer that is the average of the months 06, 07 and 08
new_df['summer'] = new_df[['06', '07', '08']].mean(axis=1)
new_df.head()

Remember, all of this code was written by NBI — I just wrote the prompts — and I ended up with the table that I wanted

Now I wanted a chart and that was easily created with the following prompt:

Draw a matplotlib bar chart of 'summer' over 'Year'

This gave me a suitable chart but all in one colour, so I selected the code in the cell hit <Ctrl-G> and asked it to make the positive bars red and the negative ones blue. NBI came up with a diff pane.

You can see the original code was adequate and the new code added the colours that I wanted. But the x-axis is a bit of a mess so I went through the same procedure to ask it to rotate the labels by 45 deg. and to only mark every 10th year. The resulting chart was fine.

That’s good but how about using Plotly? Again, I selected the codel and went through a series of prompts. The first asked NBI to convert to Plotly. That gave a standard chart but I then asked it to use the ‘plotly_white’ template and the ‘inferno’ colour scheme. The resulting code was this:

colors = px.colors.sequential.Inferno
fig = px.bar(x=new_df.index, y=new_df['summer'], color=new_df['summer'], color_continuous_scale=colors)
fig.update_layout(xaxis_tickangle=45, xaxis_dtick=10)
fig.update_xaxes(title='Year')
fig.update_yaxes(title='Summer Anomaly (deg C)')
fig.update_layout(title='Summer Anomaly over Year')
fig.update_layout(template='plotly_white', width=800, height=600)

And that is pretty much the result that I wanted. I’ve skipped a few steps here but you can see the detailed version in the Notebook in the GitHub repo.

Share

Creating a complete Notebook

As in VSCode NBI has a chat interface. You can use this as a way of generating code that you can insert into a Notebook and it is here that you can use the /newNotebook and /newPythonFile commands. These last two allow you to create entire Notebooks or Python programs.

To demonstrate this I concatenated prompts from the previous exercise to create a complete Notebook in one go.

The screenshot below shows the chat interface after I have prompted NBI to create a new Notebook.

The input for the chat interface is at the bottom and the textual response above it. The code response is in the new Notebook you see on the right of the window. If you zoom in on the image you can see the complete prompt (otherwise look in the repo) and the Notebook is a complete, nicely formatted and documented version of the previous code that produces the same Plotly chart.

Creating a complete Streamlit app

Creating a complete app was as simple as creating the last Notebook. Instead of /newNotebook the prompt is newPythonFile and I gave it the following prompt:

Create a Python Streamlit app in the file weather.py that does the following:
Create a new pandas dataframe with the following data:'HadCRUT-global-monthly.csv'
The dataframe contains the following columns: 'Time', 'Anomaly (deg C)', 'Uncertainty (deg C)', 'Lower bound (deg C)', 'Upper bound (deg C)'.
The column 'Time' that contains a date in YYYY-MM format. Split this into two columns, one for years and one for months. Convert the numbers to names, e.g. 01 becomes January, 02 becomes February, etc.
Create a new dataframe that uses the data from the current one. 
The new dataframe should have columns for each of the months and rows for each year. 
The cells should contain the 'Anomaly (deg C)' values.
Create a new column in the new dataframe for the summer temperature that is the average of the months June, July and August).
Draw a Plotly Express bar chart of summer temperatures over each year.
Use the plotly_white template, change the size to 800 x 600 and use the 'inferno' color scheme.
Turn the x-axis labels 45 degrees and only show a label for every 10th year.
Add a suitable Streamlit title and subtitle for the page.

NBI went on to produce a complete app in a Python file that is the equivalent of the Notebook.

Conclusion

This little experiment was done with v1.0.0; with this version, there was no <Ctrl-G> command when editing Python files. I suggested to Mehmet that this would be a good idea. Hey, Presto! There is now a v1.1.0 with that feature!

I’m seriously thinking that Jupyter Lab with NBI might replace my usual VSCode environment. I often find that I prototype stuff in a Notebook, first, and then move on to a Python program when I’m happy with the result. Jupyer Lab lets you do this as easily as VSCode and it supports Markdown, has a terminal and Git integration (via an extension) — what’s not to like?

Notes

The data used in this exercise was downloaded from Met Office Hadley Centre observations datasets and is used under the UK Open Government Licence for Public Sector Information v3.0

RAG apps don't need to be difficult to develop. Pinecone Assistant abstracts away all of the complexity and lets you interrogate your documents via a console or a Python API.

Subscribe now

In the following article, I've explored a copy of the US Constitution and documentation from Wikipedia to simulate the sort of queries that a student or teacher might make. I also wrote a Streamlit application to answer questions about the Constitution, generate a study plan, and create multiple-choice questions (formatted as JSON).

The combination of Streamlit and Pinecone Assistant made writing the app remarkably easy.

Read the article here (Medium members) or here (my website)

Share