So, before I review Markov chains and their important properties, I’m going to sketch out a landscape to fit those abstractions into.

Specifically: Why are Markov chains and their properties prerequisite topics in machine learning mathematics? What do they help us achieve?

(Note: If you’ve forgotten lots of previously-studied maths, like I have, check out the exercises at the end of this post. They should help refresh your memory, starting from probability basics.)

1. Orientation

We are building a machine learning model. We have a set of training data. We want to build a model that will make good predictions about data points that we haven’t seen yet.

We need to know what the model’s parameters (θ, or “theta”) should be.

Sometimes we are happy with a single best θ, i.e. a model that outputs one good answer.

In these situations, finding θ is an optimisation problem.

But, at other times, we prefer a spread of “good candidates” for the parameters. We don’t want to find a single best θ, because that is just one prediction without any information about how uncertain it is.

Instead, we want many plausible thetas, providing a range of predictions for good model parameters, with a sense of how good of a prediction each θ is.

…having the full posterior can be very useful for a downstream task.

Deisenroth et al, ch. 6.3

Searching for one optimal theta vs a sample of candidate thetas.

2. A Bayesian approach

We want a set of good candidates for θ. So we can be fancy and take a Bayesian approach, by considering a probability distribution over theta.

With that, and the training data, we can define the probability P(θ|data), known as the posterior distribution.

Some values of θ are more likely to be good (higher posterior probability), while others are less likely (lower posterior probability).

And we’d like to gather a collection of likely-good values by sampling from this posterior distribution, once found.

Finding this posterior distribution is known as Bayesian inference.

We want to sample from the posterior distribution.

3. Some Bayesian problems

So we need to understand the posterior distribution. However, we cannot compute it directly.

We might have an intractable integral to deal with:

More specifically, if we do not choose a conjugate prior on the parameters (Section 6.6.1), the integrals in (8.22) and (8.23) are not analytically tractable, and we cannot compute the posterior, the predictions, or the marginal likelihood in closed form.

Deisenroth et al, ch. 8.4

In other words:

But for many other distributions that are of scientific interest, the normalizing constant in the PDF or PMF is unknown, and beyond the reach of the fastest available computers and the fanciest available mathematical techniques.

Blitzstein and Hwang, ch. 12

That is to say, even though the denominator on the right-hand side of this statement of Bayes’ theorem looks very inoffensive to me:

…it actually causes some significant problems here. We cannot find our P(θ|data)!

4. A Markov chain appears

It turns out that we can build a Markov chain with a stationary distribution that is the posterior.

(The chain must have the right properties: irreducibility, and aperiodicity.)

When we run this chain and collect the states it visits (a process called Markov chain Monte Carlo, or MCMC), we obtain a collection of points.

These points are samples from the posterior that we care about. They will be mostly good candidates for θ.

Here, we can use stochastic approximations, such as Markov chain Monte Carlo (MCMC) (Gilks et al., 1996)…

Deisenroth et al, ch. 8.4

And:

The basic idea is to build your own Markov chain so that the distribution of interest is the stationary distribution of the chain.

Blitzstein and Hwang, ch. 12

When we eventually use our model to make predictions, the model will use all of the candidate θ values each time, outputting each prediction as an average over them.

Good θ values are more common in the collection of candidates, so they help push that average in the right direction.

And the spread of the candidate θ values is our (un)certainty score.

[The model’s prediction is] an average over all plausible parameter values θ, where the plausibility is encapsulated by the parameter distribution p(θ).

Deisenroth et al, ch 8.4.2

Markov chain Monte Carlo allows us to gather a collection of samples from an approximation of the posterior distribution that we care about, P(θ|data).

This leaves me with more questions than answers, but that’s the goal.

With this rough sketch of the landscape, it’s easier to see what I need to look at next.

(It’ll be stationary distributions of Markov chains, and the mysterious but important properties a chain must have for MCMC.)

The following exercises should help refresh the memory of anyone who (like me) has forgotten lots of the maths they previously studied.

I was aware that straight line distance, and distance along a curve, are two importantly different things.

After all, if I am at the North Pole, and you are at the South Pole, and I am hosting a garden party, then we must be realistic about your travel time.

(Or, in machine learning terms, if my data points lie on a manifold, then I must calculate the distances between them appropriately.)

But, even if we’re traversing a completely flat plane, we can still have an argument about how much distance we’ve covered.

Choosing how we measure distance

In statistical modelling, the distance between two points often represents the error between a model’s prediction and the actual value.

Our entire goal may be to minimise this distance, as this means minimising the error in the model.

Since we can’t minimise a hand wave, we need a precise distance measurement.

And that means we need to make a choice. For example:

1. Would we like to consider the length of the straight line between two points?

2. Would we prefer to count the total number of steps in all directions required to get from point A to point B?

3. Would we prefer to think about only the direction that requires the most steps?

There are some interesting implications from these choices.

Option 1 involves Pythagoras’ theorem, and thus squaring. If you square 2, it only goes to 4. But if you square 22, it goes all the way to 484. So, with this choice, we must remember that exponentiation gets wild, and this will impact the distance measure.

So, with Option 1, our model’s error might seem much worse than if we did a simpler count of steps, e.g. Option 2 or 3.

Note that these options are more formally known as the Euclidean norm, the Manhattan (or taxicab) norm, and the Chebyshev (or chessboard) norm.

The two-step process we must follow

(Pun intended, apologies.)

To compute the distance between two points, we:

1. Find the differences between each component of the two points, and thus obtain the difference vector, v

2. Choose a norm, such as one of those mentioned in the previous section, that will turn that difference vector into our final distance value

So we see that distances are norms of differences.

The following table is a bit of a notation-dump, but examples will follow, and I’ll add exercises (with solutions) at the end for anyone who learns best by doing (like me).

Here’s an example of each of these three choices giving us slightly different results.

Organising our options into a family

This article’s title is “Study with me: The Minkowski distance family” because this was my favourite thing to discover this weekend.

Minkowski essentially gives us a machine that outputs the norms we’ve been looking at here by simply adjusting its parameter, p.

The Minkowski distance family is defined as:

Setting p = 1 gives us the Manhattan norm:

Setting p = 2 gives us the Euclidean norm:

And p = ∞ gives us the Chebyshev norm:

(Notice that we factor out the largest component of the distance vector to ensure the rest become zero as p approaches infinity.)

Isn’t that lovely?

My beginner’s introduction last week actually surprised me multiple times.

I will share exactly why.

1. The practical surprises

Indexed sets give us ordering, and permit repeated values.

Imagine we want to use social media posts’ like counts to represent how engaging each post is.

Post A has 2 likes, Post B has 1 like, and Post C has 2 likes.

If we want these like counts to be easy to iterate through, we might collect them into a set. But sets don’t keep track of duplicates, or ordering.

We try to store 2, 1, and 2, but end up with {1, 2}.

The solution to this is intuitive. We need to mark each value in our collection with something that represents its position.

If I annotate my collection of post like counts to do this, it might look like “first: 2”, “second: 1”, “third: 2”.

How can we use plain sets to achieve this?

This is quite a tricky question, IMO, but luckily Kuratowski already figured it out.

We can use plain sets to make ordered pairs: {{1}, {1, 2}}, {{2}, {2, 1}}, {{3}, {3, 2}}

(I’ll include an exercise at the end that involves looking more closely at this.)

And we can use a neater, and probably very familiar, notation for these ordered pairs: (1, 2), (2, 1), (3, 2)

Our set of post like counts now looks like this: {(1, 2), (2, 1), (3, 2)}

These ordered pairs allow us to retain repeated post like counts, because position values will not be repeated.

And, although the set doesn’t care much about ordering, the position value in each pair in the set still provides the information we need.

After all, ordering isn’t so much about arranging things nicely, but more about understanding where the elements of our set sit relative to one another, and about being able to locate one easily.

(At least, these things matter to someone working with computers, which a person working on a machine learning model probably is.)

So now we have an indexed set.

And to pick apart exactly what we have, it is:

1. An index set, I, containing the position values that we’re using (in our case, I = {1, 2, 3})

2. A mapping, x, from each i in the index set I to the values we care about (in our case, i = 1 maps to 2, i = 2 maps to 1, and i = 3 maps to 2)

Those are the tools that allow us to label each element in our set of post like counts with a position value.

There is some tidy notation, too: X = (x_i)_{i ∈ I}

And we can also write out X as (x₁, x₂, …, x_n).

This probably looks very familiar, as it’s used for things like listing the components in a vector, or the set of vectors in a spanning set, or an ordered list of objects for programming purposes (e.g. Python’s tuples).

But as someone who has used this kind of structure countless times, without ever questioning how it was itself constructed, this was all a surprising realisation.

And it wasn’t the only one!

2. The unintuitive and thought-provoking surprises

The next unexpectedly interesting question I ran into was, “What happens when we try to find unions and intersections of families of indexed sets?”

(To be clear, now the sets in the families are indexed, not the values in the sets.)

For a beginner like me, this means first considering small indexed families, and thinking about what they look like.

And then asking which elements are in at least one of them, or in all of them:

If the family gets very large, there’s a bit more reasoning required in deciding what is in their intersection.

What do you think the intersection of A_i = {n ∈ ℕ : n ≥ i} will be?

To increase the challenge a little, we can consider a family where each set is an open interval:

Here, it’s as though we’re asking, “What number is smaller than 1/i for all i in ℝ⁺?”, which is a question I simply did not expect when I first started looking at indexed sets last week.

So there you have it: All the ways I was wrong about indexed sets being dull. I’m glad to have been so mistaken.

Some exercises below, for anyone who might find them useful.

Exercises

1. Which of these two data sets requires order and disambiguation of duplicate values, and why?

   1. A week’s daily step counts

   2. The unique hashtags across a social media user’s posts

2. Take a look at the Kuratowski definition of ordered pairs and write out some examples, to show yourself how (1,2) ≠ (2,1)

3. Consider the indexed family A_i​ = {i, i + 2}, i ∈ {1,2,3,4}. Enumerate the family by writing out A₁​, A₂​, A₃​, A₄​.

There are more exercises available for subscribers.

Subscribe now

To be honest, I generally don’t understand anything.

But I have discovered a workaround.

If I skip trying to understand, and instead only investigate why I don’t understand, then… some understanding occurs as a byproduct.

(Although it’s important not to get overexcited and look too closely at it, because it evades inspection and may disappear entirely.)

Here are a few examples.

1. Implicit Differentiation

I would apply this technique to problems that I recognised as requiring it, but fail to do so whenever the problem was disguised in some way.

When this happens, it means I don’t understand the fundamental usefulness of the technique.

In this kind of situation, I’m often rescued by one or more posts on math.stackexchange.com, where brave people ask questions and knowledgeable ones kindly reply.

It’s a good place to find “human” explanations, where informal language and personal choice of emphasis give the information dimensionality and context.

(Example: In person, I might say, “This is the part to notice!” with some gesticulation, while a textbook typically lists facts neutrally as though meant to be ignored.)

This thread was helpful, particularly because one comment linked out to a resource that human-ly communicates the significance of not being able to fully separate y from x in some equations.

Notice the capitalisation of EXPLICITLY and IMPLICITLY. Now I understand what is important.

This resource also reminds me that some graphs look like this:

Here, the answer to “What is y when x is 1?” is not clear. (See: the vertical line test.)

So: Explicit differentiation finds an expression for the slope of a function for any value of x, while implicit differentiation gives us an expression for a particular point’s slope.

(This is why the result may contain both coordinates.)

So now I know what to look for in varied problems that require this technique.

2. The Rank-Nullity Theorem

This theorem is very simple to look at:

But the implications have not been immediately obvious to me.

In “Mathematics for Machine Learning” (2020), Deisenroth et al mention that this theorem is also referred to as “the fundamental theorem of linear mappings”.

And they list three of its consequences, including that it can tell us when a system of linear equations has infinitely many solutions, or when a mapping is bijective.

(I recommend reading chapter 2.7.3, as they include helpful informal statements alongside the formal ones, plus a diagram.)

However, the higher-order consequences of this theorem are beyond my current linear algebra knowledge.

I can appreciate that, if a mapping is not bijective, then it effectively causes us to lose information. Perhaps this is connected to compression, in some way. Although I know this loss is not always a bad thing, such as when using principal component analysis (PCA) to simplify problems by discarding carefully-chosen, less-important dimensions.

And it’s always useful to know when a system of equations has infinitely many solutions.

It might also be significant that we know exactly which vectors we will “lose” through the mapping, as this set is simply the kernel.

But this is something I am not yet sure about, so I will be reading more.

3. Probability Distributions

This section’s title should be even broader. It could even be “Statistics Fundamentals”.

My issue here is knowledge organisation. I’ve studied these topics without building a mental map of how they connect to each other, or a timeline of when and why they were new and exciting.

These structures give ideas a landscape to populate. They get homes, with signposts. Their characteristics make sense because of the environment they inhabit.

But currently, concepts like “z-score”, “Poisson distribution”, and “density function” hover in random places, so I can’t find them when I need them.

Fortunately, many great versions of these topic maps and timelines have already been created by others. No need for me to reinvent the wheel. I’ll search for my favourites, and make flashcards from them, to most efficiently resolve this issue.

There are countless more things that I don’t yet understand well, but I’ll pause here.

If you have favourite resources for any of these topics, feel free to recommend them.

I think this is something that does naturally become clear over time, with further studying. But I find it difficult to retain information when I don’t have much context for it, and feel a little frustrated about this question at the moment.

So it’s time for some side-tracking into how the plane equation forms I’m reviewing fit into the bigger picture, where the bigger picture (for me) is machine learning mathematics.

So my question is actually: Why are there so many forms of a plane equation, and when do we use each form in machine learning?

I’ll be looking at these forms:

(Note that these forms are listed in two columns, with each column containing two very closely-related forms. The left-hand forms might be generally referred to as “parametric representations”, and the right-hand ones as “implicit representations”.)

And I’ll be drawing inspiration from this paper, most of which is far beyond my current level, but which mentions something quite incredibly useful for me in its introduction:

To give a simple example, the parametric description [of a rational algebraic curve or surface] is better for drawing a surface, as it allows to rapidly generate points on the surface, which can then be interpolated, whereas an implicit representation is better adapted for testing if a given point lies on the surface, since one only needs to check whether the point verifies the algebraic condition that defines the surface.

Now, I don’t know whether this can exactly be applied to the ways of describing two-dimensional planes that I’m reviewing currently.

But Wikipedia tells me that “a plane is a surface”, so the following will be my simple interpretation of the ideas in the quoted text, applied to the planes I’m reviewing currently, sitting in three-dimensional space.

I believe this will provide some of the context I am looking for, even if imperfect and needing further iteration in future.

Generating Planes

Sometimes, we want to generate, or navigate around, a plane. The parametric and vector forms of a plane equation are the most useful for that.

Recalling the vector form of a line, one point (with position vector) plus one more vector allows us to reach and traverse a line, or one-dimensional space:

And the intuition for the vector form of a plane equation follows, where we add one more vector to be able to navigate along an additional dimension, i.e. a plane:

In machine learning, principal component analysis (PCA) provides an example of where we might imagine ourselves using vectors to discover and navigate a plane.

In PCA, the goal is to reduce the dimensionality of a data set. We find new directions through the data, ranked by how much variance lies along each, and keep only the most important ones.

This involves finding the data’s mean, and centring the data there. And then finding the eigenvectors of that centred data’s covariance matrix.

The eigenvectors are the principal components (directions of maximal variance). We’ll keep the most important ones of those.

In this example of data in a three-dimensional space, the covariance matrix gives us three eigenvectors. We keep the top two, and they span the two-dimensional plane we’re navigating.

Not all of the data points sit on the plane, and we see their projections onto the plane appear at the end.

Those projections represent the reconstruction error, which is the information we lost by discarding that dimension.

Separating Planes

Sometimes we care more about how a plane divides a space, perhaps acting as a boundary between the points on either side.

And in these times, we might use the Cartesian or normal forms of a plane equation, perhaps as a function that takes a point, x, and returns how far that point sits from the plane, measured along the normal direction.

This makes it easy to see that the plane or boundary itself is comprised of the points where this function outputs zero.

It’s also easy to see when points sit on one side of the plane, or the other, and how far from the plane they are.

In machine learning, this is (in my opinion) a useful visualisation for the decision boundary created by a binary classification model.

During training, the model will have to keep checking whether it’s figured out the best decision boundary to correctly classify data points, by seeing whether the boundary divides the points in the data space correctly.

If there’s too much error (points on the wrong side) when checking against real data with known classes, then the model keeps adjusting the boundary until it divides the data points more correctly.

I’m still learning Manim, so those animations took far too much time (although they might not look like it!).

However, I love creating maths visuals, whether they particularly need to exist or not, so I’ll be back next week with more.

More notation that I should be able to parse but have completely forgotten how.

The common thread between tensor contraction and double summation notation, and the initial trouble I had with them, might be the multi-dimensionality hiding behind such a small and innocuous-seeming set of symbols.

So let’s take a second to look at what this notation is conveying so efficiently.

The easiest example

For anyone familiar with programming fundamentals, double summation can be thought of as a nested loop:

total = 0
for i in 1..m:              # the outer loop
    for j in 1..n:          # the inner loop
        total += a[i][j]

We can also visualise a nice and tidy grid:

The important idea is that we evaluate the inner sum first, initially fixing the outer index at its first value.

Then we increment the outer index, and evaluate the inner sum again.

And so on.

Quick Practice (1)

Calculate the value of:

Solution at end of article.

A slightly less easy example

Here we have a product of two terms in the sum, where one term (b_i) only varies across one index, but the other (a_ij) varies across both.

We can understand this one using the same grid as before, keeping in mind that each a_ij is now multiplied by the relevant b_i.

Quick Practice (2)

Calculate the value of:

Solution at end of article.

An interesting example

This is quite a fun one to apply the grid visualisation to, and best (re-)discovered for yourself if it’s been a while since you last saw a double summation like this.

Why are we looking at double summations?

A very good question. My understanding at present:

• The course I’m following (see “References and resources” below) likely uses double summation as practice for dealing with a more general increase in dimension on sums (and their continuous counterpart, integrals), since machine learning often involves working with very high-dimensional data, and with joint probability distributions

• Bishop’s “Pattern Recognition and Machine Learning” textbook includes some double (or higher!) summation notation in several chapters: “8.4. Inference in Graphical Models”, “9. Mixture Models and EM”, and “13.2. Hidden Markov Models”, so we clearly must be well-prepared for these encounters

To be honest, my study goal for this summer is to not be stressed out about studying. So I’m not sure yet what next week’s post will be about. But it probably won’t be long before I’m tripped up by some spooky notation again, and I’ll write it all up when it happens.

I love them.

They make it very easy to see and appreciate some of the most well-known Boolean functions, NAND (“not AND”), NOR (“not OR”), and XOR (“exclusive OR”):

If you’re looking to pick a favourite, you might consider NAND, which offers functional completeness.

This means that we can construct all other Boolean operations from NAND. For example, “NOT x”:

Another example, “AND”:

But, in my opinion, you should also consider XOR for top spot.

In computer science, there are “XOR tricks” that can be used to solve certain problems.

For example, when working with binary numbers, XOR can be used to “flip” bits from zero to one, or from one to zero:

x    = 1010   # This binary number represents 10
mask = 0110   # This binary number represents 6, but it will also allow us 
              # to flip bits 1 and 2 of another binary number

x XOR mask = 1100   # This binary number represents 12 but, more importantly,
                    # we see the expected result of XOR's "flip" of bits 1 and 2

Interestingly, XOR can then be used again to obtain the unmodified binary number:

(x XOR mask) XOR mask = 1010

That is to say, XOR is its own inverse, and this is handy in certain situations. For example, if we want to toggle certain configuration or permissions flags on or off (and then back again), we can do this efficiently with XOR.

But in machine learning, XOR managed to create a problem. One that caused a significant slowing in the rate of research into neural networks when it was written about by Minsky and Papert in their 1969 book, “Perceptrons”.

The problem is that XOR is not linearly separable. That is to say, we can’t draw a single straight line that separates the two output classes.

And a single-layer perceptron (a type of neural network) can only learn linearly separable functions. So it cannot represent XOR.

However, in the 80s the pace of research picked up again. Multi-layer networks (with a hidden layer) trained via backpropagation could solve the XOR problem, because their multiple layers allowed them to divide the input space with more than one boundary.

The following are some animations I made to share my currently-simple understanding of the XOR problem and its solution. (See “Resources” at the end of this post for a video lecture I found with much more detail, if you’re curious.)

We can graph XOR on a two-dimensional plane, creating a unit square:

If we try to separate the points where A XOR B = 1 with a straight line, we simply cannot:

And this is a problem because, if we are building a binary classification model, we need decision boundaries that divide the space into classification regions.

(Very simply: If a data point sits in one region, then it’s in one class. If a data point sits in the other region, it is in the other class. Thus, the data point has been classified.)

The solution is, essentially, to transform the input space itself. Once a hidden layer has done its work, we can imagine the data points sitting in a warped space that positions the 1s in such a way that they can then be linearly separated from the 0s:

Bishop’s book (referenced below) has a good short summary of the history here, and you can of course read the original works by Rosenblatt, and Minksy. Let me know if you do!

I’m not sure exactly what I’ll write about next week.

The course I’m following (included in the resource list below) interleaves topics. Whenever something seems far more mysterious than it should, I read more widely and summarise my findings here.

(One known study activity next week: Some visuals to improve my ability to parse double sum notation! I’ll probably just post them as a note, so feel free to follow if you’d like to see them.)

(I started undergrad without the kind of preparation that I would now recommend to anyone else. Hence all this review that I’m doing.)

And now I’ve reached logic fundamentals in the online course I’m following (“Mathematics for Machine Learning” on MathAcademy), and would like to avoid one of my prior learning mistakes: Not putting abstract information into concrete context.

So here I’ll try a few ways of creating initial connections between some logic basics and some ML concepts, to help with retention and intuition development.

Classifier decision regions and predicate sets

To get started with logic, we need to learn what statements and predicates are.

A mathematical statement is an expression that is either true or false:

And a predicate is a statement with a variable, so its truth depends on the value taken by the variable:

We can apply logical connectives, AND, OR and NOT, to statements and predicates in the way we’d expect, too.

To put these abstract ideas into a (slightly) practical context, we can consider a classification scenario, where our goal is to label images according to the kinds of animals in the image.

Our predicates might be P(x): “x contains a cat” and Q(x): “x contains a dog”, where the variable references a particular image we are labelling.

The truth sets {x: P(x)} and {x: Q(x)} then become decision regions of our classifier.

And we can visualise the logical connectives, such as AND and OR, in the familiar Venn diagram-like way.

Decision trees with logical leaves

Truth tables are another early logic fundamental. And they are already simple, and visual, which I appreciate.

In machine learning, decision trees can be used for classification tasks, and a decision tree can be interpreted as a truth table with a conjunction (“AND”) column.

(Hopefully that isn’t too inaccurate, biologically-speaking.)

Decision trees might be built recursively, followed by pruning, where ‘less useful’ nodes are removed to obtain a subtree.

Although the definition of ‘less useful’ is rather more complex in practice, I’ll stick to a very simple example of pruning a tree to remove redundant paths.

Shown alongside a truth table, this illustrates another fundamental: Logical equivalence.

Two statements are logically equivalent if they have identical truth values in their respective truth tables (i.e. two matching columns).

And this is something we can also see visually in a decision tree that contains some redundant paths:

I wanted to also review Boolean functions and logical operations this week, perhaps in connection to perceptrons, but that’ll have to be next week’s post.

Of course, in a physics or pure mathematics context, tensors are a deeper topic than in ML. This post will treat tensors as multidimensional arrays used in computational work, but I do recommend further study of tensors in other contexts to anyone curious.

Back to my tensor contraction notation jumpscare: By resetting my expectations to just figuring out some ML tensor basics last week, and patiently revisiting those ideas a few times, I was able to feel a sense of progression. And since I study because I like to progress my understanding, that was the ideal result.

In particular, I’ve had time to just get used to adding an extra axis to a matrix.

(Or many more than one axis, of course, but if you know how to visualise that then do share in the comments.)

Try it yourself:

1. For each of the following objects, what is its order (i.e. how many axes does it have)?

   1. The scalar a

   2. The vector [x, y z]

   3. The matrix [[a, b, c], [d, e, f], [g, h, i]]

2. If we represent an order 3 tensor, T_ijk, as follows, and use 1-based indexing, what is T₂₃₂?

Answers at end of article.

Tensor permuting and reshaping

In machine learning, we might be working with tensors as multidimensional arrays, and we might sometimes need to mould one into a more convenient shape.

To date, most convolutional neural network architectures output predictions by flattening 3rd-order activation tensors, and applying fully-connected output layers.
—Kossaifi et al, “Tensor Regression Networks”, July 2017

Managing computation cost is a likely reason for this:

Although the computational cost (measured in number of scalar multiplications) is the same both ways, it is usually preferable to recast [binary tensor contraction] as multiplication as modern hardware performs vectorized operations much faster than when using the equivalent FOR loop.
—https://www.tensors.net/tutorial-1

This recasting can involve permuting a tensor, as well as reshaping it.

Permuting a tensor reorders its dimensions by swapping its indices. This doesn’t change the tensor’s entries, just the location they can be retrieved from.

Swapping the indices of an order 2 tensor, visualised as a 3x3 matrix, results in the familiar transpose:

(As a sidenote here, Deisenroth (2020) specifies: “When we ‘transpose’ a tensor, we mean swapping the first two dimensions”.)

If we visualise an order 3 tensor as a stack of matrices, and choose to swap the index that tracks the ‘stack layer’, then it may look like this:

One PyTorch tutorial gives an example scenario in which someone might permute a tensor:

Imagine you have image data stored as (Channels, Height, Width) but need it in (Height, Width, Channels) format for a specific library or visualization. [PyTorch’s function] permute() is the tool for this.

Try it yourself:

Given the same order 3 tensor representation as before, T_ijk:

If we permute T_ijk by swapping its j and k axes to obtain T′_ikj, what entry do we find at T′₂₁₂?

Answer at end of article.

Now, onto reshaping, where we might combine any number of tensor indices into a single, larger index, or separate a larger index into smaller ones.

Again, this is easily visualised with matrices:

But moving up an order isn’t too bad:

The same PyTorch tutorial suggests that “[tensor reshaping is] highly useful for tasks like flattening a multi-dimensional tensor before passing it to a linear layer”.

And Deisenroth briefly describes how we can compute the gradient of a matrix either by finding the partial derivatives and collating them into the order 3 Jacobian tensor, or by flattening the matrix into a vector to compute the Jacobian matrix, and then reshaping the result into the Jacobian tensor.

(Here’s an article with more about the Jacobian and its uses in ML, if useful.)

Try it yourself:

Given the same order 3 tensor representation as before, T_ijk:

If we reshape T_ijk by flattening the j and k axes into l to obtain T′_il, what number is found at T′₂₁?

Answer at end of article.

I’ve barely begun to scratch the surface of tensors and their operations, but the course I’m following doesn’t dwell on them much. Over the next few weeks, I’ll be sticking to its roadmap.

This will mean returning to reviewing the many topics I’ve half forgotten across linear algebra, calculus and statistics, which I’m finding so fun to do without the stress and deadlines of a formal course (for now).

It’s funny because, although I love studying mathematics, and have spent a lot of time unlearning a fear of it, I still sometimes get completely in my own way.

And perhaps my brain also sometimes demands reassurance that whatever I’m looking at has a utility.

These days, I know the answer (for me) is to pause to figure out a safe entry-point. Ideally the simplest possible one. And then to look for a real-world usage example.

This post is an example of that, where I reduce my ambition (for now) to just understanding something about what a tensor is, deciphering that spooky notation, and seeing where these ideas might be applied in practice.

Tensors

In a machine learning context, tensors are multidimensional arrays.

Deisenroth provides a nice visual:

So, a tensor extends a matrix along a new axis. And, with every axis, the order of the tensor increments. Starting with a scalar (order 0), up to a matrix (order 2), and so on.

Since I’m learning the Manim library at the moment, here’s an example of an order 3 tensor visualised as four stacked matrices:

Tensor Contraction

Tensor contraction is described as a generalisation of the trace of a matrix, and as an extension of matrix multiplication.

Along with the notation, this left me with more of a puzzled feeling than any initial intuition, which I decided to address with a practical walkthrough.

Starting with a reminder that the trace of a matrix is the sum of the matrix’s diagonal entries:

When we find the trace of a matrix, we reduce the matrix to a scalar.

This is a reduction from order 2 to order 0, if we consider the matrix a tensor.

In simple index notation, the trace is:

Now, to understand the connection between a matrix trace and tensor contraction, which Deisenroth also comments on briefly:

I appreciated, but did not immediately gain anything from, Deisenroth’s example, and so worked through my own example of the trace of a third-order tensor yielding a first-order tensor (i.e. a vector):

In simple index notation, this trace is:

Next, we consider matrix multiplication.

If we write matrix multiplication in simple index notation, we have:

To quote Wikipedia: “Notice how the index j, in between, disappears; this is the essence of tensor contraction.”

Here’s an example of the contraction of two third-order tensors, where we contract two index pairs so that the result is a second-order tensor (a matrix), for easier visualisation:

In this example, indices j and k vanish.

So it seems that tensor contraction gives us a way to combine tensors without increasing the order each time (as an outer product would), to avoid rapidly-increasing computation cost.

Perhaps the discarded dimensions can also be those that don’t matter so much to the model, when using tensor contraction in a machine learning context.

Searching around for an example of tensor contraction being applied to a machine learning problem, I found this paper, which mentions “combin[ing] tensor regression with tensor contraction to further increase efficiency” of convolutional neural network output.

And that’s all for me this week. Daunting notation: clarified!

Next week, I’ll take a look at another tensor operation from the tensors.net tutorial, or perhaps at tensor broadcasting.

It’s also a satisfying topic to study, as organising objects into collections that have useful properties makes immediate sense to anyone who’s ever sorted out a clutter of items at home.

Set theory is quite an early topic in mathematics, so I needed to refresh my memory and put the ideas into a machine learning context to suit my current learning path.

The following are a few examples, in case they can be useful to anyone else studying at a similar level (“thinking back to undergrad”).

1. Cartesian product of sets

If A and B are sets, the Cartesian product A × B consists of all ordered pairs (a, b) where a is a member of A and b is a member of B.

Importantly, the sets A and B do not need to contain objects of the same type.

Machine learning example

In the same way that the Cartesian product ℝ × ℝ gives us all coordinate points in two dimensional space, the Cartesian product of a dataset’s features can give the (probably quite high-dimensional) ambient space that the data points will be found in.

We can then think geometrically about the distances between data points, and how densely they are clustered, as an indication of similarity.

For example, in the K-Nearest Neighbours algorithm, the distance metric might simply be the straight-line/Euclidean distance between two points.

And, if our dataset’s features are quite different (e.g. “age” might be any real number in the interval [0,123], while “is subscribed” might be one element from the set {0,1}), then we’d have no trouble defining the feature space using the Cartesian product, as it is quite agnostic about the sets it operates on.

2. Describing planar regions with set-builder notation

Set-builder notation allows us to define a set by applying a rule or some conditions to a base set.

If our base set is all points in ℝ², then our condition could restrict our set to only the points on one side of a line (or inside a circle).

Machine learning example

When classifying data with support vector machines, we’re looking for a linear separator between two classes.

As we’ll be working in a feature space with any number of dimensions, the boundary between the two classes will be a hyperplane.

We can define that hyperplane using set-builder notation as follows:

…where f(x) = ⟨w, x⟩ + b.

3. Indicator functions

An indicator function maps values from a set to either 1 or 0, to “indicate” whether an element is a member of a subset.

Machine learning example

This example comes from the foundational principles of machine learning in the first chapter of Vapnik’s book, “The Nature of Statistical Learning”.

Here, he presents the problem of risk minimisation, where we want to minimise the “risk functional”, i.e. the expected loss, to find the best approximation of an unknown probability distribution.

Vapnik lists three main approaches to defining the loss function, with one being pattern recognition, where he defines a loss function that is an indicator function:

In fact, here we have an indicator function being scored by another indicator function, just in case we thought such a simple idea might not be important.

Although I didn’t get much time to study last week, putting some set theory fundamentals into a machine learning context was an enjoyable use of that time. 10/10, would recommend.

If you have a favourite idea in set theory, please do comment to share it.

So, while I’m studying independently before resuming formal studies (currently following Math Academy’s Mathematics for Machine Learning course, and planning to do an MSc. course in future), I’ve been taking some time each week to look at an ML technique that involves the maths I’m reviewing.

For this reason, as I review certain probability distributions on Math Academy, I’ve started to look at binary logistic regression algorithms, where a model will generate classification label predictions that are samples from a Bernoulli distribution.

Here’s a simple flow diagram I made to see the steps in a logistic regression algorithm applied to an email spam classification scenario:

Along the way, I discovered that the ‘classic’ gradient descent formula I’d seen in many places (example) is not the default optimiser in scikit-learn’s LogisticRegression class. (The optimiser is specified by the `solver` parameter.)

In fact, the default solver used by that class is an algorithm called L-BFGS.

Once I’d recovered from my surprise that we wouldn’t simply calculate x_{n+1} = x_n - α∇f(x_n) until we’d arrived at the minimum point of the loss function, I wondered what else could happen, and what kind of mathematics would be involved.

A quick aside: The L-BFGS algorithm is a clever application of many good ideas, so I recommend the resources listed at the end of this post for anyone who’d like to dig into the details.

As someone who’s still working on their statistics and ML fundamentals, my only goal here is to begin to appreciate why L-BFGS is a commonly-used optimisation algorithm.

One thing to clarify early on: The “L” part of L-BFGS refers to it being a “limited-memory” approximation of the BFGS algorithm.

However, both algorithms “use an estimate of the inverse Hessian matrix to steer [their] search through variable space” towards the minimum point of the loss function, and I’ll be looking at this shared behaviour here.

To understand what the Hessian matrix is, and why its inverse can make gradient descent more efficient, we can recall some mathematics learned at school.

Specifically, we might remember that the Taylor expansion of a function can include a term containing the second derivative:

When we include this term, we discover a quadratic curve that fits the function at that point:

And a good step to take towards the minimum of the function that we’re approximating is the step that takes us to the minimum of that quadratic:

If we were working with one-dimensional data, we could find the minimum of that quadratic from the function’s second derivative.

Newton’s method for optimisation does this, first by reworking the Taylor series slightly to more easily use it iteratively:

And then by setting that quadratic approximation equal to zero, to obtain its minimum:

This gives us additional information to use when choosing step sizes during our gradient descent process, essentially suggesting that we step towards the minimum point of the quadratic that we were looking at in the graphs above.

This is more sophisticated than ‘classic’ gradient descent, and can help make it more efficient.

Of course, in higher dimensions, things are a little less simple, as we need second-order partial derivatives to obtain this kind of information about the optimal descent direction.

And this is where we look to the Hessian matrix: a square matrix of second-order partial derivatives of a scalar-valued function.

In fact, to step to the minimum of the quadratic, we take a step that is proportional to the inverse Hessian times the gradient:

As indicated in the above screenshot, the BFGS algorithm actually approximates the Hessian, because it can be expensive to compute and store the full Hessian matrix.

And, for even more efficiency, the L-BFGS algorithm stores some information about its previous Hessian approximations, to remove the need for recalculation of the inverse Hessian at each step.

Note that L-BFGS has limitations, too. I found that scikit-learn’s logistic regression guide helpfully summarises some practical details. For example, the type(s) of regularisation that each solver supports, and the datasets that a solver might be more or less compatible with.

But for now, this quick look at some of the foundational mathematics underlying the BFGS algorithm achieves my goal for this week, so I’ll wrap up here.

I’ll be reviewing second-degree Taylor polynomials of multivariable functions in the Math Academy course soon, and I won’t need to wonder why we’re looking at matrices of second-order partial derivatives!

Specifically: the L-BFGS optimisation algorithm, regularisation, and how to tell whether a model is overfitting or generalising well.

Given my learning style, I will need to find the simplest possible explanations and connect them to the level of mathematics that I’m comfortable with.

So I’ll be reading up on these topics (and writing some simplified explanations to share) over the next few weeks. And here’s my plan.

The L-BFGS Optimisation Algorithm

In scikit-learn’s LogisticRegression class documentation, I saw the solver option for choosing the algorithm used in the optimisation problem.

The default solver option is L-BFGS, because “it works reasonably well for a wide class of problems”.

I assume this refers to a balance between efficiency and accuracy. But I’d like to look at some of the details here and, if possible, find a bit of mathematics I can understand at my level.

Planned starting point:

scikit-learn’s User Guide page about the differences between solvers, specifically the “Solver’s details” dropdown, which looks full of interesting details.

I also see some MIT Open CourseWare Numerical Methods lecture notes that cover the BFGS algorithm, and some Purdue lecture notes that look quite friendly.

Regularisation

While reading the scikit-learn documentation pages referenced above, I also ran into the term “regularisation” many times.

I only understand this as some way of avoiding overfitting, at present.

Although regularisation is critical in machine learning, it’s a more broadly-applicable idea across statistics, finance and computer science.

So I’ll try to find simple examples from these other fields, to avoid overfitting my own understanding.

Planned starting point:

A closer look at the overview on Wikipedia, and at the visualisation in scikit-learn’s Regularization path of L1- Logistic Regression.

Also, the textbook that I’m probably a little too attached to at this point, Deisenroth’s “Mathematics for Machine Learning”, has section 8.2.3, “Regularization to Reduce Overfitting”.

Detecting and preventing overfitting

The previous topic leads me here, to spend some time understanding where regularisation fits into the bigger picture.

What are other ways to avoid overfitting (people seem keen on cross-validation)? What formulas and notation are used to describe problems and solutions here? What happens with more complicated models, like artificial neural networks?

Planned starting point:

As well as Deisenroth’s textbook, Google’s Machine Learning Crash Course has an overview that looks straightforward.

I also have the “Introduction to Statistical Learning” textbook (James et al) that discusses overfitting throughout, and in particular in section 2.2 (“Assessing Model Accuracy”).

And Bishop’s “Pattern Recognition and Machine Learning” has section 5.5, “Regularization in Neural Networks”, where I’ll probably just see if the introduction has any simple-ish takeaways.

Well, it looks like I’ve planned out the next 6-8 weeks of my available study time, if not more! But it’s such a luxury, in my opinion, to study by following your curiosity.

(Although I always follow a structured course at the same time, because guidance is important, too. Currently: The “Mathematics for Machine Learning” course on Math Academy.)

If you have any beloved resources that you’d recommend for any of the topics listed above, please feel free to comment to share them.

This isn’t entirely straightforward because, of course, machine learning models aren’t built by simply choosing the equations you feel like using (unless you’re at a level of expertise that makes model development feel this way, perhaps).

And I feared that LLM sycophancy would cause wasted time, as I find them to be a little “garbage in, [beautiful] garbage out” when given an unwise prompt.

So I started with a naive web search for human-written ideas, and was lucky to find something relevant immediately.

In this thread on stats.stackexchange.com, someone asks, “Do we ever use maximum likelihood estimation?”. And I sympathise with them, honestly. Sometimes we are taught things in such isolation that we learn without understanding.

The responses in that thread quite kindly and patiently explain that, yes, the MLE is used frequently.

I found this response to be a particularly useful signpost:

A very typical case is in logistic regression. Logistic regression is a technique used often in machine learning to classify data points. For example, logistic regression can be used to classify whether an email is spam or is not spam or classify whether a person has or does not have a disease.

(I’d also call out this response for including many helpful links, plus a mention of regularisation, which is something we’ll stumble into again later.)

As much as I enjoy taking advice from random discussion threads, I should mention that I also had Bishop’s “Pattern Recognition and Machine Learning” textbook open to section 4.3.2, where we see:

We now use maximum likelihood to determine the parameters of the logistic regression model.

So, feeling confident that this wasn’t an entirely nonsense mission, I chose logistic regression to pair with MLE and began my journey of discovering what this might actually look like.

(Note that this topic only occupies 1.5 pages in Bishop’s book, vs. several hours of my lifetime, but that is typical.)

Finding a data set

There’s a repository called awesome-data-sets, which I found overwhelming (although a wonderful resource), so I also prompted Claude for a recommendation.

We settled on the palmerpenguins dataset or, specifically, the penguins dataset provided by this package, which is a simplified version of the raw dataset that it also contains.

Taking a quick look at the penguin data:

# First five rows:
  species     island  bill_length_mm  bill_depth_mm  flipper_length_mm  body_mass_g     sex  year
0  Adelie  Torgersen            39.1           18.7              181.0       3750.0    male  2007
1  Adelie  Torgersen            39.5           17.4              186.0       3800.0  female  2007
2  Adelie  Torgersen            40.3           18.0              195.0       3250.0  female  2007
3  Adelie  Torgersen             NaN            NaN                NaN          NaN     NaN  2007
4  Adelie  Torgersen            36.7           19.3              193.0       3450.0  female  2007

# Dataset shape:
(344, 8)

# Columns:
['species', 'island', 'bill_length_mm', 'bill_depth_mm', 'flipper_length_mm', 'body_mass_g', 'sex', 'year']

To make it easier to visualise/plot later, I chose to create a model that will classify penguins into one of two species (Adelie or Gentoo), using just one feature: bill length.

Since I chose logistic regression as my model class, the probability of a penguin being an Adelie is represented as:

Here, w and b represent the unknown parameters the model needs to pick the best values for. And σ, the sigmoid function, makes sure we have values between 0 and 1 for our probabilities.

Since I’m modelling a binary event (essentially “Adelie or not”), there’s a Bernoulli distribution, so the probability of either outcome can be written in one expression:

(If this looks strange, see here for an interactive refresher on Bernoulli distributions.)

Putting the MLE to work

Now to finally see the MLE in action, picking values for the parameters w and b that maximise the probability of the observed training data under the model.

I saw in my previous post that the MLE turns probability products into a sum of log-probabilities (a log-likelihood). Those sums are much easier to work with.

So my two-in-one probability becomes:

I’ll briefly mention that the negative of L(w, b) is the expression for binary cross-entropy loss, so when we hear about “minimising cross-entropy” in machine learning, we can recall the MLE and understand what’s happening there.

Now, since the goal is to maximise L(w, b), and we know from calculus that the maximum of a function occurs at points where the gradient is zero, we might take the partial derivatives of L with respect to w and b, and set both to zero.

(For a well-explained working of this, see the “Derivatives Without Tears” section of this Stanford CS109 Course Reader.)

If we find a (w, b) where both partials equal zero, we have found a candidate for the maximum likelihood estimation. But, unfortunately, we can’t isolate w and b algebraically because they sit within the sigmoid on the right-hand side of each equation.

So we turn to numerical optimisation, where an algorithm like gradient descent could help us by iteratively searching for the values of w and b that yield the zero-gradient point.

In fact, I used scikit-learn here, so the algorithm was L-BFGS (which is a candidate for an upcoming post, to understand why it’s described as “a good default solver”).

from palmerpenguins import load_penguins
from sklearn.linear_model import LogisticRegression

# We load the dataset
penguins = load_penguins()

# We create a smaller data frame containing only Adelie and Gentoo penguins,
# and the drop any rows where the bill_length_mm value is missing/NaN
dataframe = penguins[penguins['species'].isin(['Adelie', 'Gentoo'])].dropna(subset=['bill_length_mm'])

# Next, we split the data into model inputs, X, and target labels, y
X = dataframe[['bill_length_mm']].values #=> (number_of_rows, 1)
y = (dataframe['species'] == 'Adelie').astype(int).values  #=> [1] or [0]

# The decision boundary is where the sigmoid outputs 0.5, i.e. where
# the model can't decided between the two classes
decision_boundary = -b / w

# scikit-learn's LogisticRegression classifier applies regularisation
# by default, but we can essentially turn that off (in order to see the
# MLE at work most clearly) by passing a big number, like 1e10, as the
# C parameter
# See: https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html
model = LogisticRegression(C=1e10).fit(X, y)

# We can take a look at the values for w and b that the model found:
print(f"w = {model.coef_[0][0]:.3f}")    #=> -1.214
print(f"b = {model.intercept_[0]:.3f}")  #=> 52.412
print(f"decision boundary = {decision_boundary:.3f} mm") #=> 43.165 mm

(This is where I ran into regularisation again, which I will also investigate in another post. But back to the MLE for now.)

The MLE has given a value for w and a value for b. If I plug those values into L(w, b) then I get L ≈ -38.256.

Or, by comparison, I can make a no-information guess of (w, b) = (0, 0), and see L ≈ -189.922.

So the MLE gives a higher log-probability of seeing the training data than a wild guess, which seems a good thing. Of course, this isn’t a production model and there are no profound insights here. But it’s nice to see some outputs from real data.

My goal was to get a sense of the MLE’s role in machine learning, so that I’m not one day asking, “Is the MLE used anywhere?”, and I’d say that’s been achieved.

To close, a graph showing where a few bill lengths from the dataset lie in relation to the sigmoid generated by the parameters w = -1.214 and b = 52.412:

Since a shorter bill length correlates with Adelie penguins (a class label of 1), the sigmoid curve approaches 1 as bill length decreases. As bill length increases, it approaches 0, for the Gentoo class label of 0. And we see the decision boundary sitting at x = 43.165.

Very simple, but I liked this as a visual of the sigmoid’s role in logistic regression.

References and resources:

• CM Bishop, “Pattern Recognition and Machine Learning”, 2006

• https://stats.stackexchange.com/questions/267393/do-we-ever-use-maximum-likelihood-estimation/

• https://github.com/allisonhorst/palmerpenguins

• https://en.wikipedia.org/wiki/Regularization_(mathematics)

• https://www.acsu.buffalo.edu/~adamcunn/probability/bernoulli.html

• https://docs.pytorch.org/docs/2.12/generated/torch.nn.BCELoss.html#bceloss

• https://chrispiech.github.io/probabilityForComputerScientists/en/part5/log_regression/

• https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html

• https://en.wikipedia.org/wiki/Regularization_(mathematics)

Next, let’s set up a simple example for connecting abstractions to the real world:

We’re building a model that will tell us how likely it is to rain on a future day.

We pretend that previous weather has no impact on later weather, so our training data points are independent.

Our training data might look like this:

In other words, we have the training set:

Now we can work through the notation piece-by-piece, using the example to illustrate what each part is doing.

The simplest possible model

The simplest possible model has one parameter, θ, which represents the possibility of rain on any given day:

The semicolon in p_model(x; θ) is just a notation meaning: “This is a function of x, and it’s parameterised by θ” so that different choices of θ give different models.

θ = 0.5 means equal chance of rain or no rain. θ = 0.9 means it’s very likely to rain.

Our task is to pick the best θ, where “best” means the value of θ that makes our actual training data look as likely as possible.

Recall that our simple example training set is:

If we pick θ = 0.6, then our model tells us the probability of seeing the exact sequence {1,0,1,1,0} is:

But if we picked θ = 0.3, we’d find ourselves with:

So θ = 0.6 makes our data more likely than θ = 0.3 does.

So we’re looking for the value of θ that maximises this probability which, currently, we’re calculating as a product.

At this point, our “work-in-progress” maximum likelihood estimator looks like this:

Problems with products, solved by logarithms

We notice that, if we had a real-world training set containing a million data points, this product would become extremely small.

It is called “numerical underflow” when numbers become so small they are essentially lost from floating point arithmetic, as they are rounded to zero during computation.

And we also know that we’re using the argmax function, which finds the input that yields the largest output for a given function.

So we think about taking the logarithm of our probability product, because:

1. The logarithm function turns products into sums, so we won’t multiply tiny numbers until they disappear completely

2. For numbers between 0 and 1, the closer the number is to zero, the further from zero (in the negative direction) the logarithm will be

   1. More generally, logarithms preserve the order of numbers: If a is greater than b, then log(a) is greater than log(b)

(We’ll use the natural logarithm here.)

For θ = 0.6, the logarithm of our probability product is:

And if we set θ = 0.3:

Of course, the sum allows us to avoid the multiplication of increasingly tiny numbers.

Our log-probabilities are negative, but higher probabilities become negative numbers that are closer to zero, so argmax can still identify the “best” value for θ without any trouble.

So we’re very happy with this logarithm decision.

And we can update our “work-in-progress” maximum likelihood estimator to:

Why not take the average?

At this point, we spot a new opportunity.

If we take the average of the log-probabilities for each value of θ by dividing their sum by m, this won’t change the order of the values, either.

So we can update our “work-in-progress” equation to:

(This is just the first half of the opportunity, so continue on to see the whole thing.)

Next, we consider what would happen if we randomly sampled a data point from our training set to compute the log-probability for, with an equal probability of selecting any one of them.

In other words, we introduce a random variable, X, with probability distribution, p̂_data, to represent a sampled data point and the likelihood of it taking a particular value.

Recalling our simple example training data:

If we pick one of the days at random, with equal probability of picking any day, then the likelihood of picking either a 1 or a 0 is:

We can then ask what the expected value of the log-probability would be for this sampled data point.

Since expected value is a weighted average of all values a random variable can take, we see that the expected value of the log-probability of a randomly sampled data point would be the average of all of the log-probabilities.

Returning to our example, and choosing θ = 0.6 to see the result for this expected value calculation, we can take a direct look at the log-probabilities first:

We can sum these log-probabilities and divide by 5, if we want, to obtain approximately -0.673.

And, recalling the likelihoods we saw in the previous example section, we can compare this to the expected value weighted average calculation:

That is to say:

So the expected value of the log-probability for a randomly sampled data point equals the average of the log-probabilities. Interesting!

And not only interesting but significant, because p̂_data is an empirical distribution that becomes a better approximation of the true distribution, p_data, as the training data set grows. This is the law of large numbers.

And p_data is what we really want to model (although we can never do so perfectly).

So we swap this expected value into our “work-in-progress” equation:

But we have, in fact, arrived at the “real” equation. So we can remove the “work-in-progress” label, and pause to appreciate our progress (and the equation itself).

Time for a cup of tea ☕️

For example, when I started learning about stochastic gradient descent two weeks ago, I suddenly realised that my understanding of random variables wasn’t good enough.

I knew the formal definition, but that was not making it any easier to parse the new textbook sections I was reading.

Something can be turned into a random variable? That wasn’t mentioned in the formal definition! How and why would we do it?

I realised I needed to do some wider reading, revisiting the fundamentals armed with some new questions.

And here’s what I learned, when I thought I already knew enough.

Not random, not a variable

First, I reviewed definitions across a few textbooks and websites.

Deisenroth points out that the words “random” and “variable” must not be taken literally:

And while Goodfellow gives us a definition that does use these words, he follows it with a descriptor that sounds very function-like to me.

This is a good moment to return to the formal definition, and this diagram I found that pairs well with it (annotations mine):

Here, the random variable, X, is shown visually as a function mapping one space to another.

So we see how X is doing the job of telling us what states, or outputs, are possible, as Goodfellow describes.

In other words:

For anyone else who finds that concrete examples are the best scaffolding for intuition, let’s imagine a simple 6-point dataset:

(You might recognise these numbers from my previous post on stochastic gradient descent. They are just being used as example values here, but it’s a fun callback (for me, at least).)

Let X represent the random index drawn when we sample one data point from it.

Then X is the function that maps a number between 1 and 6 to a data point value.

If we draw a sample and observe that n = 3, then the data point value is -15.

The question X represents is, “What is the value of a data point sampled from this dataset?”

What random variables are for

While I found it interesting to carefully review the definition of a random variable, I always need to put abstract ideas into context to actually deepen my understanding.

Specifically, I always have the common student question: Why is this important?

In this case: Why do we want these functions? What do they allow us to do?

As a software developer, I found the following suggestion interesting:

In programming, it’s common to have an entity that will output something when instructed to but, in itself, just represents the ability to obtain an output.

But this idea became more useful when I was reminded of something it doesn’t explicitly mention: We do actually know quite a lot about the values that a random variable might take.

In other words, random variables, as functions, are meant to make it possible to work in a normal mathematical way with values that we don’t exactly know (yet), but which we do know certain things about.

Returning to our 6-point dataset: We know X will only ever return one of six indices, and therefore one of six data point values. And we know each is equally likely.

So while we don’t know what value we’ll get until we sample, we do know what we might get (and what we definitely would not).

One practical tip

One of the best things I came across in my reading on this topic was this advice:

It didn’t fit into either of the previous sections about the definition and purpose of random variables but, IMO, deserves its own space here. Any time another student can share a hard-won insight from their time studying a topic, I’m listening!

Referring briefly back to our concrete example: We know X is the random variable, and -15 is a possible realisation.

And if, after this whole journey, we look again at the snippet from Deisenroth’s book, the implications of a parameter vector being turned into a random variable are easier to appreciate.

θ was an unknown but fixed quantity that we wanted to estimate, but now we know it is a function associated with a range of values and their probabilities.

And this paves the way for a (slightly!) easier time studying Bayesian linear regression.

(But, importantly, we do know the range of values it might return, and the likelihood of it returning any particular one of them.)

A random variable has an expected value, which is simply a weighted average of all the possible values that the random variable might return.

For example, if a random variable represents a roll of a fair 6-sided die, the expected value is:

The expected value operator, 𝔼[.], has the property of linearity. For any random variables X and Y:

…which is nice and simple to consider! But what can it help us understand in machine learning?

(Sidenote: While trying to answer this question, I realised I didn’t have a particularly nuanced understanding of random variables, so I did a bunch of extra reading about that first. It was a worthwhile diversion! I’ll write up my findings to post here next week, and here’s a shorter post on Instagram in the meantime.)

Back to the present topic: The linearity of expectation, and how surprisingly easily we can see its importance in a machine learning technique called stochastic gradient descent.

The problem: Training a regression model can get expensive

I think the easiest way to understand this is to look at a concrete example, even if the example is greatly simplified.

So let’s imagine we are fitting a line, ŷ = wx, to a data set of just six points:

We can see from the table that w should be around 2, but we’ll imagine that it’s not so obvious.

Our first guess will be that w = 1.5. And we’d like to improve it by taking a gradient descent step. That is, we’d like to see whether our next guess should be higher or lower.

To keep things simple, we will choose a classic squared error to represent the loss (“wrongness”) of our current guess for w per data point:

Our objective function will be the sum of all of these losses:

(The objective function provides an overall score for the model’s predictions.)

As our goal is to reduce loss using gradient descent, we will compute the gradient of the loss for each data point.

The gradient is a derivative, because it’s the slope of the function at that point that will tell us whether we’re moving in the right direction:

Let’s make this all concrete with an updated table, to see what we get from our first gradient descent iteration.

We see each data point’s gradient is negative, which makes sense because our current guess, w = 1.5, is too small.

(Reminder that gradient descent involves stepping in the opposite direction of the calculated gradient, so a negative gradient means we should go in the positive direction for our next parameter value.)

Since our objective function is the sum of all of these losses, we have that our “full-batch gradient” is ▽L(w) = -131.

And that means our average gradient per data point is approximately -21.83, which we’ll come back to in the next section.

But to conclude this section first: If we needed to repeat a set of calculations like this for a real-world data set that contained millions of points, even one step of gradient descent would be compute-intensive, and we might need thousands of steps.

A solution: Random sampling

Let’s remember that the average gradient per data point in our example data set is approximately -21.83. We found this by calculating the full sum of gradients across all of the data points. But we’d like to cut down the computational work.

Can we estimate that average using a “mini-batch” of data points, to save us having to do so much work? Let’s see.

We’ll say we can only afford to find the sum of gradients for two data points, so our mini-batch size is 2.

We can randomly pick two indices from the first column of our table. We draw {2, 5}.

So the average gradient is:

…which is close to the full-batch gradient average.

But if we had drawn {1, 3}, we’d have an average mini-batch gradient of:

…which is not close to the full-batch gradient average. So we see that individual mini-batches can be noisy.

However, since we’re now randomly sampling from the data points, each drawn gradient ▽L_{n_i} is a random variable. It’s equally likely to be any of the six values in the fifth column of our table above.

And most importantly, its expected value is just the average of all of these possible values.

That is to say:

We were calculating the average of a mini-batch gradient as:

…where n_1 and n_2 are the indices of our two sampled data points.

So the expected value of that average is:

And, by the linearity of expected value, we have that this equals:

So we see that the expected value of the mini-batch average gradient is equal to the full-batch average gradient.

And this all means that the noise we see in the random sampling approach will, over many gradient descent steps, average out so that we still head in the right direction. But using far fewer computations than a full-batch method.

It’s interesting how some quite simple ideas work together here for such a useful result!

Next Steps

• You can test out the ideas here yourself by calculating the expected value of the average mini-batch gradient for our example data set, using a batch size of 3 to see how the expected value

• If needed, you can strengthen your statistics and probability fundamentals using free resources like KhanAcademy.org, Harvard’s free courses, and MIT OpenCourseWare

• Learn more about gradient descent using resources like KhanAcademy (free), Coursera (some free), and Google’s Crash Course (free)

• Start implementing your own stochastic gradient descent learning routine in a linear regression model by following a tutorial on scikit-learn.org, which will handle the calculations for you (but you’ll know how and why they work!)

References:

• https://en.wikipedia.org/wiki/Expected_value

• Deisenroth et al, “Mathematics for Machine Learning”, 2020

• Goodfellow et al, “Deep Learning”, 2016

• James et al, “An Introduction to Statistical Learning”, 2023

• The “Mathematics for Machine Learning” course by MathAcademy