Overview

The Part 4a: RSA Theory post has 4 predictions for the RSA-ridge relationship. Here, we illustrate them empirically:

1. Ridge vs RSA: does the theoretically-predicted ridge-vs-RSA frontier formula hold, at least approximately?

2. Predicting high-level behaviors: does the ridge metric of neural fit better predict metrics of human visual behavior patterns, as compared to the RSA metric of neural fit, as weak-strong equivalence results would suggest?

3. Low-level visual phenomenology: which neural-fit metric better predicts V1 physiology metrics?

4. Sampling issues: which neural-fit metric better aligns different data samples of the same modality and brain area?

1. Ridge vs RSA

Part 4a: RSA Theory told us that we expect an RSA-vs-ridge plot to look something like one of the below plots:

The left plot above obtains if the RSA is pretty close to “canonical” — e.g. not contaminated by task-irrelevant noise. The right is more like one would expect if task-irrelevant noise dominates.

How does RSA relate to ridge in practice? Let’s look at some plots. We’ll start with data from the Natural Scenes Dataset (NSD). Each panel below corresponds to one of the brain areas in the NSD localizer scheme. Each dot is a model, and the model zoo ranges over a wide set of distinct vision models.1 The x-axis of each plot is RSA similarity and the y-axis is Pearson’s r between data and the ridge-fit model predictions.2

To my eye, the NSD results seem pretty canonical, actually — fairly clean-looking.

Here are the results for the LAION-fMRI dataset — pretty similar looking:

Here is the same for the Bold Moments Dataset (BMD). Much the same pattern emerges, a bit less clean — a little more confounded than the NSD or LAION-fMRI results3:

And finally, here is the same for the Things Ventral-stream Spiking Dataset (TVSD):

Results in V1 and V4 look “kind of ok” in an absolute sense, but comparatively noisy in comparison to the fMRI cases, presumably due to the difficulty of obtaining a nonbiased sample with electrophysiology methods, especially in early visual areas.

2. Predicting high-level behaviors

Another prediction of the contravariance theory for RSA and ridge is that raw RSA will be less predictive of behavior across a wide variety of model implementations than ridge, because raw RSA doesn’t obey Zippering (e.g. control by downstream task) while linear similarity does. We can test this by plotting behavioral metrics against the same neural fit metrics as above.

Brain-Score implements a wide variety of behavioral metrics and has a strong model zoo, so it’s a good place to start. We are interested in asking: “To what extent does a model’s being brain-like, under one or another quantitative metric of brain-likeness, predict its behavioral consistency?” Thus we consider brain-likeness as assessed relative to several recent large-scale neural datasets, including NSD, TVSD, and the just-released LAION-fMRI dataset.

Behavioral consistency can be assessed by any of the “behavioral” metrics on Brain-Score. There are 43 individual (non-aggregate) behavioral metrics, and we consider them all. Each of these measures a model’s consistency with some pattern of human behavior, rather than its raw performance accuracy on some task. Each metric is based on comparing model error patterns to those measured in a real human psychophysical experiment on some task.

Here are the results of one, the Rajalingham-i2n metric, which tests the ability of a model to predict image-by-image human object-recognition confusion patterns across several thousand naturalistic images. The x-axis is model-to-human behavioral similarity according to this metric, the y-axis remains ridge-based neural similarity. Each row is a different neural dataset and each column is a different brain area (V1, V4, and IT).

We notice that the correlations are pretty decent from IT, and nontrivial but less good from V1. This is not a major surprise, since IT is thought to support object recognition behavior, but it’s good to see the intuition confirmed.

Now let’s have a look at the same Dependent Variables (DVs), but with different Independent Variables (IVs) — in this case, RSA similarity:

We only show NSD and LAION-fMRI comparisons here because Brain-Score doesn’t implement RSA for the TVSD dataset. But the conclusion is pretty clear: ridge tracks behavior substantially more accurately than (raw) RSA, despite ridge and RSA being decently correlated as shown in the last section. Attempting to predict behavioral similarity pulls RSA and ridge apart.

This pattern is not confined to the Rajalingham-i2n metric. Here are similar results for the Geirhos2021eidolonIII-error_consistency metric, which measures model error-pattern similarity to human choices in images with blurred object appearances but natural pixel distributions (see Fig. 12 in the Geirhos texture-bias paper). First from ridge:

Then from RSA:

The absolute correlations are lower (like ~0.65 from IT to the metric rather than ~0.85) but the ridge-RSA pattern remains the same.

In fact, taken across all the behavioral metrics in the Brain-Score collection, the pattern is extremely robust:

Breaking that out more finely, out of 172 metric / brain area combinations, RSA is more correlated to behavior than ridge in only 3 situations. This is summarized here:

3. Low-level visual phenomenology

One might wonder if the ability of ridge to better predict measurement variables than RSA similarity is confined to “high-level” behaviors. To test this we consider the V1-Marques metrics in Brain-Score. These metrics were compiled from the literature on “classic” V1 physiology, and measure detailed single-neuron phenomenologies such as circular variance and texture selectivity. There are a total of 22 such metrics in Brain-Score.

Here are ridge-vs-metric results for texture selectivity in the Freeman-Ziemba V1/V2 dataset:

Overall absolute predictivities are not amazing numerically, but it’s hard to know exactly what the noise ceilings are (this is definitely a question for another time!). Relatively speaking, however, the pattern is maintained — predictivity drops substantially when we pass to RSA:

Looking across all areas and V1 metrics, we see the pattern robustly. For example, using the NSD dataset as the prediction signal, and looking across the main areas covered by Brain-Score we see:

We find that V1-Ridge results are meaningfully better than other areas and RSA at predicting low-level V1 physiology. And here is the summary result for predicting low-level V1 metrics from V1 neural predictivity metrics, over all the main datasets in Brain-Score that support this kind of comparison, including NSD V1, LAION-fMRI V1, Triple-N V1, and the underlying Freeman-Ziemba V1 data:

Again, while overall predictivities are not amazing4, ridge retains a meaningful edge over RSA for all datasets and essentially all metrics.

Why do we see all these results pointing in the same direction, whether they are “high-level behaviors” or low-level V1 physiology measures? We posit that the underlying cause of these observations is the contravariance theory consequence described in Part 4a: RSA Theory: that the task-linked core geometry component of RSA — and only that component — obeys weak-strong equivalence and zippering.

4. Sampling issues

One of the major issues discussed in Part 4a: RSA Theory is the fact that raw RSA is susceptible to sampling issues, essentially because sampling issues are the “measurement consequences” of the task-irrelevant symmetry nuisances identified in the Theiß et al paper.

The issue of sampling is especially pronounced in ephys datasets because of the nature of sampling, especially in older-school single-electrode datasets (such as the Freeman-Ziemba). It is thus instructive to compare the alignment of RSA and ridge on two different neural samples. In this case we compare the older Freeman-Ziemba V1 dataset against TVSD-V1. The Freeman-Ziemba V1 dataset consists of 105 single electrode sites tested on 315 oriented Gabor and texture stimuli, while the TVSD V1 dataset consists of 960 array electrode sites tested on approximately 22k natural images.

Under ridge we can see that these datasets are both measuring a common physical system (macaque V1) with a common methodology (electrophysiology); under raw RSA this is substantially obscured.

There are many metrics NeuroAI people use for comparing neural network models to real brain data. There’s also a lot of argument about which one(s) are “right”, and to what extent the metrics agree with each others’ assessment of model correctness. Here we address one of the main arguments in this field (“regression vs RSA”) from the Weak-Strong Equivalence perspective. (See series Part 1 for a refresher on Weak-Strong equivalence.)

Contents

1. The RSA vs linear regression debate

2. Decomposing RSMs into core and noise terms.

3. Connection to behavior and mechanism

4. Canonical RSA and weak-strong equivalence

5. The RSA-ridge relationship

6. Neural sampling is “symmetry-Like noise”

7. Does the choice of metric matter? Revisited.

1. The RSA vs Linear Regression debate

Two of the main metrics the NeuroAI community uses are:

1. Linear regression similarity (basically the same as “weak alignment” in earlier posts in this series). Linear regression similarity metrics come in a number of flavors, but they all basically work by (a) fitting a linear map between source and target using some training set; (b) measuring per-unit (neuron/voxel) prediction accuracy on held-out stimuli; and (c) reporting an average over units.

2. Representational Similarity Analysis (RSA). RSA also comes in a number of flavors, but basically works by computing, for each feature set, a Representational Similarity Matrix (RSM) — that is, a NxN matrix of correlations between feature vectors, where N is the number of stimuli. (The features can come from either neural data or model activations.) The (i,j)-th entry of the RSM is a similarity measure: it’s 1 if the feature set “thinks” stimulus i and stimulus j are essentially the same. The whole RSM is like a layout across all the stimuli of which stimuli the feature set “thinks” are relatively similar to each other and which are not, which is why the RSM is sometimes called the “representational geometry”. Two feature generators are compared by correlating their RSMs (thought of as long flattened-out vectors); the output of this comparison is the RSA score between two representations, which is high (close to 1.0) whenever two feature sets basically make the same default judgements about which stimuli are similar/different.

Each of these metrics has surface-level pros and cons. Regression similarity can lead to high prediction power (which is useful for practical purposes), since it optimizes a bunch of parameters (the regression weights) explicitly to improve predictions. By the same token, regression needs quite a bit of data to fit the regression parameters well. Conversely, RSA doesn’t need to estimate any additional parameters (and so can be useful in low-data regimes). But by the same token, RSA might underestimate the latent representational capacity that could be extracted from a set of features by a downstream interpreter that has the ability do some reweighting (such as a real neuronal “readout” might do).

But this direct accounting of practical pros and cons doesn’t really capture a deeper issue about metric choice in NeuroAI. A major question (maybe you could even say “controversy”) that has arisen in the field asks something like:

“Which metric is more the right way to compare candidate models to neural data for the purposes of true understanding? Is there a tradeoff between maximizing prediction accuracy and achieving scientific meaning?”

With this issue in mind, let’s dive into what the new version of the Contravariance Theory paper tells us about RSA and its relation to linear regression. (For all the gory math details see §A.12 of the paper’s appendix. Also see §A.13 for a similar treatment of another popular metric, Centered Kernel Analysis (CKA).)

2. Decomposing RSMs into core and noise terms.

Much of our thinking about RSA in the Contravariance Theory paper is built on recent work from Theiß, Braun, Saxe, and Grant, entitled Parameter symmetries determine representational geometry in overparameterized nonlinear networks, from here referred to as [TBSG].

The first main result of [TBSG] is that RSMs decompose into two parts1:

where:

• the q_j factors are computed from the privileged axes,

• the 𝛾_j factors are controlled by reweightings and multiplicity of the privileged axes, and as such do not affect the downstream computations, and

• the a matrix is determined only by function-preserving symmetries that are irrelevant to the represented function. Theiß et al call the a matrix “noise”, because it’s entirely composed of nuisance degrees of freedom.

The [TBSG] decomposition has two major consequences.

First, the “raw” RSM can be totally dominated by factors that are independent of functionality. Indeed, the 𝛾_j values and a matrix can be chosen so that two representations with identical privileged axes (and thus identical q_j factors) can have arbitrarily different RSMs — e.g. be arbitrarily far apart under the RSA metric — even though they compute the same downstream function. These factors can obscure or even dominate the part of the RSM that reflects the features actually used in the computation. They are “functionally silent but geometrically visible”.

Second, the RSM has a uniquely-determined core geometry that is task linked, and where perturbations to that geometry would have substantial consequences on the downstream represented function. To get at that core geometry, one needs to remove the a matrix and normalize out the 𝛾_j factors — that is one needs to canonicalize.

3. Connection to Behavior and Mechanism

The above result has a key conceptual implication for contravariance. Recall from Parts 1 and 2 of this series that in the hard task-constrained regime, linear similarity is both connected to lower-level mechanistic structure recorded in the privileged axes (Weak-Strong Equivalence), and highly constrained by the task (through Zippering). However, as we’ve just seen, RSA can be dominated by task- and privileged axis-irrelevant factors. Thus, we should predict that RSA similarity will in general be less correlated across trained neural network models than ridge similarity with other factors that themselves do depend on the core computation — such as downstream behavior (e.g. task error patterns) or upstream privileged axes metrics (e.g. V1 unit tuning-curves) .

See Part 4b: RSA empirics for an investigation of this prediction from an empirical viewpoint.

4. Canonical RSA and Weak-Strong Equivalence

Canonicalization is a process by which the task-irrelevant factors are removed to reveal the core geometry. In the Contravariance Theory paper appendix §A.12.5, we describe two ways to do this:

Privileged-Axis Normalized RSM: First, you can work directly with the privileged axes, ensure only one copy of each privileged axis type is used, and divide by its norm, before computing the RSM. More formally, for each privileged axis f_j, define the normalized RSM component for that axis by

(this is an a NxN matrix, where N is the number of stimuli) and, assuming there are m preferred axes, define the axis-normalized canonical RSM by:

In words, this makes each essential feature’s response profile unit length, gives each essential axis one copy and equal weight, and computes the usual RSM of that canonicalized representation.

Minimum-Norm Balanced RSM: Unlike the axis-normalized idea above, in [TBSG] a more sensitive (but complicated) approach is taken: instead of discarding axis weights, [TBSG] describe procedures for choosing specific weightings that depend on the size or cost of a generated feature and on the downstream readout that it uses, rather than giving every essential class one equal vote as RSM_ax does. There are multiple cost functions that make sense, including minimum representation (MR) and minimum weight (MW) costs (see the original [TBSG] paper for details). Such weightings attempt to take biological implementation costs into account: MR might approximate activity/metabolic cost, while MW might approximate synaptic or wiring cost.

Whether one prefers the axis-normalized or minimum-norm versions depends on one’s purposes: if one just cares about “the computation”, the former is more appropriate; while if one is interested in “the biophysics”, the latter may be more on target.

The key point with respect to contravariance for RSA is that, regardless of which canonicalization method one picks, canonical RSA obeys Weak-Strong equivalence, unlike raw RSA. That is:

Theorem (informal): In the hard-task context, if two same-architecture networks A and B are linearly aligned at two adjacent layers l and l+1 then they have matching canonical RSMs at the earlier layer:

(See the Contravariance Paper appendix §A.12.5 for formal mathematical details.)

We can also show a soft version of this that relates defects in linear alignment to small mismatches in canonoical RSA, just as with privileged axes (see Part 1 of this series for a refresher on those results). Similarly, a full zippering result follows for RSA as well (see Part 2 for more on zippering).

5. The RSA-Ridge relationship

Given that RSA and Ridge are both standard metrics, it is worth understanding what their relationship is.

After canonicalizing to reveal the core geometry (either method is ok), it turns out that there is a very simple relationship:

for some constant C. The funny ≾ symbol means that the relationship is a frontier, rather than a strict equality, but that under reasonable conditions, the equality is close to being achieved. (See the Contravariance paper appendix §A.12.6 for the math details on how this is shown.)

This relationship means that if you plot canonical RSA (x-axis) vs ridge score (y-axis), you should see a scatter plot that looks roughly like a square root relationship:

On the left, I’m plotting a schematic canonical RSA-vs-ridge relationship. On the right, I’m plotting a schematic for raw RSA-vs-ridge. There, a square-root relationship might exist to some extent (depending on how bad the effect of the nuisance variables are), but it will be a bit more obscured and perhaps in some cases entirely absent. See Part 4b: RSA empirics, for more on this topic from an empirical viewpoint.

6. Neural sampling is “symmetry-like” noise.

Neuroscience experiments are, and have always been, heavily affected by issues of undersampling or biased sampling.

In classic old-school single-electrode electrophysiology experiments, only one or a small number of neurons were explored at a time, with perhaps a few tens of neurons experimentally tested in a series of separate recording sessions, over the course of an experiment. This sampling sparseness was exacerbated by a strong bias in stimulus sampling, since neurons were identified for recording in the first place (during the process of lowering the electrode through cortex) by virtue of their ability to respond to a specific test stimulus set, so neurons not excited by that set might be just ignored.

Array electrophysiology improved this by allowing hundreds of units to be recorded simultaneously for tens of thousands of stimuli, and reduced the stimulus-biased localization problem. Nonetheless, even the largest array electrophysiology datasets only record several thousand out of tens of millions of units in an area, sampled in a highly spatially non-uniform fashion.

Though fMRI has full coverage, it too has strong non-uniformity in signal strength across the cortex, leading to complex spatial biases and distortions in response patterns. Other neuroscience measurement modalities (Ca²⁺ imaging, ECoG, &c) have related issues at different scales.

Moreover, these effects are especially important when tuning is itself spatially organized, as it is in many real brain areas. A spatially localized sample from such a system is therefore not merely a smaller statistically-equivalent subset of the full system: it is biased toward the functions represented in that part of the cortical sheet.

The key point for us here is that the effect on RSA of different kinds of sampling biases are analogous to the same nuisance parameters discussed above. In fact, spatial bias is the measurement version of the [TBSG] ambiguities: the same underlying computation and the same privileged axes can yield different raw RSMs because experimental biases assign them different effective scales, multiplicities, and mixtures.

The effects are summarized in the following table:

Some of these effects are mitigated by canonical RSA, but not all. It can help with recording gain, e.g. when the same response profile is multiplied by a scalar; or with exact multiplicity, e.g. if one identifiable tuning/axis class is recorded several times on one side and fewer times on the other. For example, if two arrays both observe orientation and color axes, but one records many near-copies of the orientation axis, an axis quotient could—in principle, after identifying that class—properly count orientation once rather than according to the number of recorded copies. But it will not mitigate the main spatial-sampling problems, such as:

• a localized array missing whole functional patches;

• topographic sampling changing which distinct feature axes are represented;

• fMRI distortions mixing nearby axes;

• or finite sampling omitting rare axes.

So, you know, nullius in verba &c . . .

See Part 4b: RSA empirics for further data-driven investigation of this point.

Does the choice of metric matter?, revisited.

[FYI: though I haven’t described it here, b/c this post is already long enough, the Contravariance Theory paper §A.13 treats the CKA metric, finding very similar results to the RSA situation.]

Despite taking a certain amount of flak on social media (and probably quite a bit more behind closed doors) for making the claim in an earlier post that perhaps the choice of metric didn’t matter all that much (because weak equivalence established that a strong and a weak metric were actually pretty similar), I nonetheless mostly persist in this view, and see the RSA/CKA results as mostly reinforcing this outlook.

Basically, I interpret the above results as saying: canonical RSA/CKA comes along for the weak-strong equivalence ride with privileged-axis and linear alignment, and the non-canonical part of RSA/CKA should probably be discarded anyhow. Do you interpret things very differently?

Transformers are the dominant microcircuit architecture of AI, due to their robust learning and generalization capacities. How they might relate to neurons is an open question, but regardless of the resolution of that thorny topic, any theory of the Contravariance Principle in DNNs would be incomplete without a treatment of the transformer circuit.

Here we do a deep dive on that topic, presenting a (hopefully!) reader-friendly version of the material discussed in appendix section §A.8 of the Contravariance Theory paper. The answer is pretty interesting!

All transformers are divided into three parts — one of which is the residual stream, the MLP block is another, and what is called in the language of machine learning the attention block, the third.1 It turns out these behave somewhat differently with respect to contravariance. Specifically:

• Weak-Strong Equivalence (WSE) happens for the MLP and attention blocks. WSE is the local phenomenon in which, under hard-task constraints, weak linear alignment at neighboring layers converts into strong axis (or transformer head) alignment. (See Part 1 of this series for a refresher on WSE.)

• … but the residual stream is different. WSE does not happen for the residual stream. There is an uncontrolled parameter space in each residual connection that destroys privileged axes there.

• We find that Zippering occurs in the residual stream of transformers. Zippering is the global phenomenon in which, under hard-task and minimality constraints, weak alignment at a downstream layer cascades upstream through a network, providing the core mechanism of the Contravariance Principle. (See Part 2 of this series for a refresher on Zippering.)

• The above two results combine to imply that there are privileged representational axes throughout transformer hierarchies! . . . but you can’t look in the residual stream to find them.

See below for a detailed account of each of these ideas.

A graphical representation of the situation of contravariance for transformers. In the image, A and B are two networks being compared. We find that (i) Weak Zippering obtains for the residual stream and (ii) Weak-Strong Equivalence obtains for the MLP block in the usual way, and modulo “head-gauges” for the attention block. Privileged axes thus arise in the MLP and attention blocks, but not in the residual stream.

The Transformer circuit has three parts.

To establish basic background and common terms, we distinguish three parts within the transformer circuit:

• the attention block (red in the diagram above) is the component of the transformer circuit in which inputs are mixed using a learnable pairwise multiplicative operation,

• the MLP block (green) is the portion in which inputs are mixed using a short linear-nonlinear cascade,

• the residual stream (blue) of the transformer combines the MLP and attention blocks additively and passes forward to the next layer.

Much ink has been spilled in discussing the functional implications of these structural components, but I will note here one reasonable (imo) interpretation: that the MLP block and residual stream support a linear representation of entities, and that the attention block supports efficient detection of relationships between entities.

The Residual Stream Supports Zippering. . .

At a global level within the network, the key observation is that the transformer residual stream supports zippering. The reason for this is essentially the same as why regular affine-nonlinear (e.g. ReLU) networks support zippering (see Part 2): that if your network is minimal (informally, not wasting units and layers), all solutions are either a single canonical solution, or one of a set of “silly solutions” that pointlessly complicate the pathways relative to the unique minimal solution — but these silly solutions turn out to be very rare (that is, can be proven to be “measure zero” in the set of all networks).

To understand intuitively why this is, suppose two candidate networks became misaligned at one layer. The next layer’s nonlinearities turn that disagreement into new, visible differences; later nonlinearities generally create still more structure that must be reconciled. To ultimately arrive at the same terminal computation in a minimal way, all of those differences would have to be canceled with great precision (the “silly solution”). That high-precision cancelling is exceptional rather than common. So networks can in principle “wander apart in silly ways and then return to the real computation”, but doing so requires special tuning that is extremely unlikely to arise during real-world optimization.

Thus, a canonical solution (up to weak alignment) dominates at each layer of the hierarchy. The full mathematical formalization of this argument, which is a little bit more involved for the transformer circuit than for the simpler affine case, is in appendix sections §§A.8.4-5 of the Contravariance Theory paper.

These theoretical results are consistent with recent empirical work both in vision and LLMs. In vision, [Kapoor et al., 2025] show that hierarchical correspondences exist for transformers under the linear regression metric across distinct networks solving the same task (and actually for stricter metrics than linear as well, including Procrustes, permutation, and soft-matching). In language, across several dozen open-weight LLMs, layers at corresponding relative depths induce similar activation geometries [Wolfram and Schein, 2025].

. . . but NOT Weak-Strong Equivalence or privileged axes.

Interestingly, the residual stream of the transformer does NOT support the conversion of linear alignment on two adjacent layers into a set of preferred single-unit-level axis choices.

This is because, built into the very definition of the residual computation, is an arbitrary invertible affine change of basis that can mix the residual-stream coordinates without changing the Transformer’s computation. Imagine rewriting every residual vector in a new, thoroughly mixed coordinate system. As long as every attention and MLP operation is adjusted to read from and write to that new system, the Transformer computes exactly the same function. The residual stream of the original and rewritten transformer are then perfectly linearly related, even though no individual coordinate in one corresponds to an individual coordinate in the other.

This observation is consistent with the interesting empirical observation from [Kapoor et al., 2025] finding a lack of privileged axes in transformers.

There should be privileged axes in the other blocks.

Residual streams therefore only zipper weakly—they contain the same state up to an affine change of coordinates—without their native axes aligning strongly. But, interestingly, the trivial mixing construction that instantly destroys canonical axis bases in the residual stream does not apply to the other branches of the transformer.

The key difference is where the coordinate change sits relative to the nonlinearity:

• For the MLP block: MLP layers are different from the residual stream because each hidden coordinate is passed separately through the ReLU. A dense mixing of the coordinates before that operation cannot easily be fully undone afterward if the gates are truly used by the network to solve the task. This is just the same argument as “regular” zippering/Weak-strong Equivalence. (See Part 1 of this series.)

• For the attention block: the residual-stream privileged axis-destroying argument actually does partially apply. The query/key (q/k) space and value/output (v/o) space can each be changed by an arbitrary invertible basis transformation without changing the head, just like the residual stream. HOWEVER: for the head as a functional unit, arbitrarily blending two heads does not trivially compute the same function as the original heads. In fact, under minimality assumptions, each distinct head turns out to generically have a unique identifiable role. As a result, Weak-Strong Equivalence applies. The formal math behind this conclusion is shown in §A.8.1 of the Contravariance Theory paper.

There hasn’t been a ton of empirical work verifying these ideas, especially in vision networks, so this is a frontier for future empirical NeuroAI research. One observation from LLMs that would seem to be consistent with these ideas is the whole collection of “biology of LLM circuits” interpretability results suggesting that specific attention heads reliably take on certain roles. Clarifying the connection between LLM interpretability and the privileged attention heads predicted by the contravariance theory is also a natural area for future research.

So how similar are ViTs and CNNs?

The theoretical results don’t explicit say that two networks of apparently different architectures solving the same task have to end up being similar, either up to weak (linear) or strong (privileged axis) equivalence. But they do kind of raise the question: to what extent do the ideas that allow for proving weak-strong equivalence and zippering suggest that similar structures will appear across architecture types? How similar are the linear spans and privileged axes in corresponding-ish layers of CNNs and Vision Transformers (ViTs)? Are there corresponding-ish layers in the first place? After all, even though ViTs seem on the surface like they’re very different from CNNs, the ViT class and the CNN class could in principle learn similar functions after optimization for a common task. This is a great set of questions for future investigation.

Fig. 1: End-to-end optimization often gives rise to strong upstream constraints: e.g., optimizing for ImageNet categorization reliably yields a mixture of biologically recognizable monochrome Gabor wavelet and color-opponent center-surround filters, in an early V1-like layer.

Q: Why does this kind of thing happen?

One of the most striking observations in NeuroAI is that end-to-end optimization for downstream tasks often seems to strongly constrain upstream representations. And because (as we saw in Part 1 on Weak-Strong Equivalence) weak alignment of representations itself implies strong alignment of single-unit axes (assuming a hard enough task), this means that end-to-end optimization strongly constrains mechanism-level structures at a pretty fine-grained level. (And thus we find convergent evolution, as the Contravariance Principle claims; see Part 0.)

But why do we get strong upstream representational constraints in the first place? Nothing in the Weak-Strong Equivalence theorems implies this, since they start with the assumption of weak alignment in adjacent layers.

An Empirical “Zippering” Observation

Suppose you had a bunch of DNNs with the same architecture optimized for the same thing on the same dataset, but differing only in the random seed used at the start of training (so that weight initialization and training order differ). How would the representations of the different resulting fully trained networks compare?

Presumably the networks would be similar at the first layer (because they’re computed on the same images) and at the end, because they’re optimized for the same answers. But will the intermediate representations be highly constrained? Or will they “wander” far afield, finding mechanistically very different solutions away from the pinned-down first and last layers?

Here’s the answer we found in Thobani et al.:

Fig. 2: y-axis represents R^2 on held-out stimuli of ridge-regularized linear regressions between AlexNet architectures trained for ImageNet categorization from different seeds (error bars are SEM over 5 seeds). Dark green = post-nonlinearity; light green = pre-nonlinearity. (Reproduced from Thobani et al.).

Let’s focus first on the dark green bars, which represent comparisons after the nonlinearity at each layer — probably the most natural place to make the comparison from a neuroscience point of view.1 If you just looked at those layers, you’d be forgiven for thinking that the answer was “yes, lots of wandering happens”. Especially in intermediate layers 3 and 4, the representations are really quite different — even up to the “loose” weak-alignment standard of linear regression.

But now let’s look at the light green bars. These represent comparisons before the nonlinearity at each layer — that is, after the linear convolutional projection, but before ReLU. Now suddenly we see that the representations aren’t that different after all. Even intermediate layers are highly constrained to be similar. Somehow, what’s happening is that the resulting networks are quite close to being linearly aligned at each pre-nonlinearity layer. But then at each post-nonlinearity layer, the networks come apart. This alternation leads to a characteristic “zippering” (or “accordion”) pattern in which the networks’ representations repeatedly converge and diverge.

Apparently the system is not wandering that much after all. What’s going on?

With the Weak-Strong Equivalence theorems, we can interpret the post-nonlinearity divergence phenomenon: if the preactivation alignment is genuinely non-axis-preserving, and the used gates are used by the (presumably) hard task, then perfect post-ReLU linear alignment cannot (à la Part 1) persist. But we don’t see this: the next layer of affine operations, which are different for each seed, must suppress the seed-specific variability to cause re-convergence.

Of course, since invertible affine operations cannot reduce the dimension of a representation, the fact of re-convergence means that the affine operations are compressive. Specifically, the task-irrelevant dimensions have been pushed into seed-specific dimensions crossed by the used gates at the previous layer, and then are “squeezed out” by the re-convergent affine operations at the next layer. The empirical zippering observation is a visualization of the network iteratively focusing seed-independent common task-useful information into shared privileged axes at each layer.

Why does zippering happen in the first place?

BUT… what leads to the repeated re-convergence at each pre-nonlinearity layer in the first place? The Weak-Strong Equivalence mechanism does not answer this at all.

It turns out that there is a much stronger set of theorems, stated in section 4 of the Contravariance Theory paper, that explains this. The essential logic of the Weak-Strong Equivalence mechanism can be dramatically expanded to show that, under the right conditions, downstream weak equivalence induces weak equivalence at the previous layer; that equivalence, in turn, induces weak equivalence at the layer before it; and so on, “zippering” upstream (Fig. 3).

Fig. 3: The mechanism of zippering.

The intuition behind this zippering mechanism is a generalization of the intuition behind the WSE mechanism: if two networks fail to be linearly equivalent at an upstream stage, and their nonlinearities are necessary for the task, the disagreement will be amplified by the next nonlinear stage, and then by the next, eventually preventing the two networks from solving the same terminal task.

To justify this underlying reasoning and prove a zippering theorem, we need to upgrade from mere single-gate “usedness” (see Part 1) to layerwise minimality.

With this notion in mind, we can state a mathematically robust mechanism for the zippering phenomenon:

Zippering Theorem (informal). Let A and B be same-depth networks. Suppose they are weakly equivalent at a terminal layer s. Suppose further that at each backward step from s−1 down to q, the layers are minimal. Then, generically (that is, except on an exceptional set of network parameters of zero measure), alignment zippers upstream: for every r = q, . . . , s − 1, the layer-r representations are also weakly aligned.

The effect of the Zippering theorem is to make observations like those in Fig. 1 — and all those other ones throughout NeuroAI — a mathematical inevitability. That is, if one believes the assumptions the result needs.

Minimality avoids silly circuits . . .

Minimality is a nontrivial condition. It is possible for networks to be non-minimal, and non-minimality allows for all sorts of mischief to occur. The core problem is the theoretical possibility of null networks (see Vlačić and Bölcskei [2021] and Fefferman and Markel [1993]): networks that have multiple nontrivial layers, each of which is not itself the identity function, but whose total end-to-end effect is to compute the identity. Null networks are best thought of as “silly complex solutions to trivial problems”.

The existence of null networks would seem to impede the zippering theorem. For example, one might imagine inserting nontrivial identity blocks at different depths within a network, such that the end layer was the same for both but which as a pair clearly are not axis-aligned at each layer:

Fig. 4: A shifted identity block inserted either before or after G makes two terminally-equivalent networks (A and B) that have internally inequivalent representations. Minimality rules out silly circuits like this.

There are actually many different types of null networks2 but, as shown in the appendix of the Contravariance Theory paper, minimality either rules them out or makes them extremely unlikely.

. . . but is it biological?

Is minimality something that applies to real brains? Maybe it does: we speculate that minimality in real brains is an energetic efficiency that is evolutionarily advantageous. Perhaps the fact that model-brain similarity is actually observed empirically is as good evidence as any that the zippering theorems apply to real cases, and thus real brains are probably close to minimal. But this is far from an airtight argument.

It will be important going forward to evaluate the empirical status of minimality, or departures therefrom, both in “real world” artificial neural networks, and also in actual brains. Estimating the extent to which such systems satisfy the kind of nondegeneracy conditions needed for the zippering theorems is a crucial ingredient for creating a solid theoretical foundation for NeuroAI.

References

Thobani I, Sagastuy-Brena J, Nayebi A, Prince J, Cao R, and Yamins D (2025). Model-brain comparison using inter-animal transforms. Proceedings of the Conference on Cognitive Computational Neuroscience 2025. (Here is the original CCN version: you should read the Arxiv version but cite the CCN version.)

Yamins D and Nayebi A (2026). Contravariance Theory: Strong Alignment for Minimal Solutions to Hard Tasks. arXiv preprint arXiv:2607.08561.

Fefferman C, and Markel S (1993). Recovering a feed-forward net from its output. Advances in neural information processing systems 6.

Vlačić V, and Bölcskei H (2022). Neural network identifiability for a family of sigmoidal nonlinearities. Constructive Approximation 55, no. 1: 173-224.

A big question in NeuroAI is: what is the right metric for measuring alignment of neural networks to the brain? This question has been the subject of several workshop series and multiple recent papers.

Weak and Strong Alignment & the Strict-Loose Continuum

One overarching idea that has emerged from this literature is that some metrics are strict and some are loose. Strict metrics require very fine-grained matching between neural network parts and brain parts, while loose metrics require matching in a less constrained fashion.

The canonical example of a loose matching type is linear alignment — what we will henceforth call “weak alignment” — in which one fits a linear mapping between model feature bases and a dataset of “neuroids”.1 Linear alignment mapping is loose because it allows the source representation (the model) and the target representation (the brain) to differ up to linear reweighting of the constituent units.

For me, the purest (if not yet exactly “canonical”) example of a strict matching type is privileged axis alignment. Privileged axes are specific response functions that repeatedly arise within a model layer or brain area, across many seeds / individuals, and which are thought to capture the underlying mechanism of the computation in a finer-grained way than weak (linear span) alignment. Gabor-filter-tuned units in V1 are a canonical example of privileged axes in neuroscience (another being grid cells).

Because of this apparent deeper connection between privileged axes and mechanism, we will use the term “strong alignment” to indicate the alignment metric measuring coincidence between two networks’ privileged axes. The “strong” and “weak” terminology is also justified by the straightforward mathematical fact that strong alignment implies weak alignment — if two representations have the same privileged axes, they’ll be linearly equivalent, but not (usually) vice-versa.

There are a bunch of other metrics that people have thought about (RSA, CKA, IATC, etc) that kind of lie on a continuum along the “Strict-Loose axis”:

Weak-Strong Equivalence

It seems obvious that strong alignment is stronger than weak alignment. However, in the Contravariance Theory paper, our first main result is a proof that in an important case – namely, where two adjacent layers are weakly equivalent and where the task the network solves is suitably hard – the distinction between weak and strong alignment collapses.

Wait — we’re saying that two metrics of apparently very different strengths aren’t really that different. How can that be?

The core intuition is that if two networks are weakly aligned at layer ℓ, but have misaligned privileged axes, then after the nonlinearity at layer ℓ+1, they will be different enough that they will not even be weakly aligned there. In other words, if the two task representations are weakly related before and after a nonlinearity, the weak map must preserve the nonlinear signatures of individual coordinates. This is the essential mathematical reason that native axes can become privileged in the first place, rather than always remaining arbitrary.

This intuition relies on the nonlinearity having a nontrivial effect -- that is, it must be used:

_{Intuition for weak-strong equivalence: only a crossed, downstream-used zero trace set contributes a task-visible nonlinear signature. A used gate creates a slope change, and an injective affine map cannot erase that change.}

The opposite of used is “degenerate” — an axis that can be removed from the nonlinear gate requirements of the network. A degenerate gate contributes nothing downstream, while a gate whose trace is not crossed is affine or zero, and can be absorbed into the adjacent affine part. The contravariance theory paper formalizes this observation via a “shrinkability lemma” in the appendix section §A.2.

The ideas of usedness and shrinkability in turn motivate a measure of task hardness in terms of how many used gates it requires. Let L be the loss function for a task. Let F_ℓ,≤k be the set of networks of a fixed macroarchitecture that solve the task using at most k used layer-ℓ axes. Then define the required used-axis budget as:

The quantity m_ℓ(ε) is a measure of task difficulty: it counts how many nonlinear axes the task forces this layer to use.

With these ideas in mind, we can formalize the intuition described above.

Theorem (Weak-Strong Equivalence, informal): Fix adjacent layers ℓ and ℓ + 1. Suppose networks A and B are exactly weakly equivalent at both layers, and suppose the layer-ℓ axes used by the task satisfy usedness assumptions. Then every used A axis is exactly matched to a distinct B axis. Consequently, if A solves the task to loss at most ε, then

where AxisAlign_ℓ(A,B) is the fraction of axes alignable between networks A and B under the most axis-aligning linear map, and d_ℓ^A is the ambient dimension of network A at layer ℓ. In words: the aligned-axis fraction for adjacent weakly-aligned layers is constrained by the capacity ratio of the network (at that layer), m_ℓ(ε)/d_ℓ^A.

The proof of the theorem is slightly nontrivial and is given in Appendix §A.2 of the paper. But its implications are easy to understand. A network optimized to perform a task that strongly constrains it will have many or all of its gates used. Thus, the theorem says that two layers of successive weak alignment, for a sufficiently hard task, force strong privileged axis-alignment at the earlier layer.

A Softer WSE

The assumptions of the WSE theorem stated above are fairly strong: that two adjacent layers are exactly weakly equivalent. Of course, in the real world, we don’t usually observe exact equivalence. Instead, we observe equivalence with some measurable deficit:

The quantity ω_ℓ captures how weakly misaligned two networks are at layer ℓ. ω_ℓ = 0 means that two representations are exactly weakly aligned.

Critically, we can prove a kind of “soft” version of WSE that shows:

for some constants α and β, where AxisAlign_ℓ^θ denotes the number of axes aligned to similarity level θ. In words, if the error is already small on both sides of a nonlinear step, then many task-required gates cannot be badly misaligned, and the defect from axis alignment is an error term controlled by the two adjacent weak equivalence deficits ω_ℓ and ω_ℓ+1.

The key effect of this result is that an asymptotic equivalence applies for real-world situations. Imagine a sequence of networks A_n and B_n both being optimized for the same hard task. Then adjacent-layer convergence in weak-aligned representations translates into convergence in the strong axis-alignment sense:

So does the choice of metric even matter?

The WSE results make one wonder: to what extent do choices of metric matter? If the two opposite ends of the Strict-Loose continuum are actually pretty equivalent in an important case, does it matter which one we choose? Our results suggest: not as much as might have been thought. This is probably why the two “main” metrics people actually use — namely, regularized linear regression and RSA — agree in a rank-order sense, to a pretty large extent, across a wide variety of models. (Note, though, that RSA is not covered by the WSE theorem, and its divergences from that framework will be a subject of a future post.)

What needs to be explained in NeuroAI?

Why is there apparent convergent evolution between task-driven neural network models and the brain? Over the past 15 years in NeuroAI, results in a wide variety of domains (in, e.g. vision, audition, somatosensation, navigation, motor, decision making, and human language) have found that optimizing neural networks end-to-end to solve strong cognitive-level (that is, AI-hard) tasks leads to models whose internal layers mimic the neural response properties of the brain. This happens for (i) linear span of neural responses within an area, (ii) the existence of cross-area neuroanatomical hierarchies, and (iii) even the specific privileged axes at a single-unit level. Why does all this convergent evolution happen?

_{(A) Correlations, up to linear span alignment, between performance and neural alignment of models occur across domains and modalities (each dot in each panel is a neural network model). What causes these correlations? (B) Within a model, neuroanatomically consistent hierarachies emerge. Why? (C) And, privileged axes (such as the Gabor-like structures of V1) also reliably emerge. Why? In fact, why do privileged axes exist at all?...}

How does the Contravariance Principle help?

The Contravariance Principle in NeuroAI is an idea that I helped develop a few years ago while working together with the redoubtable philosopher of neuroscience, Rosa Cao (the original arXiv version appeared in 2021, but it was finally published in 2024). The contravariance principle says, basically:

Easy toy tasks will have many spurious solutions, and though your conceptually-clean hard-coded solution to it will totally work in the narrow case in which you conceived of it, that solution will be brittle and utterly fall apart outside that case when exposed to the complex situations of the real world.

In contrast, the somewhat hard-to-understand solution to the harder problem will be forced by the hard constraints of the task to look more like a real brain, which was evolved to deal with complex real-world circumstances in the first place.

This figure captures the concept graphically:

_{As you increase task constraint strength along the x-axis, the size and dispersion of the solution set to the task decreases, and the likelihood of observing “convergent evolution” between solutions increases.}

Though the contravariance principle is, at least in the initial formulation of the paper with Cao, fairly mathematically obvious, it nonetheless has non-obvious consequences for neuroscience. Specifically, it has a clear prescription:

This prescription is rather unintuitive for many (neuro)scientists, because they are used to following the standard physics-reductionist procedure of trying to simplify the phenomenon they’re interested in down to the most reduced version that captures its essence, and then completely and totally characterizing the neural mechanism for that reduced version. The contravariance principle says that, for studying solutions to complex real-world evolutionary problems (which the brain is), the normal reductionist idea will end up being very misleading.

I think it’s fair to say that, while this realization is gaining a little bit of ground in NeuroAI, it remains mostly unaccepted by (and probably unknown to) the systems neuroscience community at large.

A Mathematical Theory of Contravariance

The original version of contravariance is pretty informal, being a primarily philosophical contribution. While that status has still supported some conceptual utility for the NeuroAI community, it has for several years been on our minds to try to see if the concept could be formalized. Given the existence of quite a bit of confusion in the field, some definitive mathematical theorems could potentially be helpful.

Visions of conjectures danced in our heads, as we tried to understand not only how to define such terms as “task difficulty”, but also to imagine what types of nontrivial and useful results could likely be derived. Recently, however, Aran and I found a route in.

The basic idea is to start with what turns out to be an easier question — namely, the relationship between linear similarity of representations at the population level, and privileged-axis alignment at the individual neuron level. As detailed in the companion post Contravariance Theory, Part 1: Weak-Strong Equivalence, it turns out that these notions are tightly bound together, in the case of hard tasks — the kind of hard tasks that contravariance asks for. In fact, we find that two apparently different metrics (“weak” linear-span equivalence and “strong” privileged-axis alignment) end up being essentially equivalent, in the hard-task limit.

The technical tools and conceptual ideas needed for Weak-Strong Equivalence (WSE) allowed us to see how to measure task difficulty. With one additional ingredient — that of minimality — we were then able to see how to extend WSE results to full contravariance: a proof that for minimal solutions to hard tasks, there are very few solutions and they are essentially all the same.

Our new arXiv paper, Contravariance Theory: Strong Alignment for Minimal Solutions to Hard Tasks, explicates the mathematical basis for these ideas. We hope you read the paper — at least the initial part, which is a gentle-ish mostly non-mathematical summary. (There is also a ~70pp. appendix with all the hairy math that you might find fun if that’s your sort of thing.)

As a guide to the new paper, in the following posts — which will appear over the next few weeks — we will discuss:

• Weak-Strong Equivalence: how weak alignment of neural representations up to linear equivalence, in the context of hard tasks, forces strong alignment up to single-unit privileged axes. Along the way we will see how to formalize “hardness” in an appropriate way. [Posted 2026/07/13.]

• Zippering Theorems: How weak alignment at a terminal layer, for a hard task solved with a minimal solution, can “zipper upstream” to force full contravariant similarity throughout a network hierarchy. [Posted 2026/07/22]

• Contravariance for Transformers: How the theory of weak-strong equivalence and zippering apply to the case of transformer architectures, explaining some interesting recent mysteries in the pattern of NeuroAI results. [Posted 2026/08/05]

• Contravariance and Representational Similarity Analysis (RSA): The new contravariance theory touches primarily on linear similarity and single-unit preferred axes. But the theory provides a framework for thinking about other metrics like RSA as well. [Posted 2026/08/24]. And here is the empirical companion post.

• Contravariance vs the Platonic Representation Hypothesis (PRH): The PRH is a recent cool proposal from [Huh et al]. Our new theory sheds light on how the PRH is both related to, and quite different from, Contravariance, and maybe provides a path to PRH formalization.

• The Math Itself: A tour of the mathematical concepts behind our theory of contravariance and how they combine algebraic geometry, manifold theory, and deep learning (fairly specialist-level).

• How we used LLMs in our research workflow. The mathematical capabilities of LLMs were an integral component of our being able to make rapid, strong progress on these problems. You might find our workflow practices helpful for your own research.

We hope you enjoy the series!

Lots of smart people have started blogs and Substacks and related things in the last few years. Why am I coming to the party now?

Well, for a long time I’d been comfortable with, or at any rate resigned to, the idea of being the traditional academic: communicating with people via papers and departmental talks. I’d done a few podcasts here and there, but had largely not embraced the modern social media matrix. In fact, I would say that when I started my lab almost 10 years ago (!) I was downright disdainful of that type of engagement. Science Twitter just rubbed me very wrong: I guess it felt too flashy for me, and seemed to encourage a kind of personal self-promotion that made me feel icky.

But as I’ve gotten a little more familiar with my job, the importance of science communication has slowly, painfully, awkwardly, dawned on me. Like many people in my position, I’ve felt inexpert and frankly, ineffective, at launching ideas in today’s roiling attention economy, where papers and talks seem inadequate. And plenty of times, I see ideas discussed in science-y social media, and think, “I’ve got something good to say about that… but should I really take the plunge and put something out there?” I guess I want to influence the conversation as much as the next person, but how do I do that without piercing my self-regard as a non-self-promoter?

Actually, the main thing is that I’ve realized that I want to communicate some high-level ideas that are too “delicate” for X/Bluesky/LinkedIn and need a longer and calmer format to explicate, and too agile for the plodding cycle of peer-reviewed publications. Essentially, I think the conversation in my field — NeuroAI, let’s call it, for the time being — is ripe for disruption. I think there’s a hunger in the community for some more sophisticated treatments of intermediate and advanced-level concepts. My first substantive post, about Kolmogorov complexity for DNN brain models, is a example of this.

My plan is to try to post something substantive once every two weeks or so. I hope you find it useful!

(0) What’s the concern about compactness in NeuroAI models?

Scientists and engineers rightly want simple models of systems, because simple models are easier to understand, analyze, manipulate, and improve. Large neural network models of things are often supposed to be not simple in this way, perhaps in part because they have lots of parameters and because they are not pre-wired by hand with a specific algorithmic solution to each task. Sometimes you hear a related complaint that DNN models of the brain are “black boxes” that are as impenetrably complex as the brain they’re supposed to be explaining.

But is this charge really true?

(1) What is the Kolmogorov Complexity of DNN Brain Models?

One fairly decent2 way to measure complexity of a mathematical object is Kolmogorov Complexity (KC) — the length of the shortest program needed to generate that object:

KC applies to scientific models in an intuitive way: at a given level of data predictive accuracy, the lower KC, the more compact the theory. The ideal is a compact model: one that is low-KC, while being highly predictive of data.

Most people don’t dispute that deep net-based brain models are highly data-predictive. But are they compact? In other words: are they low-KC?

(2) Bounding the Kolmogorov Complexity of Optimization Products

Calculating KC exactly for any particular mathematical object is often pretty hard.3 However, we can get a rough picture for DNNs. The key point is that DNNs are the result of optimization, and every optimization process has four key components:

• the architecture — the function class whose parameters are optimized

• the loss function — the objective function that is minimized during optimization

• the learning rule — the algorithm by which parameter updates are computed

• the training data — the inputs on which loss is calculated during optimization

The optimization process itself is just a for loop applying the learning rule repeatedly. Thus, the KC of a DNN is bounded by the sum of the complexities of each component:4

(3) Analyzing the Terms.

So let’s look at each component separately:

1. The For-loop. Specifying a for loop is really low KC, since it’s a short string in any programming language.

2. The Architecture. The KC of deep net architectures is very low because it takes very few bits to write down any neural network architecture (like a ConvNet or a transformer).

3. The Loss Function. The KC of any reasonable deep net loss function is very low; after all, it takes very few bits to write down e.g. next token prediction loss or the contrastive-type loss functions.

4. The Learning Rule. Gradient descent through backpropagation is a very simple formula. It takes very bits to code up, even if you use a non-trivial optimizer like AdamW. It has very low KC.

So far, we have:

It’s looking pretty good. But here approacheth the rub.

5. The Data. Good data for training brain-like models comes from a unconstrained real-world system, like the text on the web or YouTube videos. What the length of the shortest description of the data generating program of the world? That’s probably pretty high.5 And though simple artificial stimuli have their place in creating model test evaluations, it is not easy to substitute these simpler controlled alternatives in place of complex real-world data during training.

This split between compactness of the rules describing how a system is created vs complexity in the “boundary conditions” of the data those rules operate on is very natural, and comes up repeatedly all across scientific discourse.

(4) But the complexity of the world isn’t the brain’s fault.

So are DNN models still compact even though KC(Data) is probably high? Yes, because the complexity of the environment shouldn’t be “charged against” (a model of) the brain. The data of the world is presented “for free” to the organism. If the constraints behind creating (“programming”) the brain-device that does the handling of the signals from the world is itself compactly describable, that is what really matters, even if the signals it operates on are complex. In other words, we are trying to understand a theory of the brain itself, as opposed to the specifics of the world the brain is situated in.

With this in mind, what we end up with is:

This calculation has two implications:

(5) Constraint-based theories (like task-driven DNNs) are beautiful.

Even though DNN models of the brain are big in the sense of having many parameters, they are nonetheless compact in an important sense, because the key components needed to construct the model itself are describable very simply. In particular, the task-driven theory of the brain compactly describes the constraints that create brains. Appreciating the beauty of a constraint-based description is an aesthetic for scientific theories that is a bit new, but we should become comfortable with it. It’s what taking ideas like Kolmogorov complexity seriously demand of us. More over, it’s a good aesthetic for understanding the output of an evolutionary optimization process (like the brain), where it’s the constraints, rather than the final evolved product, that has the best chance of being simple.

(6) Be wary of theories of “representation” or “the neural code".

Classical theories of the neural code — like the Gabor filterbank model of V1 — are very attractive, because if they can be made to work, they are even more compact than any constraint-based optimization theory. And similarly for explicit descriptions of the representational geometry of a system. But any theory concept that seeks to explain not only the constraints on the brain, but actually says that it will give a compact description of “the neural code” or “the representation” — like a closed-form formula for neural responses or tuning curves, or a detailed block-diagonal description of an RDM matrix — is implicitly saying that it will give a compact description of that data also, implying that KC(Data) would have to be low.6 That is probably not true; at any rate, we should definitely not a priori expect it.

But in any case, we shouldn’t be too down about this conclusion, because even if the “neural code” route to a theory of the brain doesn’t end up being viable, the constraint theories (see point 5 above) are viable and, when properly appreciated, very satisfying.