I dunno, are you sure it isn’t causation? Image from https://www.tylervigen.com/spurious-correlations.

These are all fine things to talk about (OK, to be fair, I do get a little tired of hearing the correlation/causation thing - trust me, my colleagues and I know!) but these criticisms are more about general science practice and the challenges of designing experiments that will reveal something important about how our minds work. However, there are also a few points my readers frequently raise that I find much more interesting because they get at some more fundamental questions about how to build a Science of Chess:

1. Why do researchers so often focus on stronger players rather than novices, for example? Is it more meaningful to understand how dramatically over-trained minds play chess, or to examine the processes involved when mere mortals play?

2. How much can we learn from big databases of chess games and puzzles? Mining the Lichess db for insights is sort of fashionable right now, but necessarily lacks a lot of context for all that data one can so easily scrape - see my last blog post for a good example of an intriguing result that lacks some important info we wish we had!

3. And finally, the one I want to unpack for you here: Isn’t it really reductive to keep using descriptors like Elo, accuracy, and centipawns to characterize performance?

Alternatively, instead of worrying about these various concerns we could also study a single GMs rating over time using a big database. Louis Hardy, CC BY-SA 4.0 <https://creativecommons.org/licenses/by-sa/4.0>, via Wikimedia Commons

This last concern is related to something I think about a lot in the lab - how do we balance (1) studying behavior using experimental settings we can control and measurements we understand, and (2) capturing the richness of everyday cognition? It’s usually straightforward to ask participants to do a proscribed task for you and measure how they performed using tools like accuracy, reaction time, and related measurements. But what else is going on in the participant’s mind? What are you missing about the way even a simple task works by reducing it a right/wrong answer and a timestamp that tells you how long it took someone to make up their mind?

Focusing on the eval bar, Elo, and other easy-to-measure variables raises that same concern that we’re missing something about the game and about how people play it. Chess is more than just a huge set of “best” moves in some gigantic parameter space of possibilities - chess is creative, a “minor art” (Osborne, 1964), and sometimes it’s even beautiful. None of that is stuff we want to miss if we’re really interested in understanding how chess works in the mind and brain. But how do we include what other writes have called chess aesthetics into the Science of Chess? How do we ask questions that aren’t just about whether you can see the best move, but also speak to what we find innovative, surprising, and beautiful on the chessboard?

Face recognition vs. Face attractiveness - A case study in scientific aesthetics

Most of my own scientific research focuses on face recognition, and I’ve written before about interesting parallels between how most people (but not all!) perceive faces and how expert chess layers visually process chess positions. This is another case where I think face recognition research offers a nice analogy for thinking about how to approach chess cognition research. Specifically, the trajectory of research focusing on facial aesthetics offers a nice roadmap for elaborating the science of chess aesthetics.

Objective vs. Subjective Face Judgments

The vast majority of my own work investigating how people recognize faces has focused on objective performance - can you tell if two faces are the same person or different people? What about estimating someone’s age by looking at their face, or deciding if a face is familiar to you or not? Using tasks like these to understand the nature of face recognition is a fairly straightforward enterprise, but like focusing on Elo, runs the risk of ignoring a lot of interesting natural behavior related to seeing faces out in the world. Besides trying to get to the “right” answers about faces, people make a lot of subjective judgments of facial appearance too! We use facial appearance to make judgments about personality traits, we try to read emotional states from facial expressions, and we decide if faces are beautiful or not.

Figure 1 from Kleinhans et al. 2023). Facial attractiveness judgments are quite reliable and are unfortunately often used as a proxy for other personality traits as well: Observers tend to think more attractive individuals will also be more honest and are more willing to behave altruistically towards them. So what makes a face look attractive or not?

But how do we start studying these kinds of judgments when there isn’t a “right” answer? Beauty is in the eye of the beholder after all, so are we sure that we can make the subjective scientific? A first place to start is making sure that there is something systematic for us to measure: Is attractiveness a meaningful perceptual property of faces, or is it so subject to individuals’ whims or personal experience that there won’t be a clear relationship between what’s in a face image and how attractive someone thinks that person is? Happily for us, there is remarkably high agreement across observers as to which faces are attractive and which ones are not (Kramer et al., 2024). This strongly suggests that though facial aesthetics are subjective, they also must depend on something that’s in the image rather than heavily dependent on the specific contents of each person’s mind.

What makes a face beautiful?

The next obvious step is to try and work out what it is about some images that makes them more attractive than others. The difficulty with this step is that images are complicated things, and even if we’re only thinking about faces viewed from the front there are a lot of possibilities out there. Does eye shape matter? What about eyebrow thickness? Surely skin smoothness has something to do with it?

The only real way forward is to make some guesses and see what does and doesn’t seem to matter. I’ll spare you the long history of ideas that didn’t quite work (the Golden Ratio turns out to not have much to do with facial beauty, for example - Rossetti et al., 2013) and leave you with some image-computable facial features that do turn out to predict how good-looking we think a face is:

• Symmetry - More symmetric faces tend to look more attractive. (but see Perrett et al., 1999 for some interesting nuance!)

• Sex-specific contrast relationships - Men have lower contrast faces that are redder, women are higher-contrast and greener, and faces of both sexes look more attractive to us when they adhere to those dimorphic features. (Russell, 2003)

• Averageness and Distinctiveness - Faces that are more typical (closer to the middle of “face space”) tend to look more attractive (Rhodes & Tremewan, 1996), though some of the most attractive faces are also more distinctive! (Debruine et al., 2007)

Figure 2 from Debruine et al. (2007) - The average of a collection of faces tends to be more attractive, but the average of the *most* attractive faces in the sample has distinctive features that make these individuals stand out in an aesthetically pleasing way.

These aren’t the only factors that affect the way beauty is perceived, but my point here is that careful experimental work across many groups and over many years has given us a decent understanding of the perceptual basis of finding faces attractive. This understanding is detailed enough that computational models can do fairly well at predicting human attractiveness judgments from photographs (Brostad et al., 2008). Beauty may be in the eye of the beholder, but at least as far as faces are concerned we can say a lot about its basis in the mind.

Identifying aesthetic principles of beautiful chess

Can we do the same for chess? While there aren’t nearly as many attempts to characterize chess aesthetics as there are studies on facial attractiveness, there have been a number of interesting attempts to characterize what makes us see some moves (or move sequences) as beautiful.

Do people agree on what is beautiful OTB?

Let’s start with the question of how meaningful a construct “beauty” is in the context of chess. Are we sure that’s a real thing people can agree on, or do different people have vastly different ideas about whether or not a particular move sequence is beautiful or not? To some extent this will likely depend somewhat on players’ understanding of the game (if you’re still learning about tactics like pins and skewers you may not get why some moves are forced, or others are ill-advised, for example) but if we’re talking about people who know the game fairly well, will they tend to agree on what is beautiful and what is not?

A recent paper (Scherbakova et al., 2025) provides a decent answer to this question as part of their attempt to develop a psychometric scale of chess aesthetics. Briefly, their instrument involves rating candidate solutions to chess problems according to six different judgments (see their table below) that are their presumed components of aesthetic quality. Each of these is its own subjective judgment, but psychometric scales are often attempts to arrive at a measurement of a complex quality (in this case, beauty) by measuring many simpler qualities that are less ineffable (or is it more effable?).

Table 7 from Scherbakova et al., 2025, listing the reliability of each component of their proposed aesthetics scale across 3 Expert players. Numbers closer to one indicate stronger agreement across judges, while numbers closer to zero indicate weak agreement.

I’ll have more to say in a moment about where one gets these simpler qualities from, but for now the key observation has to do with the numbers in the righthand column of the table up above. These are values of a statistic called Cronbach’s alpha, which is used to compare the similarity between the responses made by different judges to the same items. It ranges between 0 and 1, with numbers closer to 1 indicating stronger agreement. Overall, these values aren’t so bad psychometrically speaking, but it’s interesting that some judgments (Originality, in particular) appear to be harder to agree on. All in all, however, I’d say that this is a decent endorsement of the idea that chess aesthetics are systematic rather than random. Like facial aesthetics, chess aesthetics appears to be something that exists in a position rather than just in an individual’s own mind.

What makes some moves more beautiful than others?

If beauty on the chessboard is something we can agree on enough to study, what is its basis? What is it that gives some combinations an aesthetic appeal? The starting point for most all of the existing work aimed at answer this question is a lovely paper by Dr. Stuart Margulies. In this study, the author derives a set of aesthetic principles for chess on the basis of pairwise judgments that he asked 30 expert chess players to make. Critically, he carefully constructed pairs of chess positions that presented nearly the same situation to his players, but differed either in the specific move he presented as the solution or in the piece configuration that made the same mating move work. Faced with these two versions of effectively the same problem, Margulies asked his experts to choose the one that they thought was more beautiful. Take a look at the image below (poor scan quality aside!) and see what you think: Which checkmate do you prefer on aesthetic grounds?

Figure 1a from Margulies (1977). Given the same starting position, is it more beautiful to capture on c7, or to move the knight to b6 without taking the rook?

Of the 30 players he asked, Margulies found that 27 of them preferred the checkmate on the right. That strong bias favoring that response is further support for the idea that there is something systematic (and perhaps measurable) about aesthetic chess judgments, and also gives us a small hint about what kinds of moves strike us as more beautiful. Through what the author describes as a lot of “trial-and-error investigation” (which we sadly do not get to hear the details of), he used a large set of paired positions like this to gather more of these clues, ultimately arriving at a set of principles that capture the strongest biases he saw across his experts across all his test items:

1. Successfully violate heuristics

2. Use the weakest possible piece

3. Use all of the piece’s power

4. Give more esthetic weight to critical pieces

5. Use one powerful piece rather than many minor ones

6. Employ chess themes

7. Avoid stereotypy

8. Neither strangeness nor difficulty produce beauty

Some of these I’m sure you can understand right away (and maybe even agree with yourself!) while others are rather obscure. Personally, I find this to be part of the charm of this paper, especially because Margulies provides illustrative pairs of solutions for each to give you a sense of what he means. Consider “Use all of the piece’s power” which I admit I didn’t have a clear sense of at first. Here is a pairwise comparison that Margulies says supports this principle’s inclusion:

The solution on the left was judged as more beautiful by 28 of the 30 experts, apparently on the grounds that the Queen gets to flex more by moving further! Why should that matter? No particular reason, but the important thing is that it does. Developing the science of chess aesthetics depends on identifying modes of thought like this and it’s not a matter of asking why these preferences exist, but determining what the preferences are as precisely as possible . These minimal comparisons that elicit such strong agreement across experts are fascinating to me and really feel like they’re laying bare qualities of the game that we find exciting or satisfying even though they have no effect on accuracy or the eval bar. The challenge, of course, is to turn this collection of biases into something truly measurable.

Predicting and “discovering” beauty with a computational model

As intriguing as I find the Margulies paper for it’s simplicity and the insights delivered by the clever construction of critical puzzles, those principles are still just attempts to take a lot of data and distill it into something that offers a summary of what expert players seemed to like. One of the reasons I brought up facial aesthetics up above is that the attractiveness literature has done an excellent job at moving beyond hazy intuitions about facial beauty (”Having good proportions makes a face beautiful”) to a set of measurements we can obtain from any face (”Measure left-right symmetry by correlating the left half of the image with a flipped copy of the right half”). Compared to natural images of faces, chess positions are incredibly easy to describe - everything happening on a board can be described incredibly compactly with Forsyth-Edwards Notation (or FEN), for example. Can we use Margulies’ principles to start with something like a FEN and end up with an estimate of perceived beauty?

Perhaps the most valiant attempt to do so belongs to CHESTHETICA, a computational model of chess aesthetics that was designed to turn Margulies’ principles (and some variations of these) into concrete and quantifiable properties of moves and move sequences. To give you a sense of what this means, consider Margulies’ 7th principle up above: Employ chess themes. What he meant by this was that beautiful solutions tended to incorporate pins, skewers, forks and other familiar tactical motifs. What CHESTHETICA means by this is the following:

This isn’t even the whole list of terms in the evaluation function they use to quantify beauty in a position using this principle! My point here isn’t just to boggle your mind with a bunch of letters and subscripts - what I want to get across here is that this is the kind of thing we’re looking for: Beauty over the board translated from an indescribable sense of appreciation into a language of specific things we find compelling about aesthetically pleasing moves.

But does it work? As a sort of proof-of-concept, the authors offer a few different demonstrations that the model is able to arrive at conclusions that appear to match our own regarding chess aesthetics. One of the simplest is a comparison between critical moves that arose from positions reached in real games and the solutions to chess compositions. Given that the latter have been carefully constructed to be aesthetically pleasing and/or surprising and difficult to see at first, one would hope that the model would acknowledge that with a higher score. You can see the model’s comparison of the two below.

Figure 1 from Iqbal (2008) - sorted aesthetic values for chess problems (white bars) and mates that arose in real games (black bars).

Problems were much more aesthetically pleasing than real checkmates, though not without some overlap! The least pleasing problems aren’t quite as appealing as the most attractive OTB mates. Still, this suggests that the model is at least capturing something (however coarse) about what we do and don’t like out of the solutions to critical positions. What I’d really love to see, however, is a much larger-scale look at the aesthetic value assigned by humans to specific positions and how these relate to the values assigned by models like CHESTHETICA. To the best of my knowledge there isn’t such a database out there and until someone makes one (or a very popular chess website starts asking players to rate puzzles for aesthetic appeal! - hint, hint) we may not be able to do much better.

I will leave you with one rather neat demonstration that suggests this model captures something important about our sense of beauty over-the-board. To my reading of this example, this is a neat intersection of the computational approach to formalizing beauty with mathematical terms and the incisive behavioral work evident in Margulies’ pairwise aesthetic judgments. Without further ado, take a look at the position below which turns out to have a solution that was rated as moderately aesthetically pleasing by the model (White to Move). More specifically, there is a fork that the model detected.

Did you find it? If not, don’t feel too bad - I may have made it a bit harder for you by telling you to look for a fork. See, the thing is that the model scored this position high for aesthetic value on the basis of a fork being present for White, but it isn’t a fork that involves attacking two pieces. Instead, it’s what the author calls an “invisible fork”: By moving the bishop to e6, White attacks the rook on h3 while also threatening to mate on f5. The fork isn’t an attack on two pieces, it’s an attack on one piece and a threat to move somewhere else that will end the game entirely.

Now look, I can already hear some of you telling me you’ve heard of this kind of thing before and for all I know it even has another name. Also, this isn’t even the best move in this position! But whatever - we were trying to ignore Elo, accuracy and all that, right? What’s compelling about this from the standpoint of the model and for ongoing research into chess aesthetics is that this neat variation on the fork theme wasn’t explicitly put into the code - instead, it’s a by-product of the terms that were included. Moreover, it highlights the possibility that a stylish move isn’t necessarily the best one, or even a good one! Separating subjective appeal from objective accuracy is both scientifically interesting and potentially useful for improvers to think about - how often are we drawn to an ineffective move because we just like something about it?

None of this is to suggest that the CHESTHETICA model is complete, but it does seem to appreciate some of the same things we do and maybe even for some of the same reasons. We’ve a long way to go, but I think this is an exciting foundation for thinking more about subjective impressions of chess play.

Future directions

As much as possible I like to try and identify where I’d like to see the field go next when I write about a big topic like this one. In this case, it’s quite easy because it feels like there is a lot of work to be done in this direction! For one thing, beauty isn’t the only aesthetic judgment we make about chess: We also talk about creativity, which is it’s own difficult concept to pin down. What (if anything) is the relationship between principles of chess beauty and principles of chess creativity?

More than anything though, I think we just need more data. There are lots of existing resources if you want to analyze chess games in terms of move-by-move accuracy, move times, and other objective features of play. Likewise, there is also a ton of available data about players’ progress in terms of rating change, streakiness, and time spent on different time controls, etc. What we don’t have, however, is anywhere near that amount of data about chess aesthetics. If we really want to find beauty on the board, we’ll need to get serious about recording not just the wins and losses, but the style points, the “disgusting” moves, and the appreciation of a winning move that isn’t just accurate, but damn good-looking too.

Support Science of Chess posts!

Thanks as always for reading! If you like reading my Science of Chess posts and would like to send a small donation my way ($1-$5), you can visit my Ko-fi page here: https://ko-fi.com/bjbalas. Never expected, but always appreciated!

References

DeBruine, L.M., Jones, B.C., Unger, L., Little, A.C., & Feinberg, D.R. (2007). Dissociating averageness and attractiveness: attractive faces are not always average. Journal of experimental psychology. Human perception and performance, 33 6, 1420-30.

Fine, R. (1978). Comment on the Paper, “Principles of Beauty,” by Stuart Margulies. Psychological Reports, 43, 62 - 62.

Iqbal, A. (2012). Knowledge discovery in chess using an aesthetics approach. J. Aesthetic Educ. 46, 73–90. doi: 10.5406/jaesteduc.46.1.0073

Kleinhans, C., and N. Nicholls. 2023. “ Beautiful Inside and Out? The Role of Physical Attractiveness in Predicting Altruistic Behaviour.” International Social Science Journal 73: 613–626. https://doi.org/10.1111/issj.12422

Margulies, S. (1977). Principles of Beauty. Psychol. Rep. 41, 3–11. doi:10.2466/pr0.1977.41.1.3

OsborneH. (1964). Notes on the aesthetics of chess and the concept of intellectual beauty. *Br. J. Aesthet.*4, 160–163. doi: 10.1093/bjaesthetics/4.2.160

Perrett, D. I., Burt, D. M., Penton-Voak, I. S., Lee, K. J., Rowland, D. A., & Edwards, R. (1999). Symmetry and human facial attractiveness. Evolution and Human Behavior, 20(5), 295–307. https://doi.org/10.1016/S1090-5138(99)00014-8

Rhodes, G., & Tremewan, T. (1996). Averageness, Exaggeration, and Facial Attractiveness. Psychological Science, 7(2), 105-110.

Rossetti, A., De Menezes, M., Rosati, R., Ferrario, V. F., & Sforza, C. (2013). The role of the golden proportion in the evaluation of facial esthetics. The Angle orthodontist, 83(5), 801–808. https://doi.org/10.2319/111812-883.1

Russell, R. (2003). Sex, Beauty, and the Relative Luminance of Facial Features. Perception, 32, 1093 - 1107.

Scherbakova A, Engelhard Jr G and Bahar AK (2025) Development and validation of the scale of aesthetics and creativity in chess. Front. Psychol. 16:1545846. doi: 10.3389/fpsyg.2025.1545846

Before I get to our featured bit of chess science, I want to tell you about an opportunity for you to help us learn more about chess and human cognition. Myself and Dr. Steve Emrich at Brock University are working together on a new study about chess expertise and its relationship to other cognitive functions. We would love to get as many folks as possible taking part, including players across the ELO landscape. Whether you’re a titled player of just learning how to play, we’d be grateful if you could spare some time to help us out.

Full details and a link to the online study can be found here: https://www.emrichlab.com/wp/participate-in-a-scientific-study-of-visual-imagery-in-chess-for-a-chance-to-win-a-50-usd-gift-card/). Thanks in advance for considering, and now on to the main event...

Improvement by analysis - but analysis of what?

Some time ago, I was fortunate to be invited to be a guest on NM Ben Johnson’s Perpetual Chess Podcast to talk about winning streaks in chess. Towards the end of our conversation, he asked me a question I still think about from time to time, and it’s a question that’s always worth asking yourself if you have aspirations of improving your game: “How serious are you about getting better?”

I only just now realized how similar our overall look was during this podcast. Maybe I should go back to some brightly-colored eyeglasses.

My answer involved a bit of hemming and hawwing, followed by vague appeals to the demands made on my time by my job, my everyday duties, and my love of watching stupid videos on the internet. Look, here’s the thing: Like so many other things in life, I’d love to be better at playing chess than I am but thus far it turns out that I’m not so interested in this goal that I’m willing to carve out the time to actually work towards it with any real diligence. I have some idea what kinds of work I could be putting in to try and improve my play, but those activities are never as appealing as simply playing more games or spending my time reading books, drawing or painting something, or taking a walk.

But which activities actually do support chess improvement? If there is one piece of advice I think all improvers have heard, it’s to devote more time to analyzing your play. While it’s tempting to keep queuing up Bullet and Blitz games, I suspect nearly every chess coach you’ll meet would tell you that working through your games with a careful and critical eye is simply non-negotiable. More specifically, the idea of learning and growing from studying one’s failures is a common refrain. For example, I’m sure you’ve come across quotes like these that are intended to highlight the importance of treating mistakes as learning opportunities:

“When you lose, you have made a mistake, and that can help you learn - I never lose. I learn.” Tanitoluwa Adewumi (sourced in this NYT opinion piece: https://www.nytimes.com/2021/05/08/opinion/sunday/homeless-chess-champion-tani-adewumi.html)

“You may learn much more from a game you lose than a game you win. You will have to lose hundreds of games before becoming a good player.” -José Raúl Capablanca (Irving Chernev’s “The Chess Companion”)

“We learn the most from our aches. The 2008 World Championship loss to Vishy was one of my greatest lessons.” — Vladimir Kramnik.

I had a hard time finding good verification for some of these.

There are tons of variations on this idea out there, filling many an inspirational chess list with hope that our blunders, our misses, and our inaccuracies might be the key to deeper mastery if only we’re willing to look at them without shame. It seems sensible, right? Though I much prefer congratulating myself on the mating attack that was actually well-coordinated for once, working out what was lacking in the plans that failed is surely what I should be spending more time on.

Studying wins vs. losses - a regression analysis of ELO outcomes (Yiannakoulias, 2026)

The short article I want to tell you about here uses a straightforward statistical analysis to challenge this idea that we learn the most from failure. Specifically, the author chose to examine how three key factors might predict change in ELO over time: (1) How many games did you play?, (2) How many of your losing games did you analyze?, and (3) How many of your winning games did you analyze? The big idea here is that if failure really does teach us more than success, higher rates of analyzing losses should predict greater positive change in ELO over time.

There are two things about the data the author chose to work with that I quite liked. First, the data set in question only included Lichess players between 1600-1800 ELO. Compared to the frequent focus on GM (or other titled players’) abilities, I thought this was a nice choice and likely means we’re talking about players who are thinking about improvement a fair bit. Second, the author only investigated one month of Blitz play (3-5 minute time controls during May 2023). While I can see an argument for focusing on Rapid or Classical play instead, I think Blitz is useful for these purposes due to the increased incidence of mistakes during play (which we’d expect are opportunities for learning!). Seeing a one-month snapshot also struck me as a nice way to build in some easy potential for replication by repeating this analysis with other months and other sets of players (though this isn’t done here).

Before we do anything more sophisticated, how did this cohort of ~2000 players fare in terms of ELO change after the nearly 2,000,000 games they played during the target month? Below, you can see a messy plot of all the time-series data for ELO change for each player in the data set: On average, nothing much happened, but there are players who got better, players who got worse, and lots of variation in between. This last point is important because the author wants to try and explain that variability using the predictor variables I mentioned above.

`Figure 1 from YiannaKoulias (2026) - A superimposed plot of ELO fluctuations during the month of May 2023 for all ~2000 players in the data set used for the main analysis in this paper. Players tended to bounce around within a +/- 75 ELO band centered on their starting rating.

To achieve this, the authors used a regression model that included ELO change as the outcome we’re trying to predict and the data describing rates of game play, win analysis and loss analysis. A nice subtlety that’s worth pointing out here is that win percentage was also included in the model as a predictor variable. But why? Don’t we already know that winning more games means your ELO will go up? In a word, Yes! We DO know that and including win percentage in the full model is a way of trying to remove the inevitable growth in ELO that’s just due to winning more. The result is that any meaningful relationships we find between ELO change and the other predictors are less likely to just be artifacts of some players racking up more W’s - the author has statistically controlled for that contribution.

Speaking of those other meaningful relationships, it turns out that there are a few worth talking about and the table below summarizes the key results. If you’re not familiar with the numbers you tend to find in tables like these, the two things I’d suggest you look at to get a feel for what’s happening are the p-value (labeled Pr(>|t) in the rightmost column) and the t-value in the 3rd column. To do some grave violence to how you interpret regression results, the p-value tells you how likely it is that the relationship between each predictor and the outcome variable (ELO change) happened by random chance. This means that a small value (a low probability that random chance is responsible) signals more confidence that there was a meaningful relationship between the predictor and the outcome. For our purposes, sign of the t-value (whether it is positive or negative) tells you if that relationship was itself positive or negative: A negative t-value means that a larger value for the predictor (say, more games played) led to a smaller value for the outcome (poorer change in ELO over time).

Table 2 from YiannaKoulias (2026) - Analyzing Wins positively predicts ELO change, but Analyzing Losses does not! Maybe it’s not so valuable to study your mistakes.

With that in mind, there are two main points to take away from this table. First, let’s start with our idea that studying your losses makes you better - learn from your failures! Let your blunders teach you the hard lessons! Unfortunately, it looks like there is little evidence that these aphorisms are true. That large p-value (it only gets as big as 1) means that we shouldn’t have much confidence in the relationship between analyzing losses and ELO change (good or bad). Studying losses more didn’t predict greater ELO growth.

But what about studying your wins? That’s our 2nd row in the table and...well, things look a lot more interesting. The very small p-value and the large, positive t-value suggest that reviewing your victories likely does meaningfully predict a more positive ELO change. That is, study your brilliancies more often and it looks like you get better!

There is an interesting caveat however, that has to do with those additional terms labeled ‘wins2’ and ‘wins3’ in the table. These are transformations of the “wins” data into higher-order polynomials (introducing a predictor that is the original “wins” data squared and cubed, respectively) that the author only included because the linear term was significant. Using the model the author estimated from this data, we can look at the predictions for what will happen to ELO over time given different rates of analyzing your wins, and this lets us see what these non-linear terms mean more easily.

Figure 2 from YiannaKoulias (2026) - Analyzing more of your wins is related to more positive ELO change, but only up to a point. Players who analyzed the largest fraction of their wins (the yellow 24% line above) had a rise-and-fall over time that isn’t evident for players who analyzed fewer wins.

Briefly, analyzing your wins helps up to a point, but it looks like eventually the effect of doing so kind of saturates out. Players who analyzed 1 out of 8 wins (the blue line above) got more bang for their buck in the form of consistently increasing ELO as they played more. Players who analyzed twice as many of their wins (nearly 1 in 4, corresponding to the yellow line), however, exhibited a rise and fall indicating that this extra effort didn’t translate into a higher rating.

The second caveat is a more important one about the methodology: The author did not have access to who requested the analysis, only that an analysis was requested. This means that it is possible (and likely) that there are instances in the data where the winning player did not request the analysis themselves. To quote the author’s discussion of this point: “...some players will be characterized as having analyzed their game without having done so. Importantly, both of these issues bias our results to the null; that is, whatever effect we see is likely an underestimate of the true effect. This is because mischaracterizing players as having analyzed their game when they have in fact not analyzed their game adds statistical noise to the analysis variable that weakens associations between analysis and performance.”

From my perspective, it would obviously be better to know who asked for the analysis, but I also get the argument the author is making here. Even if you are uneasy about this aspect of the methods, how do you explain the observed effects (and the observed null outcomes)? There are probably some ways to tell a different story (and if you have one, I’d be curious to hear it in the comments), but I think it’s a little tricky. That said, this is also an interesting case where the nature of the available data limits what we know about behavior: Big databases make a lot of things possible, but also put you at the mercy of what’s in those repositories. Still, I think there is still an intriguing implication from this analysis despite these limitations: Studying your successes may be more effective than studying your failures.

But why?

As I was just finishing this article and preparing to blog about it, I happened to see a banner ad on chess.com for their Advanced Stats tools that highlighted the option to look at all of your brilliant moves. In light of the results I had just finished taking in, I decided that reveling in my best moments over-the-board wasn’t a terrible idea after all and dialed mine up.

What struck me about my own (rather paltry) collection of Brilliant moves is that there were a number of instances in which I definitely didn’t see all the way to the reason that my Brilliant move was a good idea. Conversely, do you want to know something that’s frequently the case about my blunders? I know EXACTLY why they were a mistake: I hung a piece, I missed a fork, or I completely ignored a pending mate. These kinds of mistakes happen in Blitz when you’re at my level and while it’s important to work towards making fewer of these mistakes, it’s not like there’s a deep concept I need to learn by reviewing them. To put it more bluntly, I always know that my Blunders are bad, but I don’t always know that my Brilliant moves are good.

This paper isn’t specifically concerned with single moves like Blunders and Brilliant moves, but I think the difference between analyzing wins and losses may work similarly. Being a fairly mid player who would like to get better, I can almost always tell you why I lost because the reasons are often embarrassingly simple (see my list of blunders above!). Oddly, however, I can’t always tell you exactly how or why I won. Sure, sometimes it’s down to the clock, or a consequence of my opponent making a conspicuous mistake of their own, but sometimes I’m as surprised as anybody that I ended up in a position that absolutely constricted my opponent: When did that happen? Which of my choices led to the superior position? How do I do that again? Some of the skills I need to develop have to do more with cultivating a win rather than avoiding a loss. Maybe that makes reviewing and remembering success critically important.

Obviously there isn’t a single “Do this, not that” answer to how to improve or how to study. What I think is compelling about this paper, however, is that it uses some simple tools to suggest that there may be some gaps in the conventional wisdom about failure, success, and what we really need to learn to be better chess players. With that in mind, I’m going to take a look at the last absolutely smashing victory I had and make sure it wasn’t a total accident.

Support Science of Chess posts!

Thanks as always for reading! If you’re enjoying these Science of Chess posts and would like to send a small donation my way ($1-$5), you can visit my Ko-fi page here: https://ko-fi.com/bjbalas - Never expected, but always appreciated!

References

Yiannakoulias, Niko. (2026). Learning From Wins and Losses: An Analysis of Improvement in Online Speed Chess. Simulation & Gaming. 10.1177/10468781261443352.

We can think about chess improvement at a grander scale, however. The best players of each era push the game forward into new directions, sometimes by offering specific opening novelties that force others to explore uncharted territory, sometimes by introducing new strategic approaches to the game that represent alternative ways of thinking (e.g., Nimzowisch and the hypermodern approach or Steinitz’s contributions to positional play), and sometimes by refuting old ideas that end up tossed in the dustbin of history. Chess is always changing and each innovative idea, no matter how large or small, teaches us more about how the game works and makes the games of our era look different than the ones that came before.

Of course, the way that change happens at this historical scale has itself been subject to change! For the vast majority of the game’s history, advances came from individual players (or sometimes “schools” of players, both formal and informal) studying the game, experimenting with new ideas on their own or with a set of analysis partners, and occasionally debuting their innovations in dramatic fashion. This could sometimes backfire: Capablanca famously refuted the Marshall Gambit over-the-board in 1918, for example (though further study led to its adoption into grandmaster praxis, where it remains to this day).

Capablanca’s annotations on his 1918 game with Marshall in which an attempted innovation (the Marshall Gambit) was not enough to win. In time, Marshall’s idea was re-examined and better continuations than he attempted in this game led to the adoption of the gambit in master-level play.

Presently, however, this process looks much different, thanks to the availability of chess engines that can evaluate arbitrary positions much more effectively than the best human players. Engines like Stockfish have an “understanding” of the game (yes, I think the scare quotes are appropriate here - you can debate me in the comments if you like!) that is sufficiently deeper than ours that the phrase “computer move” has become common parlance to describe moves that may be optimal, but are so difficult to understand that it’s unlikely a human player would ever come up with them. Though playing against the strongest instantiation of Stockfish feels a lot like being slowly (and sometimes quickly) crushed to death, it’s obvious that it offers players at every level an unparalleled tool for exploration and improvement.

The kind of computer analysis that’s incrediibly easy to come by these days. This is produced by DroidFish, an app based on the Stockfish engine. Peter Österlund, Tord Romstad, Marco Costalba, Joona Kiiski, GPLv3 <http://www.gnu.org/licenses/gpl-3.0.html>, via Wikimedia Commons

But what exactly have chess engines done for the game? You can find a number of quotes from modern players like Magnus and Gukesh describing how their play has evolved as a result of training with Stockfish, but without some real detail this is what I would call “anecdata.” Besides, even if we accept that specific players have changed their game due to working with a powerful engine, can we say something more broad about whether or not chess engines have more broadly influenced chess in a measurable way?

That’s the goal of the target article I’ll discuss with you here. Specifically, the authors set out to determine if there was a measurable effect of what they call the two “AI Chess revolutions” that occurred (1) in the late 1990s (corresponding to the commercial availability of more power desktop machines) and (2) in the late 2010s (corresponding to the arrival of deep-learning approaches to chess engine development). To put it plainly, did the quality of chess rapidly change in response to these technological developments?

A quick proof-of-concept: AlphaZero and the sudden onset of better moves in Go

Here’s a bit of context I found really fascinating in this paper: There is existing evidence from what we’d call “natural experiments” that this kind of sudden change in gameplay has happened before in response to tech advances. To stick with chess for a moment, West German players in the 1980s who had access to chess engines (albeit not spectacular ones) improved their play more than players in the Soviet Union who weren’t able to incorporate engines into their training regimens.

Similarly, the impact of AlphaZero (a deep-learning model capable of improving its gameplay by self-play) on Go gameplay was also revealed by the same kind of natural stratification of players. In this case, Shin et al. (2021) observed that Korean Go players serving in the Army (and who thus couldn’t access AlphaGo‘s games or the engine itself) showed less gameplay improvement compared to Korean players who did have access. Moreover, a granular analysis of Go move quality and novelty over time (Shin et al., 2023) also demonstrated a clear uptick in these metric that corresponds to players having access to AlphaGo’s play (see figure below).

Adapted from Figure 3 in Shin et al. (2023) - The novelty of human moves was declining steadily for decades, then took a mighty (and statistically meaningful!) leap after AlphaGo defeated the reigning World Champion.

Together these results make a compelling case for examining chess play at a similar level of granularity: Did chess players start making better moves once they had access to strong chess engines? Can we see the impact of sophisticated computer analysis on the quality of play?

Using Change Point Analysis to see if (and when) something changed

This kind of quantitative question - can we identify a moment in time when things changed? - turns out to be an interesting and challenging thing to think about statistically. As a cognitive scientist, I’m often very interested in quantifying the effects of change over time (I study the development of the visual system during childhood) and sometimes use data that’s densely-sampled in time (like eyetracking or EEG data) to measure responses at critical moments. The thing is, I’m almost always using tools that assume I either know how to identify a time range I care about, or that rely on talking about the shape of the entire trajectory of performance across a span of time. What do you do if you don’t know when (or if) a critical moment might be evident in your data?

Consider the data I’m showing you below, which is a plot of the volume of the Nile River year-to-year across a long-ish span of time. How would you go about finding out if something had ever happened during this span that marked a pivotal change in the river’s output? If I had a good idea about when that specific point in time happened (maybe I have a hunch that the dashed vertical line marks an important historical moment), I could split my data into two groups, one before that moment and one after, and use any number of simple tests that allow you to compare two samples to see if they differ. There are cases where you might know this, but if we’re talking about chess and the impact of computers, something like the availability and use of desktop PCs that can handle stronger chess engines is tough to localize in time. When exactly did that happen? How do we pick a moment **a priori **to guide our statistical testing? To be honest, if we really care about finding the crucial moment for change, we shouldn’t try to pick it out ourselves - we should let the data tell us if it’s there.

Image credit: van den Burg, Gerrit JJ, and Christopher KI Williams. “An Evaluation of Change Point Detection Algorithms.” arXiv preprint arXiv:2003.06222 (2020).

To get closer to the kind of answer we want, you might try fitting some kind of function to a graph like this one so you have a structural description of how your variable changed over time. This is also fairly easy to do and there are plenty of tools for either fitting one curve to this data or finding the best “mixture” of curves that accounts for the shape of the graph. The problem is that these approaches also make it difficult to talk about punctate moments in time due to the fact that usually the curves we fit to data like this don’t incorporate sharp “spikes” (or to be more technical, non-linearities like a delta function) in their functional form. This means that we might get a nice description of our data in terms of a combination of gradually unfolding contributing processes, but these won’t be great for working out when something suddenly pushed the data around at one critical moment.

Luckily for us, statisticians spend a lot of time developing new methods to answer specific quantitative questions like this. Tools for identifying “change points” in time-series data have been around for some time and I found it a sort of neat rabbit hole to explore for a bit as I was preparing to write this piece. One of the more compelling demonstrations I could find of identifying a critical timepoint where a dataset changed was an investigation of coal mining accidents from the mid-19th century to the mid-20th century. I grew up near coal country in Pennsylvania and there’s a family story involving my great-grandfather having to dig men out of a collapsed mine he owned, so I have a little bit of generational familiarity with the dangers associated with digging chunks of coal out of the ground. But how did those dangers change historically? Here’s a cumulative graph of coal-mining accidents over time - because it’s cumulative it will keep going up (flat is the best we could expect to see!), but our question is whether it looks like its rate of climb starts to slow down all of a sudden somewhere along the line. Do you feel like you see anything? Did coal-mining ever get suddenly safer?

Figure 1 from Jarrett (1979) depicting the cumulative count of coal-mine explosions over the course about a century. Is there a critical moment when the curve begins to flatten? If so, when does it happen and can we work out what may have caused it?

Of course I don’t expect you to have a perfect answer and even if you did we can’t just defer to an eyeball test! Instead, the key statistical idea is to identify two independent functions: One that we’d like to use to account for the data before some critical moment and another that’s only intended to account for what happens afterward. The important step from a stats standpoint is to include that critical moment (which we can just call “t”) as something we’re going to estimate from the data along with the other parameters that describe our “before-t” and”after-t” functions. In the paper that analyzed this coal-mining data, the authors end up with a new graph that indicates the likelihood that “t” happened at each timepoint in the original series.

Figure 1 from Raftery & Akman (1986). This plot shows you the probability density that a critical change-point in the cumulative graph I showed you up above happened at each year in the original data. That tallest spike suggests something important happened around 1889 that made that curve start to flatten meaningfully.

While there are a few serious-looking spikes in this graph, there is also clearly one that is head-and-shoulders above the rest. What happened between 1888 and 1892? This data about coal-mine explosions suggests that something changed that affected worker safety profoundly, so what’s in the historical record? In a word: Unions. The Miners’ Federation was formed in 1889, advocating strongly for improved working conditions in coal mines and apparently meeting with reasonable success! I realize this is a bit of a tangent, but I hope this gives you a sense of how this version of statistical analysis makes it possible to try and link specific moments in history to data that we’ve measured over it’s sweep.

Have the two AI revolutions in chess impacted the quality of play (Bilalic et al., 2024)?

So what about the history of chess? Is there statistical evidence for “change points” associated with the presumed revolutions in the late 90s and late 2010s that the authors propose? To answer this question, the authors developed a dataset obtained from ChessBase’s “Mega Database 2023“ - a repository with 10 million games spanning the years 1985-2021. First, they identified the 20 best players in each year overall (a cast of characters that can and does rotate, mind you!) and also put together separate Junior (under 20) and Senior samples (over 65). Next, for each year they collected all of the games the top players took part in (only including tournament play - no blitz, no bullet) and examined all of the moves they made (excluding the first 10 moves and everything after move 60).

From here, the big goal is to treat the average quality of these moves in each year like time-series data that provides a historical record of how good humanity is at playing chess: How accurate are players’ moves over this time span and do we see evidence of rapid change in the neighborhoods we might expect given the technological developments we know about? The authors use a few different metrics to characterize move quality, including centipawn loss and accuracy, both of which are measures of performance you may have heard of before. If you’re not familiar with them, Accuracy is quite simple: How often did your move match the optimal move? As for centipawn loss, a centipawn is a unit of measure (equal to 1/100th of a pawn) for estimating the advantage one player has over another in a position. Having one fewer pawn than your opponent costs you a full point, but a difference in position even when material is equal can mean one player is better off than another by a smaller amount. For example, a +0.1 evaluation in your favor means that the arrangement of pieces on the board is better for you, to the tune of 1/10 as good as being a pawn up.

By extension, when we talk about centipawn loss as a measure of accuracy, we’re referring to the difference between your advantage if you had made the best move (determined here by Stockfish 16) and your advantage after making a different move. The figure below shows you what these various measures of accuracy look like when we calculate the average value over time for the three groups of top players (Centipawn Loss is at the bottom left, with Accuracy and Optimality in the top two panels, left to right).

Adapted from Figure 2 in Bilalic et al. (2024). A range of performance metrics including Accuracy, Move Optimality and Centipawn Loss plotted over time in the critical period for the Top 20 players overall (in Red), the Top 20 Junior players (in Blue) and the Top 20 Senior players (in Green). The authors model these trajectories with a mixture of continuous functions, but that approach does not allow inferences about critical time-points corresponding to sudden change.

Do you think you see anything? Is there a spike or two somewhere in there that signals either the onset of home computers with solid chess engines or the availability of deep neural-net models? If you’d like a visual aid to help you get a sense of what could be lurking in there, the graph below shows you how the strongest chess engine rating changed during this same time span. This pretty obviously has a marked change point right where you think it should - but is that reflected in the human data at all?

Adapted from Figure 3 in Billalic et al. (2024) - A plot of how the ELO of the best chess engines changed during the critical period examined in this study. The authors suggest that one critical point for potential human improvement corresponds to increased access to these models in the late 1990s (even while AI ELO was improving slowly) and that the second corresponds to the sudden rise in ELO subsequent to advances in applying deep networks to chess in the late 2010’s.

The graph below shows you that the answer is more complicated than you’d think. What you’re looking at here is a plot much like the spiky figure I showed you above to indicate when the critical moment happened in the coal-mine disaster data, only now with a change-point analysis for each of our three player groups (Overall Top 20 at left, Senior Top 20 in the middle, and Junior Top 20 at the right). What you’re looking for here is the thin blue line underneath those densely packed black dots. Peaks in that blue line indicate strong evidence for a change point at that moment in time.

Adapted from Figure 6 in Bilalic et al. (2024). A change-point analysis of Accuracy and move Optimality both reveal critical time-points for improvement in the Top 20 Junior and Senior players, but these occur at different times. The Top 20 Overall players give no indication of sudden change in response to either presumed “AI chess revolution.”

The fact that you see some blue peaks on the bottom of some of these graphs means that there are some meaningful change points here, but it depends on what player group we’re talking about! If we look at the Top 20 players overall (these are the top-flight experts in the database), there isn’t really a sharp change in move quality at all, save for something happening right at the beginning of the sample. That is, neither of the two presumed “AI revolutions” changed the trajectory of improvement for the best players!

Breaking things down by age, however, reveals something interesting: Junior and Senior players have different critical timepoints in their data. The Top 20 Junior players do show statistical evidence of a sharp jump in accuracy in the late 1990’s (remember, this is the arrival of home PCs that could support strong chess engines) but nothing else! On the other hand, the Top 20 Senior players show evidence of an accuracy jump in the 2010’s (when deep nets like Stockfish show up). I don’t want to drown you in figures more than I already have, but this pattern of results is stable when you break things down into White and Black moves, too, which is a nice way of looking at the reliability of the results in all three groups.

But what about Magnus? (or Gukesh, or...<insert your favorite GM>)

You might be a little troubled by the finding I highlighted in bold above - what do you mean access to strong engines didn’t kick-start improved play at the top levels of chess? I mentioned at the outset that some of the best modern players have explicitly said that access to state-of-the-art chess engines changed their game. Just to underline this point, here’s a quote from Magnus Carlsen that the authors of this study included in their paper:

“Yes, I have been influenced by my hero AlphaZero recently. Essentially I have become a very different player in terms of style than I was before, and it’s been a great ride.”

So what are we to make of this kind of statement in light of the data I just described? Well, what if we took a look at Magnus’ play during this critical period of chess history to see what his various trajectories look like? The authors did just this as a sort of “case study” and you can see their plots of his performance over time on several different metrics.

Adapted from Figure 8 in Bilalic et al. (2024). Magnus Carlsen’s various performance metric including Accuracy, Centipawn Loss, etc. do not carry evidence of a change-point corresponding to the adoption of AlphaZero into his training.

The bottom line? Perhaps Magnus’ style of play changed dramatically after working with these engines, but his performance metrics don’t show evidence of non-linear improvement anywhere to be found. There are a lot of reasons to not think of this as terribly definitive, but I appreciated the attempt to put one player under the microscope statistically-speaking to see whether we could see some special point when things changed. In Magnus’ case, we can’t.

Conclusions and next steps

Of course, saying that engines like Stockfish haven’t affected the historical creep of chess improvement doesn’t mean that they haven’t been incredibly useful tools for the vast majority of players. I obviously benefit from being able to work through post-game analyses that can identify my blunders and show me all the better choices I could have made throughout a game. Still, what I think the data described in this paper show us is that following in the footsteps of these engines is much harder than it seems. I mentioned earlier that we talk about “computer moves” that seem almost alien in their strangeness in contrast to “natural” moves that we know human players will gravitate towards. To me, this suggests that even the best players have perhaps only been able to glean some hints from our computational overlords of how the game might be played differently, but wholly adopting the viewpoint of our strongest engines remains elusive.

Support Science of Chess posts!

Thanks as always for reading! If you’re enjoying these Science of Chess posts and would like to send a small donation my way ($1-$5), you can visit my Ko-fi page here: https://ko-fi.com/bjbalas - Never expected, but always appreciated!

References

Bilalić, M., Graf, M., & Vaci, N. (2026). Computers and chess masters: The role of AI in transforming elite human performance. British Journal of Psychology, 117, 585–609. https://doi.org/10.1111/bjop.12750

Choi, S. , Kang, H. , Kim, N. , & Kim, J. (2023). How does artificial intelligence improve human decision‐making? evidence from the ai‐powered go program. arXiv, 231008704.

Graf, M. , Danek, A. H. , Vaci, N. , & Bilalić, M. (2023). Tracing cognitive processes in insight problem solving: Using GAMs and change point analysis to uncover restructuring. Journal of Intelligence, 11(5), 86

Jarrett, R. G. (1979). A Note on the Intervals Between Coal-Mining Disasters. Biometrika, 66(1), 191–193. https://doi.org/10.2307/2335266

Raftery, A. E. , & Akman, V. E. (1986). Bayesian analysis of a Poisson process with a change‐point. Biometrika, 73(1), 85–89. [Google Scholar]

Shin, M., Kim, J., van Opheusden, B., & Griffiths, T. L. (2023). Superhuman artificial intelligence can improve human decision-making by increasing novelty. Proceedings of the National Academy of Sciences of the United States of America, 120(12), e2214840120. https://doi.org/10.1073/pnas.2214840120

During the 2026 Candidates Tournament, the moment that stood out the most to me wasn’t actually a moment, though - it was a LOT of moments! Specifically, it was Hikaru Nakamura thinking for just under 68 minutes during his Round 5 game with Javokhir Sindarov. Besides living up to the jokes my wife usually makes when I’m watching live coverage of a tournament (”Did one of them shift in his seat? Did he start to reach for something??”) I can’t help but find this remarkable from a cognitive perspective. Partly this is because my own calculation abilities are very limited, so seeing a player spend so much time on a single move hints at how much more recall and computation they’re capable of. Even if Hikaru ended up making a mistake after that deep think, it’s still apparent that he’s got much more at his disposal than I do in terms of working memory, possibly visual imagery, and visuospatial processing.

Image credit: FIDE.com.

The other thing about this moment that I found interesting is that it highlights the role that metacognition, or thinking about thinking, plays in decision-making. Taking time to think something through carefully can be useful, but it also has its drawbacks: Thinking takes time, thinking uses metabolic resources, and depending on what you’re thinking about it may be more or less pleasant to keep turning your options over in your head. This trade-off between the benefits and the costs of having a long think means that not only do you have to make decisions out in the real world, you also have to decide how you’re deciding.

The constraints imposed by a game of chess make this trade-off especially stark. The time it takes to come to the best conclusion isn’t abstract, but is instead a very real resource that can decide the whole game. Those metabolic costs start to feel very real during a long game as well: Spending a lot of time on a single move may leave you feeling tired, hampering your ability to make good decisions later. Chess players thus have to engage in a process of thinking about the game, but also evaluating their own thinking: When should you dig deep and calculate carefully? When is it better to just do something?

Good metacognitive decisions over-the-board depend on knowing when it’s worth it to spend resources (your time and your energy) on extended thought. But can chess players tell when a position that they’re considering will benefit from more thinking? The study I want to tell you about here was designed to answer exactly this question: Do players of varying strength allocate their calculation time in ways that make sense given what’s on the board? Even though Hikaru ended up making a mistake after his awe-inspiring think, the time he spent might still have been the right call, after all.

How do you objectively measure when a position deserves more thought?

The authors’ big goal in this study was to work out how well players of different abilities and across different time controls scaled their thinking time to the value of thinking more in real chess positions. That means they had two measurement problems to think about, one that’s quite easy and another that’s tougher. The easy part is measuring how long players think about a given position and the solution is fairly obvious: Let’s just use move time! Granted, this is not a perfect proxy for time spent thinking. Mind-wandering is a very real phenomenon with its own interesting research literature and players of all abilities certainly spend some time off-task even in crucial games. All the same, this is an easy thing to measure and certainly reflects players’ decision to consider a move more or less carefully.

The second thing the researchers had to measure was the value of thinking more in a particular position. This is a question about the state of the board rather than the players’ understanding of it (though hold that thought!) but requires a model of what outcomes you’d expect if you spent less vs. more time considering your options. Luckily, modern chess engines (in this case, Stockfish) are an excellent way to estimate this. Here, the authors used a simple means of quantifying the value of deeper thought using Stockfish assessments of the position at different depths of processing. If you’re not familiar with what this means, the classic definition of the term comes from algorithms that explored tree-like representations of how a game might proceed from a starting position. In these algorithms, depth specifically meant looking more possible steps ahead (or further down the tree) to more fully assess the potential consequences of each candidate move. The numbers associated with different depths in these models often referred specifically to a ply, or half-move, in a game.

An example of the full game tree for a much simpler game, namely Tic-Tac-Toe. In algorithms that do simple tree search, more depth means looking at outcomes that follow from more moves, moving you further down (here, across) the tree. A move that looks alright at a shallow depth might have negative consequences if you keep going. Image Credit: Mike like0708, CC0, via Wikimedia Commons

What depth means numerically in Stockfish and similar algorithms for assessing chess positions is not as simple as that, but it is still fair to say that a smaller (or we might say shallower) depth offers a less comprehensive look at possible outcomes than a greater (or deeper) search. In the current study, the authors used a fairly big difference in depth to estimate the value of extended thinking. First, they asked SF for the best move it could come up with using a depth-1 search. Next, they asked it to produce the best move at depth-15. With those two candidate moves in hand, they estimated the value of thinking more by asking depth-15 Stockfish (the harder thinker) to provide it’s usual numerical evaluation of each position. The difference between how well things turned out after making the higher depth move and the outcome from making the lower depth move is their final estimate of what thinking is worth.

Figure 2 from Russek et al. (2025). The authors asked Stockfish to come up with the best move at Depth 1 and Depth 15, then evaluated the position after each move using the higher depth. The bigger the difference between those evals, the more value there is to spending time on the position.

You can see in the figure above how this may shake out for different positions. On the left side of the figure, we see an example of a position in which the shallow move (Qxf2) leaves the position pretty much even while the deeper move (Qf5) gives Black a small advantage. On the right side, however, is a case where both the shallow and the deep search reveal the same best move: Take the horsey! In this case, there isn’t any value at all to thinking further because it doesn’t get better than the first thing you see.

I thought this approach to quantifying the value of deeper thought was a great choice and it also sounded rather familiar! I’ve written before about a similar approach to estimating what moves deserve to be called “Brilliant.” The key difference between what’s happening here and using similar depth-differences to predict perceived brilliancy is that the current study is leveraging the full range of differences between shallow and deep search. “Brilliant” moves specifically look bad at a shallow depth but look good if you look deeper. Here, the authors are interested in all of the numerical differences between a brief think and thorough one to see if these predict how quickly players make choices.

How does the time control affect players’ ability to choose when to think?

Give these two measurements, the time players take to move and the estimated value of thinking more, does the data suggest that players spend their time wisely? The researchers used a large-scale database of Lichess games (previously used to develop and test the Maia chess engine) that included half a billion moves from over 12 million games. These moves were taken from games at a wide range of time controls (which are reported in seconds rather than minutes in the text), from players spanning a wide range of playing strengths, and also limited to approximately the middlegame (between moves 7 and 37). This latter feature of the data used here is particularly important because it helps avoid any over-learned opening or endgame patterns that may contribute to very fast moves overall.

With all of this data in hand, the researchers were able to ask whether variance in move time was explained by their thinking-value variable. In the figure below (taken from their paper) you can see what this relationship looks like for different time controls, including Bullet, Blitz, and Rapid games with and without increment.

Figure 3 from Russek et al. (2025). Each panel is a different time control (reported in seconds, so Bullet 1+0 is in the upper left) and the line graphs show the relationship between the average time players’ spent thinking about moves (y-axis) and the value in spending time thinking (x-axis).

I hope it’s evident that the main story from this figure is that players do indeed spend more time thinking when the position deserves it, or else these lines wouldn’t show such a clear increase in move time as the benefit to thinking increases. We can be a little more nuanced though and take note of some more granular features of the data. For one thing, there’s a statistically meaningful flattening to this curve as both variables increase, which signals that players appear to be most sensitive to the difference between no-advantage and some-advantage to further thought. I found this interesting as a vision scientist because it reminds me a little of the data we often get out of psychophysical experiments. In those tasks we often see evidence of a relationship called Weber’s Law which asserts that the smallest difference you can perceive between two stimuli gets bigger as the intensity of the stimuli increases. You can see a nice illustration of this in the figure below: In both the left and right half of the figure, there are 10 more dots in the bottom image. This is MUCH easier to see when we’re talking about 10 vs. 20 compared to when we’re talking about 110 vs. 120 however!

Image credit: By Д.Ильин: vectorization - File:Weber-Fechner law demo - dots.png by MrPomidor, CC0, https://commons.wikimedia.org/w/index.php?curid=113926988

The shape of the curves (which I’d call logarithmic) in the authors’ figure looks a lot like the kind of data we often get from sensory threshold experiments governed by this law, and it’s consistent with the idea that the benefit of thinking is working a bit like a perceptual variable. I’m not trying to argue that this is a real act of sensation or anything, but it’s still interesting to see the form of this relationship match data from other kinds of detection tasks. Another feature of the data I thought was striking was the dramatic increase in the width of the error bars as the time control lengthened, and especially when increment is available. In particular, the immense variability at the 30+20 time control relative to 30+0 is hard to ignore, and says a lot about how increment factors into players’ metacognition: Even a little time bonus every turn makes players much more willing to sometimes spend some resources on thinking.

Does expertise lead to better decisions about deciding?

Besides examining the role of time control on the relationship between thinking benefit and move time, the authors were also able to slice their analysis up in terms of playing strength. This is especially interesting in that you could easily imagine that stronger players have better ideas about when it’s worth thinking through a position (estimating their own thinking-benefit variable) and thus are able to allocate time in a way that reflects what they might get out of thinking longer. By comparison, weaker players might not adequately sense the potential benefit (or potential peril!) that follows from spending more or less time on a particular move and thus show less sensitivity to thinking-benefit in their move times.

The figure below unpacks this for you, but it’s worth spending a little time explaining what’s being plotted. In essence, you’re about to look at graphs that show you a relationship between relationships. Let’s go back to the last graph for a moment - this included a curve for each time control that reflected how move time and thinking-benefit were related to one another. If we wanted to know what this curve looks like for players with different ratings, we could make separate lines for players of different strengths: One curve for the 1000-1200 ELO crowd, for example, and another one for the 1201-1400 ELO players, and so on. Instead of just looking at those, however, we can also measure the strength of the correlation between our two variables quantitatively. The authors did this using a value called Spearmann’s rho, which is a way of measuring correlations using the rank of each data point rather than it’s actual variable. This helps accommodate for non-linear shapes to our data like the logarithmic curves we saw in the authors’ Figure 2. Once we compute those values, we can make a new graph of how Spearmann’s rho changes with ELO as a function of each time control.

Figure 4 from Russek et al. (2025) - Spearmann’s rho values for the move time vs. thinking-benefit correlation as a function of playing strength (ELO rating) for the full range of time controls in the data set.

Again, this is a case where the main story is pretty clear right away. As playing strength goes up, so does the strength of the relationship between move time and the objective value of thinking. That is, stronger players’ move times are more closely related to the possible benefit of thinking more. This graph gets a little depressing as you start scrutinizing what’s going on down at the low end of each graph honestly. At 1000 to 1200 ELO, player’s really don’t spend more time when they should (or less time when they don’t need to) very reliably at all! What I found remarkable, however, was how the strongest players in the data consistently were doing something better with their time even at the fastest time controls. Everything moves at breakneck speed during a Bullet game, but even when there’s 60 seconds total to work with, stronger players are spending those seconds on the positions that benefit from it (at least a little).

Benefits vs. Costs to thinking?

I’ll close with an additional analysis that the authors carried out that I think is especially cool. Remember how I was kind of intrigued by the flattened curves on display in that initial look at the relationship between move time and the expected benefit of thinking? The authors were also intrigued by these and offer a nice perspective on where that shape may come from that’s based on an assessment of the costs to spending time thinking rather than just looking at the benefits.

The details of what they’re calculating are rather technical, but I want to try and give you the broad strokes. The starting point for their analysis is the observation that spending more time thinking obviously means you have less time to play, which increases the risk that you lose the game by running out of time. But how big is that risk? This is something the authors were able to estimate from the data! To do this, they sorted moves in their dataset according to the amount of time left and the position evaluation and then asked how often the active player actually won in those situations. What’s neat about this is that they’re getting all of this from the actual games, not from making many assumptions about the functions that might govern how games should turn out. The result is an empirical measurement of risk as a function of how much time you have left and how big your current advantage is: You can check out the left-most panel of their figure below to see what that looks like.

Figure 8 from Russek et al. (2025) At left (a) is the 2D plot of winning probability (brighter squares indicate more likelihood of winning) as a function of the time you have left and the strength of your position advantage, determined from the games in the dataset. This plot makes it possible to estimate optimal strategies for allocating time based on the actual risk of losing (b-d).

From this counting exercise, the authors can start talking about optimal strategies for allocating time in light of how much risk and how much benefit you’re facing in different circumstances. The big insight I want to leave you with is in the two right-most panels of the figure just above: The 2D shape that the authors started with in the left panel leads to the relationship between how long it’s worth spending on calculating in (c). If you’re interested in the technical details regarding how you get between panel (a) and panel (c), I would very much encourage you to check out the paper for yourself for the full run-down. For my purposes here, the intuition I want to give you is that this way of estimating the risk of taking up time thinking implies data that’s shaped a little like what the authors actually saw from real people. While these curves don’t quite flatten out as much as the data we saw first, this is still a really neat example of using specific computational ideas about how people make decisions under uncertainty to explain why chess players may spend time the way they do over-the-board.

What comes next?

This is one of those studies that has the potential to sound obvious to people: Oh, players spend more time thinking when there’s something worth thinking about? Big surprise. This is one of those cases in which, if you’re of that opinion, I’d encourage you examine that a little more: Why does it seem obvious? What are you assuming about what players can do during play? What do you do during a game to both manage your time and calculate as best you can? I suspect the more you consider it, the more you may realize that there is a lot you don’t really know about what makes you stop thinking and start moving.

What I like about the question the authors are asking (and answering) here is that it makes you think closely about the different kinds of decisions chess players have to make during a game. It’s easy to focus on identifying the best move as the primary goal for any player, but part of what we’re doing as real human beings playing this wonderful, complex game is try to understand other features of what’s happening on the board. For example, Julian from Chess Engine Lab has a very fun post about the idea of sharpness in chess positions which I think about a lot: How well do we sense that a position is very volatile, with chances and missteps available for either player? When do we sense traps set for us or get the feeling that our opponent has unwittingly stepped into one of ours? When does a move or a whole game seem beautiful?

This rigorous look at one aspect of chess metacognition is a great example of how formalizing different aspects of what’s on the board with sophisticated models helps us understand more about how (and in this case, how fast) we make good and bad decisions during play. The vast amount of data we have about players of all strengths is truly remarkable, too, and I hope we see a lot more work that uses all those moves to tell us more about how the game works in the human mind.

Support Science of Chess posts!

Thanks as always for reading! If you’re enjoying these Science of Chess posts and would like to send a small donation my way ($1-$5), you can visit my Ko-fi page here: https://ko-fi.com/bjbalas - Never expected, but always appreciated!

References

Russek, E.M., Acosta-Kane, D., van Opheusden, B., Mattar, M.G. and Griffiths, T.L. (2025), Time Spent Thinking in Online Chess Reflects the Value of Computation. Cognitive Science, 49: e70119. https://doi.org/10.1111/cogs.70119

McIlroy-Young, R., Sen, S., Kleinberg, J., & Anderson, A. (2020). Aligning superhuman AI with human behav-

ior: Chess as a model system. In Proceedings of the ACM SIGKDD International Conference on Knowledge

Discovery and Data Mining (pp. 1677–1687).

Osborne, H. (1964) Notes on the aesthetics of chess and the concept of intellectual beauty. British Journal of Aesthetics, 4, 160-164.

Zaidi, K. & Guerzhoy, M. (2024) Predicting User Perception of Move Brilliance in Chess. Arxiv, https://arxiv.org/abs/2406.11895

One thing about star-gazing that fascinated me as a kid is that looking at celestial objects necessarily involves seeing them how they were rather than how they are. Light takes time to get from a planet or a star to your eye, and the vast distances involved mean that this transit time becomes fairly meaningful: Jupiter, one of my favorite telescope targets, is about 45 light-minutes away, That doesn’t sound so bad, but once we’re looking at things outside our Solar System it gets more interesting quickly: The Orion Nebula is another one of my favorite destinations and it’s light takes about 1500 years to get to us! Some profound cosmic event may have changed what’s happening in that part of the sky a millennium ago and we’d still have to wait another 500 years to see the consequences.

If your brain lags behind, use your vision to think ahead!

All of that stuff is happening on a galactic scale, but a fun fact about your visual system is that it has to cope with a similar problem. While light can get from objects and surfaces out in the world to your eye in a negligible amount of time, there is still a meaningful gap between what you’re seeing and what is actually happening right now. The reason for this is that while light from the immediate environment gets to you instantly, your neurons take some time to send signals about that light from your retina to the parts of your brain that support your conscious awareness. You can see a schematic view of that pathway below, which I hope makes it easier to see that your vision is really a bit of a relay race, with cells at each stage passing the baton (neural responses) on to the next cell in line.

By Ratznium at en.wikipedia, CC BY 2.5, https://commons.wikimedia.org/w/index.php?curid=3199004

The passing of that baton turns out to take something like 50-100ms, which doesn’t sound so bad compared to our astronomical time-delays. Even though the time scale is a lot shorter, it still has important consequences for us, however. A lot can happen in 1/10th of a second, after all, and I’m sure you can imagine any number of fast-moving objects like cars, projectiles, or even other people that could cover a dangerous amount of ground in that kind of time. If you were consistently off by a 10th of a second, surely there would be many more car accidents, lots more bumping into people as you were walking around, and Pickleball would be more or less impossible.

Adobe Stock Image. Good luck with your Ladder League if your vision lags!

Fortunately for us, your visual system copes with this bit of lag handily. To keep your experience of the visual world as up-to-date as possible, what reaches awareness isn’t just what you actually saw - instead, you get a prediction of what is happening now based on that data. That is, your visual system uses the incoming sense data to project ahead a little bit to cover that 50-100ms gap, so you aren’t constantly just a little bit behind.

The flash-lag effect - seeing the future and the past simultaneously!

I recognize that this may sound pretty wild and perhaps a little unbelievable. Besides, how on Earth could we know such a thing? Look, personally, I think the fact that professional tennis happens is in itself a reasonable piece of evidence in favor of this idea, but you may be looking for something more convincing. To that end, I give you the Flash-Lag Effect. This is a wonderful visual illusion that lays bare the predictive nature of your perception by taking advantage of a visual event that you can’t predict easily, leading to a systematic error in what you see.

Many versions of the Flash-Lag Effect involve a skinny rectangle that is rotating clockwise at a fairly rapid clip. So far, this is an easy thing for your visual system to make predictions about, but we’re going to add something to the display that complicates things for you. Specifically, each time this spinning rectangle reaches a particular spot (let’s say the diagonal position in which it’s pointing from the lower-left to the upper-right) we’re going to briefly flash some new rectangles that extend beyond our spinning rectangle on both ends (see below). For a moment, this means that our spinning rectangle is lined up with some collinear partners, here’s the interesting part: Because the spinning rectangle is easy to make predictions about and the flashed rectangles aren’t, your visual system provides you with a prediction about the spinning rectangle but can’t follow suit for the small ones! The result is that even though all three rectangles were physically aligned on-screen, what you see is a “broken” arrangement of the rectangles (see the right image below).

Image Credit: Hogendoorn (2020) - A schematic depiction of the physical set-up in a Flash-Lag experiment (left) and what observers typically see (right).

You can try out a demonstration of the Flash-Lag effect at Michael Bach’s wonderful Visual Illusions website if you’re not convinced. There are also many other related demonstrations of this effect that use a translating object instead, or sometimes an object that is orbiting a central point - in each case, this disjointed experience arises consistently from the discrepancy between what you can predict and what you can’t.

Making visual predictions in chess (Ferrari et al., 2006)

What does all this have to do with chess? The authors of the study I want to describe for you here motivate their work by pointing out that strong play doesn’t just involve understanding positions in terms of the static position of the pieces at a particular moment in time, but also understanding what sequences of moves might follow from a particular position. The ease with which strong players do this was something that I found remarkable when I first started watching chess streamers a few years ago. Hearing Hikaru or Magnus casually rattle off potential sequences of moves that they thought would be good lines from their current position felt like a clear demonstration that they had an incredible ability to see not just what was in front of them, but what could be in front of them in a few moves (and sometimes many more than just a few!)

I hope this sounds at least a little familiar: What our group of researchers were arguing for in their study is that this understanding of the dynamics of a position, or more specifically, the ability to make predictions about the next steps in a game, may be an obligatory visual process that unfolds for stronger players as they look at a position. To be clear, the idea is that this is different than explicitly calculating potential candidate moves and their consequences! Certainly strong players are good at calculating and may use visual imagery to guide them, but the big idea here is there is also an involuntary process that just happens on its own. Just like you can’t help but predict the position of the spinning rectangle in the Flash-Lag effect, the proposal here is that strong chess players perhaps can’t help but predict likely next moves from a position that they are considering. To put it another way, it’s like their perception of the board has momentum. The question is, can we see that there are consequences of this act of prediction? Does seeing into the future a little bit make it so that players who predict see something analogous to the misaligned objects in the Flash-Lag display?

Experiment 1: Looking for evidence of predictions over short time-scales

One of the things that I was excited to see in this study was an experimental design that looked very familiar to me. During my PhD, I did a number of studies about how we perceive static images of objects that can move and that work included some examinations of how your visual system might make predictions about familiar objects like human bodies. Because we know how bodies tend to look when they’re walking or running, does that mean our visual system is particularly prone to projecting forward a little in time when you see a static image of a person?

One way I decided to look for evidence of those predictions was to use a simple same-different visual task with images of people walking. For my participants, the task was very simple: First they would see one picture of someone mid-stride for 500ms, then they would see a second picture of that person after a short delay. Their job was to decide if these pictures were exactly the same or if there was any difference at all between the two pictures. The sneaky part of this study was that when the two images were physically different, half of the time this meant that the second image was a little further along in the striding motion (the Forward order condition) and half of the time the same two images were presented in the opposite order (the Backward order). My logic was that if your visual system started with the first image and predicted what it might turn into, that prediction might look a lot like the second Forward order image, leading you to mistakenly say “same” even though they were different. The same process applied to the Backward pairs would make them easier to tell apart because now that forward prediction would make the first picture look less like the second. This was part of what we found, suggesting that these predictions may be happening automatically when you looked at images of people in motion.

Adapted from Balas & Sinha, 2009.

The authors of the current study adopted very similar logic and applied it to chess boards instead of pictures of people. The idea here was to present weaker players (~1500 ELO or so) and stronger players (~1900 ELO) with positions like the ones you see below and ask them to complete a same-different task with the boards. Just like my experiment with walkers, participants were asked to indicate if the two positions that they saw in sequence were identical or had any differences between them. These authors used the same sneaky design trick I included in my study, too: When the two positions are different, half of the time the difference between them is that on the second board a typical next move following from the first position has been made. The other half of the time, participants would see those same two positions in the opposite order, as though the game was being “wound back.” These two versions of what “different” positions are were referred to here as the normal order and reversed order conditions in their design. You can also see in the figure below that the authors included standard next moves and non-standard next moves, which gives them a number of interesting hypotheses to examine:

1. Stronger players may show different performance in the same-different task for normal and reversed positions due to strong predictive perception.

2. Standard positions may yield different performance than non-standard positions because predictions will be more consistent with standard next moves.

3. These two effects may interact: Strong players may have a bigger difference for normal vs. reversed boards, but only in standard positions, for example.

Adapted from Ferrari et al., (2006) Figure 1. The same position pairs can be presented in normal or reversed order to test the strength of predictions during visual inspection of chessboards by players of varying strength.

So what did they find? In their first experiment, participants got to see the first position in a single same-different pair for 5 seconds, and then had to decide if the second position that replaced it was identical or not as quickly as they could. This gave the experimenters the chance to measure players’ accuracy and how quickly they made correct responses to the critical “different” pairs. You can see in the table below that the presentation order of the two boards does turn out to matter! Stronger players (but not weaker ones) were more accurate at detecting differences between boards when the two positions were presented in the normal order than when the same two positions were presented in the reverse order. This was the case for both the standard and non-standard positions, suggesting that stronger players are making obligatory predictions about these positions and that this helps them notice when the next board has a change consistent with that prediction.

Adapted from Ferrari et al. (2006) Table 2 - Expert players are better at detecting differences between positions when they are presented in an order consistent with the sequence of moves in classic openings.

Things look mostly the same when we consider how quickly players’ were able to make correct decisions about “different” pairs, but with one crucial and interesting difference. If you take a look at the bar chart below, you’ll see the average response time across conditions for both participant groups and all of the experimental conditions. In this plot, a higher value means worse performance insofar as it means it took you more time to get to the right answer. Here, the Expert players are slower when making correct “different” responses to the reverse order boards, and unlike the accuracy data this effect appears to be weaker in the non-standard positions! To me, this is really kind of cool as it suggests a subtler aspect of predictive visual processing: One’s familiarity with specific opening sequences may make those predictions particularly strong or perhaps more impactful on your visual processing than other possible futures.

Adapted from Ferrari et al. (2006) Figure 2.

Experiment 2: Looking for evidence of predictions over longer time-scales

This is already pretty interesting stuff, but the authors have a second experiment that adds some further interesting insights into how visual predictions manifest during expert and beginner consideration of chess positions. In this second task, almost everything is the same except for one crucial difference: Instead of seeing pairs of positions one right after the other and completing a same-different judgment, now the authors asked their participants to complete a memory task. This means that now they would see a collection of positions in sequence for 5 seconds each, and then take a short break before taking a memory test. In that test, they would see some positions they had been shown before (”old” items) and some that they hadn’t been shown (”new” items) - the question is, can they tell which is which?

Like Experiment 1, the researchers built in the same forward and reversed pairs, but use them somewhat differently. The idea here is a little more like my body perception study: If you can’t help but predict the future a little after seeing a chess position, maybe what you store in your memory is not what you actually saw, but a guess about the future. This should mean that players who predict automatically are more likely to produce “false alarms” by saying that new test items that are one forward move along are actually old! The neat thing about this experiment is that unlike the first one, it puts stronger players in a position where the thing that serves them well over the chessboard makes them worse at performing this particular task.

Adapted from Ferrari et al. (2006) Figure 3.

This is indeed what the authors reported, and you can see the crucial data in the figure above. The most important parts of this graph are the “False alarm” sections because these indicate how often players at each level made mistakes by saying that a “new” item was actually something they had seen before. While Beginners make these mistakes at the same rate for both kinds of new items, Expert players fall for the positions that are a little further along in the sequence of moves.

Conclusions

The takeaway is that strong players do seem to make visual predictions automatically from chess positions, and depending on what you’re asking them to do this can be either a good thing or a bad thing! Chess expertise thus isn’t just about accumulating a sort of library of static board positions, but also about understanding sequences of play and how positions may transform from one move to the next. Over the board, as in the real world, it turns out that Wonderland’s Red Queen may have been right: you’ve got to keep moving just to stay in one place!

Support Science of Chess posts!

Thanks as always for reading! If you’re enjoying these Science of Chess posts and would like to send a small donation my way ($1-$5), you can visit my Ko-fi page here: https://ko-fi.com/bjbalas - Never expected, but always appreciated!

References

Balas, B., & Sinha, P. (2009). Learned prediction affects body perception. Visual Cognition, 17(5), 679–699. https://doi.org/10.1080/13506280802327866

Ferrari, V., Didierjean, A., & Marmèche, E. (2006). Dynamic perception in chess. Quarterly journal of experimental psychology (2006), 59(2), 397–410. https://doi.org/10.1080/17470210500151428

Hogendoorn H. (2020). Motion Extrapolation in Visual Processing: Lessons from 25 Years of Flash-Lag Debate. The Journal of neuroscience : the official journal of the Society for Neuroscience, 40(30), 5698–5705. https://doi.org/10.1523/JNEUROSCI.0275-20.2020

When I was a kid, we had two computers in the house (which seemed like a lot at the time) and my Dad’s PC was the one I could use to play some games on occasion. We had a bunch of different titles I saved up for from stores like Babbage’s, including stuff like Prince of Persia and an Indiana Jones game I never really got the hang of, but a fair bit of the time I also just wanted to sit down with something that would be good for killing some time and required just a little bit of thought. This means that I spent a decent chunk of my late elementary to middle-school years staring down Minesweeper grids and hoping I could move playing cards around with enough cleverness to achieve the intensely satisfying Solitaire victory screen.

I learned how to play Klondike (the game I think was the default variation on Windows in the 90’s) from my Grandma O’Leary, who taught me to use the 3-draw variant that gives you three cards at a time from the deck if you need some new material. There were more options available in the Windows version, though, so I picked up Pyramid and Freecell pretty quickly too. All these games felt a little like playing Candyland to me: The outcome seemed mostly pre-determined and you were just sort of moving the cards to see what it was. I didn’t really think of these games in terms of strategy, but more like a sort of ritual that was game-adjacent. I think this perspective is mostly borne out by the fact that there’s a decent amount of applied mathematics work to try and establish the solvability of Klondike and related games. I didn’t dig especially deep into this literature, but it seems like no one has quite pinned down how many Klondike games are winnable in theory, but it’s very close to about 80% (assuming the player is aware of what all the cards are - to so-called “thoughtful” variant for modeling purposes). By comparison, Freecell (which I knew I could win nearly all the time) is estimated to be winnable at a rate very close to 100%. I guess this means one could go all-in on optimizing your Solitaire play, but mostly I thought of these games as a way to chill out rather than test one’s tactical mind.

Breaking out of Windows Solitaire - Scoundrel, Ricochet, and Clear the Dungeon

Recently, however, I came across this great YouTube channel called Riffle, Shuffle, & Roll that includes a bunch of playthrough videos for solitaire games that have a lot of more going on. It’s definitely worth browsing the channel yourself if you like minimalist and abstract games (pawns, dice, cards, etc.) because there are a lot of great games to learn about. For my part, I got drawn in by the promise of a Solitaire game called Scoundrel that promised the player a “dungeon crawl.”

A dungeon crawl? In a solitaire game? Besides chess, cards, and tabletop strategy games, I’m also a life-long fan of turn-based RPGs (Wizardry 4ever!) and D&D, so this immediately caught my eye. If you want to learn how Scoundrel works, check out RSR’s video below - I have a feeling you’ll want to give it a try yourself (though be warned that it’s hard to win!).

From here, I checked out a few other solitaire games that were also very different from the variants I knew about. I think my absolute favorite is Clear the Dungeon, which plays very quickly and frequently turns into a nail-biter that depends on the last few cards you get to play. This one is great fun to play with my Cosmic Detective playing cards, which feature excellent character art from the comic in the face cards - those cards are your adversaries in Clear the Dungeon, so it’s nice to have some dangerous-looking baddies to fight against in a custom deck.

I was also impressed with Ricochet, a form of solitaire Breakout in which the cards serve as both projectiles and barriers in an attempt to escape from a three-layered cavern. This one has a big spatial footprint that makes it less convenient for playing on the go, but I love it’s potential to produce long chain reactions of card interactions.

After watching a bunch of these and trying them out, it really struck me how these games were able to take advantage of different aspects of the structures imposed by a standard deck of cards. On one hand, a 52-card deck (with Jokers for some games) doesn’t seem like it offers a lot, but the contents of that deck can be organized and combined in a lot of different ways that each offer a set of constraints and key relationships between cards that can lead to a lot of richness. I sort of knew that you can think about a deck of cards as a multi-dimensional data object and games as sort of interesting ways of slicing that data up and recombining it, but watching these in quick succession got me thinking more concretely about how this works to make for a playable game.

Here’s what I mean by this. Let’s start by thinking about what’s going on in Clear the Dungeon: The first grouping principle you have is the separation between the face cards (or a “power deck”) and the numerical cards. Assigning typical values to the Jack, Queen and King of 11, 12, and 13 respectively leaves us with a sum of 39 in the power deck and a sum of 55 in the numbers per suit. That respectable difference suggests a game in which using sums of numerical cards to overpower face cards (the basic mechanic in CtD) will give us some room for bad draws and failure, but not so much room that the game will be trivial. In CtD, the rules impose pairwise sums as a basic mechanic however, which imposes a further constraint on the numbers we have to consider - now we know we can make sums as large as 20 and as small as 3, but anything smaller than 11 will be useless for removing a Jack! The final key grouping principle the game uses are the 4 suits - without doing a deep dive into the rules (seriously, go watch the video!), CtD requires you to not just make numerical sums that are big enough to outmatch a face card, but also to include a card that matches the face card’s suit. All these constraints together mean that a good player has to track the numbers and the suits that are likely available in the deck to make the best plays in the current moment. The end result feels much less like a time-wasting ritual and way more like an actual game in which you’re making choices while dealing with uncertainty, sometimes taking some big risks, and often white-knuckling the last few draws from the deck.

The other games I mentioned rely on their own set of grouping principles, some borrowed from the deck and some imposed by the rules, each leading to a similarly satistfying challenge level. Scoundrel makes you a serious underdog by building in a sizeable imbalance between the values of the cards you can use to help you and the cards that will try their best to hurt you. Ricochet takes a somewhat hands-off approach in terms of deck-based constraints (suits don’t even matter!) but relies instead on the grouping imposed by the spatial arrangement of the cards for most of it’s playability. Other games that you’ll find on the channel have still other kinds of structure that serve as the basis for a good game, some of kinds are spatial, some kinds are arithmetical, and some are a neat blend of these.

As you can probably tell, I came away from these games very impressed and promptly annoyed a lot of my friends and colleagues by asking if they wanted to try them out. More than anything, though, I found the idea of inventing a new Solitaire game fascinating - who would’ve thought? Lots of people, as it turns out - I just wasn’t in the know. But now that I did know, I very much wanted to give it a try for myself.

How do you design a good solitaire game?

The prospect of designing a new solitaire game that would actually be fun to play reminded me very much of solving and writing puzzles for the MIT Mystery Hunt, which my family and I just participated in a few weeks ago. If you haven’t heard of the Hunt before, it’s a weekend-long puzzle competition in which players work to solve dozens of puzzles to find out the location of a coin that is hidden somewhere on MIT’s campus. The winning team gets to write the puzzles for next year. The puzzles are completely bespoke and generally come with no instructions: All you know is that the answer will be a word or a phrase, but your job as a solver is to look at what you’ve been given and try to figure out something to do with it that might lead you to an answer. If you’re so inclined, you can check out the puzzles from many previous Mystery Hunts to get a feel for what they’re like.

MIT Mystery Hunt solvers doing their thing to try and find the coin. Note the playing cards in the foreground! The structure in cards is great for a lot of applications. The original uploader was Madcoverboy at English Wikipedia. - Transferred from en.wikipedia to Commons., CC BY-SA 2.0, https://commons.wikimedia.org/w/index.php?curid=9433492

I’m fortunate to be part of a team that has won the Hunt twice, which means I’ve seen a lot of puzzle solving in my years and a lot of puzzle writing. Writing a good puzzle is really difficult! The thing is that anyone can come up with something completely obscure and impenetrable and anyone can come up with something that is trivially easy. It takes a thoughtful puzzle writer (and often a lot of test-solvers) to find the space in between these poles where a puzzle is hard, but gives the solver enough structure to find a path from a completely uncertain beginning to a definite answer. What you’re often doing as a puzzle writer is starting with some kind of structure or grouping principle you like and that you want your solver to ultimately find: Maybe it’s connections between celebrities, or the hierarchical structure imposed by Pokemon evolutions. Whatever it is, puzzle writers often start with a structure like that and build out the actual puzzle from there. Naively, I thought that designing a solitaire game would be a lot like this: What kinds of structure did I want to use in my game? What was in the deck that I wanted to take advantage of and what could I build in using my own rules? I decided I’d try to make some choices along these lines and see where they’d lead - the result is a game I call The 11th Hour.

How to play The 11th Hour

I’ll tell you how to play The 11th Hour, and I’m also going to tell you about some of the issues I ran into trying to make it work. In the end, it felt almost like I was solving a puzzle as much as writing one. In particular, there were certain choices regarding the rules that felt “messy” to me in a way that the wrong approach to a Mystery Hunt puzzle can feel. The best puzzles have an elegance to the solution that can be surprising, satisfying, and even funny, and I found that I wanted that to be true for my game as well. My hope is that the final set of rules meets that goal, ideally by making the game easier and better to play.

I should add a quick caveat: It is altogether possible that this is very close to or even a re-invention of something that already exists! Like I said, I just came across all these neat solitaire variants and may very well have missed that a game like this was already out there. If so, mea culpa, but I still had fun thinking it through on my own.

The set-up

To set up your game, the first thing you will need to do is remove both Jokers from the game (52-card deck only!) and sort out all of the Diamonds. Set the King of Diamonds aside and then arrange the remaining 12 cards in a circle with the Queen of Diamonds in the 12 o’clock position. These cards form the clock. Next, shuffle the rest of the deck and place one face-down card underneath each of the clock cards. Put the rest of the deck off to the side to serve as your draw pile and put the King of Diamonds in the middle of the clock, pointing at the Queen of Diamonds. The card under the Queen of Diamonds should now be removed and placed face-up inside the clock. This face-up card is the beginning of your reserve pile. I’ve got a photo of the completed set-up for you below.

Photo by the author.

The goal

To win the game, you must remove all of the face-down cards that are under the clock cards. If you remove the 11th hour before you run out of cards, you win! Otherwise, you lose (though see the variant ways to play at the end of the post).

The rules

On each turn, draw two cards from the draw pile. You may use these two cards and the face-up reserve pile card to turn the King to a new position. Black cards (Spades and Clubs) turn the King in the clockwise direction by the number of steps indicated by the card’s value. Jacks are valued at 11, Kings are valued at 13, and the Ace may be either a 1 or a 14 at your discretion. Red cards turn the King in the counter-clockwise direction using the same rules. Queens of either color can be used to point the King at the Queen of Diamonds regardless of where he started. After you have turned the King according to the cards you chose to use, discard both cards you took from the draw pile and discard the reserve pile card if you chose to use it to turn the King. If the King is pointing at a location that has a face-down card, remove it, and you may choose to put it face-up on top of the reserve pile. When you run out of cards in the draw pile, you may take all of the cards from the reserve pile and flip them over (no re-ordering!) to make a new draw pile. When you run out of any new cards to draw, the game is over and you lose if any hidden cards remain.

The process (which might not be over)

The first thing I decided about The 11th Hour had to do with a feature of Ricochet that I thought was neat: The rules allow you to consider the Ace both as a card that is lower than a 2 and higher than a King! Specifically, the game is based on “bouncing” cards off of one another if they are neighbors and the Ace is a legitimate neighbor of both 2’s and Kings. This allows you to “wrap around” the card order in a single turn, imposing a toroidal structure on the cards rather than just a line. I liked this and it made me wonder if there was another way to use the same circular arrangement on the cards in a different way. The trouble I was having at first, though, was working out a good excuse to work with 13 cards in a ring - should I do something with 4 rings of cards, one for each suit? Before too long, I got to the idea that 13 was a much less elegant number than 12, both because 13 is prime and 12 has lots of divisors and because we’re all familiar with a circular item with 12 things in it: A clock! It turns out that I am far from the first person to think of this particular relationship: There is a well-known game called Clock Solitaire that uses the same arrangement of cards I came up with for my game, but with very different rules.

Image Credit: By Nanami Kamimura - Own work, Public Domain, https://commons.wikimedia.org/w/index.php?curid=753491

With this spatial arrangement (that I thought I’d invented) in place, the next step was to think about what I was going to do with it. I liked the goal of clearing cards in Clear the Dungeon, so I decided to go with hidden cards under the clock positions with the goal being to get rid of them. But how? Obviously people would draw cards, but then what? A few things stood out at this point: The first is that I had a King left over from my clock arrangement - what to do with him? The second was that I liked the idea of treating Red and Black cards differently and not worrying about suit. This gave me the idea of using draw cards for turning the King, with color determining direction.

This may sound very close to the final game, but there were a lot of things left to work out:

1. How much stuff do you draw on a turn? - I’ve taken 25 cards out of the draw pile right away, so there are only 27 cards in there. I liked the idea of giving players a choice on each turn, so I decided that either 2 or 3 cards being drawn felt right, but 9 draws was obviously too few. Welp, 2 cards it is then, so you get 13 2-card draws per game with a last-ditch bonus. This felt very hard, but maybe that’s the point?

2. What do you do with the stuff you draw? - Again, I liked the idea of choice, so my first thought was to force the player to just use one card of the two they drew. Playing this myself a few times, though, it just felt deeply annoying. Too often the first few turns felt inconsequential and random (or I was forced into a choice by a missing hidden card) but then late turns would give me nothing to do. To be honest, I felt very stuck here - maybe this idea doesn’t work?

3. What about Queens? - Another wrinkle is that I originally assumed Queens would be worth 12 (in keeping with a typical face-card numbering), but in the context of this game that means they do absolutely nothing! This felt really bad and inelegant because of the 27 cards I was giving you to clear the board, 3 of them were actually useless. What are they doing there then? To be fair, Scoundrel asks you to just remove some cards from the game at set-up, but I didn’t like that for this game. But how to use them? A wild card seemed like the obvious answer, but that was definitely going to be too easy. I tabled this one, but knew I’d have to come back to it.

4. What (if anything) to do with the hidden cards you reveal? - One one hand, maybe you don’t need to do anything, but I was worried that the 13 draws and the 27 cards I was giving you to use might not be enough. Was there a way to incorporate these into the game to give you more chances? That said, just adding 12 more cards to the deck seemed like another path to making the game too easy.

Just like trying to write a good puzzle, the way to answer these questions was to play-test. Luckily, some unwitting grad students showed up in my office and asked if I had a deck of cards I could loan them (apparently this is the reputation I have in my home department!) and I asked if they’d try out the game. They were very serious about it and we ended up having a good conversation about some of the problems above. I had already come up with the idea of the reserve pile to make use of hidden cards, but I was trying out having it “loaded” with just one card at a time. The choose-one rule also was obviously not fun, so the idea of using all the cards to move suggested itself fairly soon after they started playing - this gave you multiple movements to consider on a given turn, which made the game feel much more active.

We also arrived at the Queens-as-reset rule that’s in my final description above. They agreed that a wild card was too generous, but liked the idea of a movement that was fundamentally different from the other cards. After getting stuck in their first game in my office, they reported that using this new rule they had a winning game and two losing ones, which I took as good progress. The Queen still sort of bugged me, though, as it meant that your first Queen was effectively a “get rid of 12 o’clock” card and the other two were then used strategically in order to get you to other positions. This felt inelegant to me - I wanted the Queens to all work the same! It took me a while, but the idea of removing the 12 o’clock card during set-up felt like it solved a bunch of problems at once: Now the Queens all did the same thing, the game was a little easier, and I really liked The 11th Hour as a title.

But is it any fun?

At best my new game is a modest achievement, but it was a nice excuse to think about the ways you can use a deck of cards and the groups, sums, and spatial arrangements you can get out of them to make a new puzzle that’s ready to be solved any time. At this point, I’ve played it a bunch of times and quite like it, but this may be because I’m a proud parent. What I’d really like to know is what you think of it, so if you have a deck of cards and feel like giving it a whirl, I’d love to know how it goes. Even if it turns out it’s not especially fun, I think I’ll probably keep thinking about the potential for new and different games lurking inside the patterns in those 52 cards. Who knows? Maybe I’ll find something else in there sometime soon - or maybe you will!

Image credit: Trainler, CC BY 3.0 <https://creativecommons.org/licenses/by/3.0>, via Wikimedia Commons

Bonus material: The 11th Hour variant rules

As promised, here are some other ways to try playing the game if you don’t love the rules I set out above. In no particular order:

1. Play until you win - Instead of just stopping when you run out draw/reserve cards, keep track of how many turns you have taken as you go and shuffle the deck as needed until you remove the 11th hour. Your final score is the number of turns you have taken and a lower score is better.

2. Alternate Queen rule - Instead of returning to 12 o’clock, use the Queen to draw a random card from the discard pile and include this in your movement cards as usual.

3. Alternate Reserve pile use - Instead of drawing two from the draw deck on every turn, draw two cards, one of which may come from the reserve pile.

If you try The 11th Hour out and have suggestions for rule changes, please leave these in the comments!

Thanks for reading!

If you’re so inclined, you can also send a small donation my way ($1-$5), by visiting my Ko-fi page here: https://ko-fi.com/bjbalas - Never expected, but always appreciated!

“Relax your eyes.”

“Cross your eyes a little, but don’t blur them.”

“Try to look through the image instead of at the image.”

Vision science doesn’t often get a bonafide cultural moment, but it seems to help when there is some sort of visual phenomenon that different people experience completely differently. This kind of disagreement about visual experience divided us into Team White & Gold and Team Black & Blue when the #thedress went viral, and Magic Eye images similarly divided the world into Those Who Could See It and Everybody Else. Unlike #thedress, however, this divide incorporated a healthy dose of FOMO - what was the cool thing people were seeing if they could get Magic Eye pictures to work? Staring at these in the hope that something would click and you’d become one of Those Who Could See It was a common enough experience for Seinfeld to write a Magic Eye gag into an episode.

The good news is that it turns out there’s a simple way to see the hidden 3D object in a Magic Eye picture without staring at it for hours. All you need is a digital version of the image you want to look and presentation software like PowerPoint or Keynote. To see how this works, check out my video below for a quick demo.

If you’ve never been able to see one of these before, congratulations! You just made it into the club.

To understand why my little PowerPoint hack works, we need to start with some basics of binocular depth perception. How do you see depth with your two eyes?

Binocular depth perception

An important feature of seeing the world binocularly is that your two eyes get slightly different views of the same scene. Since I started out by invoking the 90’s, I feel like I have to point out that Wayne’s World includes a random scene where Mike Myers provides a straightforward demonstration of this fact while gazing at Tia Carrere and alternating which eye he has open.

“Camera 1. Camera 2. Camera 1. Camera 2.”

The two images you have to work with, one from the left eye and the other from the right, will contain most of the same visual stuff, but they’ll differ with regard to where things are. Whatever you’re looking directly at will be in the same place in both eyes (right in the center of your visual field), but stuff that’s closer to you than that or further away will “jump” horizontally as you switch from one eye to the other. To be more specific, things that are closer to you than the object you’re looking at will end up off to the left in your left eye, and off to the right in your right eye. Things that are further away will do just the opposite: These will be off to the right in your left eye and off to the left in your right eye. How far “off” they are depends on how much closer or farther away they are.

The technical term for these differences in horizontal position between the left eye image and the right eye image of the same scene is binocular disparity, and because they’re systematic, your visual system can use them to make guesses about depth. The big idea is that since the direction (is the same object off to the left or right in each eye?) and the magnitude (how far off to the side is it in each eye?) of binocular disparity depends on how far away an object is, measuring the position of an object in each eye is enough to run the logic backwards:

1. If the same object is off to the left in the left eye and off to the right in the right eye, it’s closer to us than what we’re looking at.

2. If the opposite is true, it’s further away than what we’re looking at.

3. The more it’s shifted, the closer (or further) the object is.

The visual system’s ability to use these 3 rules to turn binocular disparity into depth also gives us a recipe for making 2D images that look 3D: If we give your left and right eye two images to see that have the right binocular disparities in them, your visual system should use those to give you a 3D experience.

Easier-to-see 3D pictures rely on binocular disparity.

It turns out to be pretty easy to do this. Stereoscopic images that relied on this principle have been around since the 19th century. Below, you can see an example of a Holmes stereoscope that contains a left eye image and a right eye image of the same scene that are delivered to the appropriate eyes by virtue of the lenses installed in the wooden frame nearest to you. The classic ViewMaster toy and modern VR viewers you can clip to your phone aren’t meaningfully different than these devices.

Image credit: Davepape, Public domain, via Wikimedia Commons

The only thing that might seem tricky about this is getting those two images in the first place. This is not especially difficult though - you can easily do it yourself with your own camera. Just pick a point to look at in the scene you want to photograph and take a picture of it with your left eye, then another with your right eye. As long as you keep your head fairly still, you’ll end up with two pictures with the right binocular disparity for stereoscopic viewing later.

Image by the author, from Chapter 7 of my textbook PRACTICAL VISION SCIENCE.

Old-school red/green or red/cyan 3D images are using exactly the same trick as well! The only difference is that the two pictures your eyes need to see are superimposed on top of one another in different colors that are matched to the filters in your viewing glasses. This ensures that each eye only gets to see one of the two pictures so your visual system can do its usual work with binocular disparity.

Image Credit: Kim Scarborough from Chicago, IL, CC BY-SA 2.0 <https://creativecommons.org/licenses/by-sa/2.0>, via Wikimedia Commons

So what’s going on with Magic Eye images?

Believe it or not, even though these other kinds of 3D images tend to be easy to see and Magic Eye images can be quite tough, it’s still the same trick. In a way, the Magic Eye (also known as an autostereogram) is a lot like a red/blue anaglyph but without the viewing glasses that help link the images you need to see to the correct eyeballs.

To understand how a Magic Eye works, I’ll start with the first thing you need to make one. Instead of taking two pictures of the same scene like I described above, we’re going to start with something called a depth map. Instead of a typical picture with different colors in different places and light and dark spots through the scene, a depth map is a picture that tells us just one thing about what we’re looking at: How far away is each part of this scene from me? Really that means that a depth map isn’t so much a picture as it is a table or spreadsheet. In the picture below, brighter parts of the image mean that that part of the object is quite close to you, while the darker parts are further away.

A depth map for the autostereogram available on the Wikipedia page about these images.

The neat thing about a depth map is this: Because we know that binocular disparity depends on depth (remember our 3 rules up above!), knowing how far away every pixel in a picture is also tells us where that part of the object should show up in the left and right eye! This turns out to mean that we can make a picture of the depth in a scene by “painting” with binocular disparity.

Here’s how it works. We’ll start with an image that’s the same size as our depth map with random pixels (or whatever pattern we like) already there. By itself, this obviously bears no resemblance to our object but that’s alright - we’re going to put disparities into this pattern so our object’s depth will be there. Every pixel in our depth map tells us how near or far that pixel needs to be from the viewer, which means it also tells us how far apart shifted copies of that pixel need to be! If that sounds confusing, think about the red/cyan image up above. You can find matching parts of the object in the red image and the blue image and you know they match because they look the same. Instead of putting a red image and a blue image on top of one another, what we do to make a Magic Eye image is (1) work out where the red image and the blue image would be and (2) make sure that matched parts of the object have matching pixels.

The end result is a pattern that looks random, but secretly has pairs of pixels shifted by different amounts according to where they’d be if we’d stacked a right-eye image and a left-eye image of the same object on top of one another. All we have to do now is get your left eye and your right eye to each see what they need to see and your visual system will do the rest just like it does with other kinds of 3D pictures.

OK…let’s cheat. So long depth from disparity, hello depth from motion!

Of course this is where things break down for lots of people. The thing that tends to happen is this: You hold up the picture to look at it and your eyes will tend to move a little inward to fixate the page you’re holding fairly close to yourself. The trouble is, to get those shifted pixel copies to the appropriate places in your left and right eyes, most Magic Eye images are designed so that your eyes need to less inward-turned than that. This is where all the advice about “relaxing” your eyes and “looking through the page” comes from - you’re trying to get your eyes to drift a little bit outward from where they are.

This takes practice and it does help a bit to know what you’re trying to do. It can also help to wrap a Magic Eye image in Saran Wrap or other cling film so that you can see your own reflection in it. Looking at your reflection rather than the pattern tends to put your eyes in the right spot! I learned this from a Magic Eye poster I was given as a gift in high school that was sealed up in plastic - it was the first one I ever got to work and it’s in my office (still sealed) to this day.

Image by the author. A bit of plastic film on a Magic Eye image helps you align your eyes the right way if you look at your reflection in the plastic.

If you don’t want to practice this, though, you can always rely on my trick - but why does it work? I hope it seems a little less mysterious now that you know what’s in Magic Eye images. By sliding one transparent copy of the image over the other, I’m trying to undo most of the pixel-shifting that was done to create the image. Next, by putting the two images on separate slides and alternating between them, I’m giving your visual system a chance to match those pixels across time instead of across space.

An animated GIF of the hidden shark embedded in an autostereogram on Wikipedia’s page about the phenomenon - this corresponds to the depth map above.

Seeing where the same thing appears at different moments instead of in different eyes is also a good way to estimate depth, it turns out. But you knew that already, so long as you’ve looked out the window of a moving car or train: If you’re looking way off into the distance, stuff that’s closer to you moves very fast, while further-off things don’t change position much at all. That relationship is a special case of depth from motion parallax and my PowerPoint hack lets you rely on this recipe for estimating depth instead of the somewhat delicate viewing conditions we need to hope for in the original Magic Eye.

Conclusion

If you were never one of Those Who Could See It, I hope this maybe gives you a little bit of closure and the chance to go see what’s hiding in these very cool images. Even if you already could get these to work for you, I hope it’s still neat to know a little more about how they work.

If you like learning more about why you see things the way you do, make sure to subscribe to my YouTube channel (http://www.youtube.com/@SeeingScience) for more science content focused on visual perception!

If you’re so inclined, you can also send a small donation my way ($1-$5), by visiting my Ko-fi page here: https://ko-fi.com/bjbalas - Never expected, but always appreciated!

On the other hand, you could also be interested in the science of chess because the game offers you a pathway to studying something else that you want to understand. That is, you might want to examine chess because it has some neat properties that make it just the right vehicle for a deep dive into your real interest. I think Alexander Kronrod’s famous quote “Chess is the drosophila of artificial intelligence” is probably one of the best expressions of this perspective. Drosophila melanogaster, also known as the fruit fly, is a perfectly interesting little organism, but it’s real value to science is as a model organism for genetics. Fruit flies don’t have a lot of chromosomes, have a short life cycle, and reliably have a lot of offspring. Those three properties mean that a genetics researcher can observe what happens over several generations of fruit flies pretty quickly and can also benefit from all the things we know about the fruit fly connectome, its genome, and its development to ask targeted scientific questions.

The humble fruit fly hanging out on a banana. By Sanjay Acharya - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=63862733

Like the fruit fly, chess also has a number of properties that make it great for asking questions about other things (including artificial intelligence). The game is rich, but has a clear set of rules that constrain how we play. There is also no hidden information in chess, so if you’re interested in judgment and decision-making you don’t need to cope with uncertainty and all the complications that adds to understanding cognition. From the standpoint of a scientist who is interested in how people play the game, it’s also especially handy that there are multiple rating systems that offer a quantitative estimate of playing strength. Whether you prefer ELO or Glicko-2, ratings save the behavioral scientist a ton of work by providing a straightforward answer to the question, “How good is this player?” Finally, perhaps the most useful property of chess for the cognitive scientist or the neuroscientist is that thousands of people around the world voluntarily play chess all the time. Together, these features of the game (among others!) make chess a wonderful model organism for learning new things about the mind and the brain.

Using chess to understand expertise in the brain

The study I want to tell you about is a great example of researchers using chess as a vehicle for asking and answering some intriguing questions about expertise. This one caught my eye in particular partly because one of the authors (Dr. Hans Op de Beeck) worked as a post-doc at MIT during my time there as a graduate student (Hans in Dr. Jim DiCarlo’s lab, me in Dr. Pawan Sinha’s). We’re both very interested in visual recognition and how experience and expertise affects how recognition works and I was stoked to see that we both apparently also think chess is worth thinking about scientifically.

This paper has a ton of cool stuff in it, but a lot of it is also very technical. I’m going to do my best to talk about some of their key results by introducing you to some analyses that are currently quite popular in modern neuroscience and cognitive science, but there are also a few things I’m going to leave out to keep the discussion here as accessible as possible to folks that don’t have a background in data science. My hope is that I can give you a sense of some of the things they learned about how chess expertise translates into differences in neural processing and what that may mean for our understanding of expertise more broadly.

The goal of the study is largely to quantify how chess expertise changes the way chess positions are compared to one another both in terms of the explicit judgments players make (Which position do I think is better for me?) and in terms of how different cortical networks respond to those positions. What I thought was really neat about this study was the way the researchers built bridges between the brain, observers’ understanding of chess positions, and those same chess positions considered on their own, independent from players’ assessments of them. They achieve this by applying a tool called Representational Similarity Analysis (or RSA) which is a powerful means of comparing how different systems respond to a specific set of items. I’ll give you a bit of primer on how this technique works while introducing you to their experimental design. We’ll get started by talking about the chess positions that serve as the starting point for their work.

How similar are chess positions in theory?

To start asking questions about how experts’ and non-experts’ minds and brains evaluate chess positions, the researchers developed a carefully curated set of chess positions to use in their experiment. The nature of these positions is already a pretty cool thing to talk about because they built in a lot of neat structure across the 40 different boards they used here.

Specifically, these 40 positions were chosen so that each individual position could be assigned a label from one of three categories:

(1) Checkmate vs. Non-Checkmate, which is exactly what it sounds like - each position either would lead to a forced mate, or did not.

(2) Strategy - These are a little trickier to describe, but positions were also selected according to tactical motifs that were critical to the position. These included strategies like “Bishop-forcing moves” and “Knight-Bishop combinations” am

ong others.

(3) Visual Similarity - Each checkmate position had a non-checkmate partner that differed only by a single piece. As such, the visual appearance of these pairs was closely matched while changing the tactical outlook substantially.

Adapted from Figure 2b of Constantino et al. (2025) - Examples of positions used in the experiment and their assignment to different categories.

This is neat both because it requires some serious care (and good chess knowledge!) to achieve and also because it means we can talk about the similarity between positions in this set of boards in different ways. Below, I’m showing you an example of an important tool the researchers use throughout their work to talk about the similarity between the items in this stimulus set: A Representational Dissimilarity Matrix (or RDM). This is a fancy term for a data object that isn’t all that hard to understand - an RDM is basically a table where we write down how similar each pair of items in our set is.

To be more specific, imagine you wrote the numbers 1-40 down the rows of a sheet of graph paper and then wrote them again across the columns. Each single grid square inside that 40x40 area is a box where we can record how similar one pair of positions is. The box in the 3rd row and the 8th column is where we’d write down the similarity between Position #3 and Position #8, for example. Once we fill up all the boxes in this grid, we’ve got a complete RDM for the chess positions we’re considering here. Because the three categories I described above each have a fixed set of labels (Checkmate vs. Non-Checkmate, for example) two positions can either be perfectly similar by having the same label, or perfectly dissimilar by having different ones.

Below you can see what the RDMs look like for these positions when we think about comparing them to one another in different ways. Depending on the category we’re thinking about, the resulting RDM has very different structure! The Visual Similarity RDM has that diagonal line all the way from the top-left to bottom-right because every position is similar to itself, but has that pair of parallel lines because each board also has a single visual similarity buddy defined by changing just one piece. Every other pair of positions in the grid gets a low similarity score, while these get a high value. Checkmate vs. Non-Checkmate looks quite different though, because every “Checkmate” pattern has the same label as every other “Checkmate” pattern, which leads to those big quadrant-sized blocks you see.

Adapted from Figure 2c of Constantino et al. (2025) - RDMs for all 40 chess positions according to different criteria for judging similarity.

The point of assembling these positions and building these matrices is to come up with a sort of language for talking about the relationships between all of the stimuli at once according to different rules. By itself, this is just kind of a neat exercise in thinking about some data (in this case, chess positions) in different ways, but the next step in this study is to use these grids as a starting point for examining expertise and the brain.

How similar are chess position in the player’s mind?

Alright, to take stock of what the researchers have done so far, their first step was developing a way to measure similarity relationships between chess positions using different rules for comparing two boards. With those different relationships defined, their next step was to collect judgments from chess experts and non-experts about those positions so we can measure how similar they are are to one another in the mind of a player. To do this, participants were asked to complete a “1-back” task while having their brains scanned via fMRI.

In general, a “1-back” task refers to an experiment in which we show participants items in a sequence (one after the other) and ask them to continuously compare the item they’re being presented with now to the one they saw just before (or one item “back” in the sequence). In a very simple 1-back task, you might ask participants to decide whether or not each item is identical to the one they saw a moment ago. In a more complicated task, you might show people lots of pictures of real-world objects and ask them to push a button every time the item they’re looking at now would be more expensive to buy than the item they just saw. Researchers often use “1-back” tasks when they want to ensure that participants are paying attention to and evaluating every item they are presented with.

Here, the researchers asked their participants to complete a chess-based version of a “1-back” task using the 40 positions described above. During each block of the full experiment, participants saw 80 positions in a random sequence and were asked to push a button if the current board was a better position (for White) than the previous one. As you might imagine, this is pretty tough! Each board is only on screen for 2.5 seconds with a 1-second gap in between and they just keep coming until the full round of 80 positions is up.

Adapted from Figure 2d of Constantino et al. (2025) - A schematic view of the 1-back task. Every time participants saw a position, their job was to decide if it was better than the previous one.

The point of doing this is two-fold. The first point is to establish a way of talking about the similarity between pairs of positions in this stimulus set based on what players say about them during a task that requires them to evaluate the position. The second point is to record the way different parts of the brain respond while participants carry out this process. We’ll start by talking about how that first part works and then move on to talking about the brain.

The big idea here is that across all of our participants in the task, we can use their decisions about whether neighboring positions in the sequence were better than one another to talk about the similarity between boards using a new cognition-based rule. Imagine that we looked at all the times Position #10 was compared to Position #22 in our experiment and found out that participants were dead-even as to which one was better. That would suggest that these two positions are pretty similar! On the other hand, imagine that every time this pair showed up participants preferred Position #10. Now these two positions seem very dissimilar (because one is definitely worse). If we do this for every pair of positions that participants compared in our experiment, we can write down all those similarity numbers in a table just the same way we did when we were only thinking about the stimulus categories the researchers had built! That is, we can build an RDM based on participants’ decisions about the boards.

Adapted from Figure 3a of Constantino et al. (2025) - RDMs for all 40 chess positions according to experts’ and novices’ pairwise preferences for positions when presented in the 1-back task.

In the figure up above, you can see what that RDM looks like for both the experts (top row) and the novices (bottom row). Happily, this is one of those cases where the key results are sufficiently clear that I don’t think I need to talk much about statistics. The first thing I hope you notice is that the experts look a LOT different than the novices. To be blunt, while the expert RDM has some obvious order to it, the novice RDM is just a big ol’ mess. The second thing I hope you notice is the first interesting connection the researchers are making between different ways of characterizing chess positions. That expert RDM doesn’t just have some kind of order to it, it has a very familiar order to it. Specifically, it looks very close to the RDM we saw for our Checkmate vs. Non-Checkmate similarity rule! That means that even though we didn’t specifically ask participants to solve for mate, their preferences about the boards in our set ended up having the same structure as our sorting of the positions by Checkmate status.

Before you tell me this is obvious, I want to point out that this could have turned out differently. Players could have consistent preferences for different kinds of tactical motifs, for example, and picked Knight-and-Bishop boards over Queen-and-Rook boards. If they had done that we might have seen some hints of the Strategy rule in this RDM in addition to the Checkmate Rule. Even if that doesn’t convince you, we’re not done yet. There is one more bridge we can build with this data, and this one brings the board, the mind, and the brain together at once.

Where do we find different kinds of position similarity in the brain?

To do this, the authors use the last part of the data collected during their 1-back task that we haven’t covered yet: The actual fMRI data recorded from participants’ brains during the experiment. While participants were making their decisions about which positions they preferred, the researchers were also collecting measurements of the BOLD (blood oxygenation-level dependent) response across the brain. That means that for each board we have a new set of measurements: How active were different parts of the brain while participants were evaluating each position? The fMRI data provides us with a long list of numbers in each brain area that we can use to compare the responses to pairs of items. How would we do that? These lists can be used to calculate a distance (think your good old-fashioned Pythagorean Theorem but with lots more terms) and THAT means we can make a new RDM for each brain area that includes all of the positions: A bigger distance between boards means less similarity in neural response, while a small distance means greater similarity.

Adapted from Haxby et al. (2014) - BOLD measurements in brain voxels yield a list of numbers that describe the neural response to each item we present to participants. Any two lists can be thought of as points in a high-dimensional space and we can thus calculate a distance between those points to estimate similarity.

Now we can ask if these brain-based RDMs share any of the structure we saw in our Strategy, Checkmate or Similarity RDMs. Instead of just using the eyeball test (like we did above for the behavioral data) to say that these look kind of the same, we can be more formal about it and actually measure a correlation between the values in the two RDMs we want to compare. A high value would mean lots of shared structure, while a low correlation value would mean very little.

Below, I’ve got some rather dense figures from the paper showing off what they found. In each figure, you’ll see the correlations between the brain RDMs and one of the original similarity rules in a collection of different brain areas for experts and non-experts. Let’s start with the Visual Similarity rule - what parts of the brain (if any) seem to process these positions just in terms of how they look?

Adapted from Figure 4 of Constantino et al. (2025) - Correlations between neural RDMs obtained from each brain area and the Visual Similarity RDM we started with.

Interestingly, none of these correlations differed between experts and novices. Both groups show us that areas in the occipital lobe associated with visual processing have the most agreement with this rule. That’s consistent with the idea that the Visual Similarity rule relfects fairly shallow sensory processing and thus doesn’t have a lot to do with chess expertise. Let’s take a look at the Strategy rule next.

Adapted from Figure 4 of Constantino et al. (2025) - Correlations between neural RDMs obtained from each brain area and the Strategy RDM we started with.

Now things are getting kind of interesting! First, it’s neat to see the shift in which parts of the brain share structure with our Strategy RDM. Instead of those visual areas in the back of the head, now there are parts of the parietal lobe associated with spatial processing, and also parts of the frontal cortex associated with motor planning and decision-making. Perhaps more importantly, those areas are also places where we see significant differences between experts and novices, to the tune of experts exhibiting that shared structure while novices kind of don’t. Another thing I want to add here is a reminder that we couldn’t see hints of the Strategy RDM in our behavioral data! I think that’s a good demonstration of why considering these stimuli in terms of their formal structure, behavioral decisions AND neural responses is so cool in this study. Even though the Strategy RDM wasn’t evident in the behavioral task, the researchers could measure that shared structure in specific parts of the brain.

Adapted from Figure 4 of Constantino et al. (2025) - Correlations between neural RDMs obtained from each brain area and the Checkmate RDM we started with.

Finally, let’s take a look at the Checkmate rule. This looks much the same as our Strategy analysis, incorporating areas for motion perception, motor imagery, and spatial processing that differ between experts and non-experts. All told, these results are a rich look at how expertise changes the expression of relationships between different chess positions across cortical sub-networks.

Expertise is sparse, but mighty

We’re really only scratching the surface of everything the authors discuss in this paper, but I’ve already given you a lot of technical stuff to think about. I do want to close though, by coming back to the idea that sometimes we study chess scientifically because we want to use it to learn about another topic. So far, a lot of what I’ve shown you is pretty chess-specific - which parts of the brain support expert-level judgments about strategy vs. checkmate tactics? The authors didn’t stop there, however, and move on to a number of neat analyses that further unpack what’s going on in the expert brain. A lot of these would take a LOT of work to unpack for you, so I’ve got just one tidbit I want to leave you with instead.

If we go back to the idea of measuring what different parts of the brain do when presented with one of these chess positions, you may remember that I said this gave us a list of numbers we could use to describe that board using neural responses. If we had lots of numbers in that list (let’s say one number from each of 1,000 brain voxels - the volume elements of fMRI), that would mean every board would be a point in a 1,000 dimensional space. What’s more, if we recorded these neural responses for lots of different boards we’d end up with lot of points in this 1000D space. What kind of shape would those points make? To be more specific, would those points make a 3-dimensional shape like a ball, a 2-dimensional shape like a flat disc, or maybe even a 1-dimensional shape like a curve or line? The number of dimensions in the shape those points make up is related to something called the participation ratio, which the authors use here to further compare what expert brains do compared to novices when evaluating these chess positions. Below, you can see how this value looks across both groups in all the brain areas they considered here.

Adapted from Figure 6 of Constantino et al. (2025) - Correlations between neural RDMs obtained from each brain area and the Visual Similarity RDM we started with.

The main takeaway is that the experts have a lower PR across the board for the most part. You can really see this in the bottom panel where the differences between experts and non-experts are being displayed across brain areas. The fact that almost all of these drop below the x-axis means experts have a lower value for this term than the novices with just a few exceptions.

What does this mean? That value tells us (we think) something about the efficiency of the neural responses to these items. In experts, the neural response is sort of streamlined, only involving a limited fraction of the full space available to describe boards using the neurons in each area. What’s more, this is true even in the parts of the brain that appeared to participate in capturing the similarity structure of the Strategy and Checkmate rules! That means that the expert brain is able to do more than the novices and can do it with less. This is something that speaks to a potentially general feature of expertise that chess just happens to help us see.

Conclusions

Like I said, this is a rich and complex study with a ton of intriguing stuff in it. I hope I’ve given you a sense of how this research team used some very cool data science techniques to ask some really neat questions about expertise. While this isn’t the kind of work that’s going to give you insights into how to get better or a new perspective on how you play the game, the science of chess in this instance is helping us understand the nature of learning and expertise in the mind and brain.

If you made it this far, thanks for sticking with me! I spent a little more time than usual working towards this post, mostly because I thought it would be fun to tell you all about a study that incorporated some frontier techniques and went beyond some of the usual expert/novice comparisons. I hope you enjoyed it and stay tuned for more Science of Chess soon, hopefully without as long a break between posts.

Support Science of Chess posts!

Thanks as always for reading! If you’re enjoying these Science of Chess posts and would like to send a small donation my way ($1-$5), you can visit my Ko-fi page here: https://ko-fi.com/bjbalas - Never expected, but always appreciated!

References

Andrea I. Costantino, Artem Platonov, Felipe Fontana Vieira, Emily Van Hove, Merim Bilalić, Hans Op de Beeck (2025) Low-Dimensional and Optimised Representations of High-Level Information in the Expert Brain, bioArXiv, doi: https://doi.org/10.1101/2025.11.12.688012

Recanatesi, S., Bradde, S., Balasubramanian, V., Steinmetz, N. A., & Shea-Brown, E. (2022). A scale-dependent measure of system dimensionality. Patterns (New York, N.Y.), 3(8), 100555. https://doi.org/10.1016/j.patter.2022.100555

Haxby, J. V., Connolly, A. C., & Guntupalli, J. S. (2014). Decoding neural representational spaces using multivariate pattern analysis. Annual review of neuroscience, 37, 435–456. https://doi.org/10.1146/annurev-neuro-062012-170325

McCarthy, J. (1990). Chess as the Drosophila of AI. In: Marsland, T.A., Schaeffer, J. (eds) Computers, Chess, and Cognition. Springer, New York, NY. https://doi.org/10.1007/978-1-4613-9080-0_14

Why faces? If you’re wondering why I spend so much of my professional life on this one topic, you’re not alone. On my first day as a post-doctoral researcher, one of my new lab-mates asked what I worked on, and my answer elicited a drawn-out sigh and a look of mild disappointment. “Another face person,” said the inimitable Dr. Jenny Richmond, “I still don’t get the big deal. Why is everyone so excited about faces all the time?” I defended myself as best I could (and I should note that we were and are great friends!), but it’s a fair question. Even after doing research on the topic for 20 years or so, I get why it seems surprising that someone would focus that specifically on what seems like one little corner of psychology.

The thing about faces that I think is so interesting is that they are probably the best example of a class of visual stimuli that most of us have extensive expertise with (save for individuals with face-blindness, or prosopagnosia) AND that we see differently than other objects in measurable ways. And here’s where the stuff I think about in the lab runs smack into the Science of Chess! The target article I’d like to tell you about offers some evidence that the way expert players visually process chess positions may have a strong resemblance to how all of us (or at least most of us!) process faces. To get us started examining what that commonality between chess and faces might be, I need to tell you a little bit about face recognition first, and explain why (and how) we think it works differently than other kinds of visual recognition.

Processing faces as a whole, (not by parts!)

So what is the big deal about faces and face recognition? There are a few reasons that faces stand out from other classes of visual stimuli. One that’s rooted in cognitive neuroscience is the existence of specialized cortical areas (like the fusiform face area, or FFA) that contribute to face recognition. If the brain has devoted some specific areas to processing these images, surely that means there is some way that they’re unique, right? For my part, even though I got my start as a researcher working in a lab that did tons of exciting work on these brain areas, I’m still much more interested in the behavioral features of face recognition that I think make an equally compelling case for faces to be considered an interesting special case of visual recognition.

You’re (probably) very good at recognizing faces

This first one may seem less than exciting, but I think it’s very important. Depending on your environment, you probably have a fairly large collection of faces that you are capable of naming, telling apart from one another, or at least recognizing as people that are familiar to you even if their name escapes you. By “large,” I mean something like 5,000 faces or so. Not only do you individuate this large collection of faces, you do so subject to some unique types of variability that other objects aren’t subject to: Faces do all kinds of non-rigid deformation, for example, when people talk, smile, or furrow their brows. Not only can you recognize people when they twist their facial features around this way, you can even use those movements to do things like read lips, recognize emotions, or make guesses about people’s mental states. If nothing else, that’s a great bag of tricks that’s hard to match with other types of objects.

Image credit: J. Campbell Cory, Public domain, via Wikimedia Commons

Not only are you capable of recognizing an awful lot of faces, you can also do a lot of good work with very little visual information. My graduate advisor, Dr. Pawan Sinha, has done many studies examining how good you are at recognizing faces when they are degraded by blur and noise, for example. It turns out that even when familiar faces are terribly blurry, you tend to do quite well anyway, demonstrating how robust your face recognition mechanisms are to fairly dramatic changes in appearance quality. To give you my favorite example of this, I have to show you what might be the most glorious case of degraded face recognition in the wild.

Back in 2018, a man in Lancaster, Pennsylvania (my home state, but not my hometown!) allegedly took advantage of an employee stepping away from their station by taking over their spot and swiping some cash when the opportunity presented itself. Obviously the man left in a hurry after doing so, but some eyewitnesses did apparently think that something looked a little suspicious and when questioned were able to help come up with a sketch of the suspect. Dear reader, here is that sketch:

I think there’s a decent chance you may have laughed a little at this sketch, and if so I don’t blame you. The thing is that despite what seem like the obvious shortcomings of this sketch, they caught him! Someone saw that drawing and it was just enough for them to realize that they might know the guy the police were looking for and they called it in. If this doesn’t convince you that your face recognition is pretty impressive, I’m not sure what would.

Recognizing upside-down faces is surprisingly hard.

So your face recognition is quite good and that might be one reason to suspect that it’s different somehow from the other kinds of visual recognition you do. This isn’t enough, however: Just because you’re particularly good at one version of a task doesn’t mean you do something qualitatively different to complete it. Your remarkable abilities at recognizing faces may just mean that the same tools you apply to recognizing other objects are just being used to their utmost in this case. Do we have evidence that you might actually use some truly different tools for faces than for other objects?

In fact, we do! One classic result in the face recognition literature concerns something a little curious that happens when you turn faces upside-down. Doing so keeps a lot of things intact in the original image, but obviously disrupts many others. If we measure how much this changes your ability to tell two objects (like chairs or shoes, for example) apart, many studies have found that you tend to do a little bit worse, but not terribly so. Turn faces upside-down, however, and something different happens: You get much worse. I could show you some profoundly exciting bar graphs about this, but instead I’ll refer you to a more dramatic demonstration of this phenomenon: The Thatcher Effect. First, take a look at the two images below, which each depict former British Prime Minister Margaret Thatcher. They are both upside-down, of course, and you can probably tell that there is some kind of difference between the two. My guess, however, is that even if you can sort of work out what we’ve done to one of these pictures that makes it different than the other, it isn’t having a huge impact on you.

Turn both images back to right-side-up, though, and the situation is probably much different. Now one of these faces probably looks absolutely grotesque while the other is obviously just fine. The classic account of what’s happening here is that the upright faces can be processed very effectively by your face recognition mechanisms, making the local inversion of the eyes and mouth immediately apparent and highly troubling. Turning these images upside-down demonstrates just how much less effective those mechanisms are at relaying the appearance of the inverted pictures - a huge different in the upright faces turns into a visible, but less affecting difference in the inverted ones.

The big idea here is that this is an important clue that the tools you use for most other objects may be different in some qualitative way: Those tools can do much the same work on an upright and an inverted picture, but the tools you use for faces cannot! I need to tell you before we move on that this specific effect remains something we’re learning about and there is more nuance to this than I’m making room for here, but for our purposes it leads us nicely to what we need to connect face processing to chess processing.

You recognize faces holistically (whatever that means!)

The inversion effect suggests that there might be something different about the way we process faces compared to other objects, but what is that difference? Another class of behavioral results from the face recognition literature has helped make this more clear, and this is where we need to go next to set the stage for our target article about chess. Together, these effects suggest that we recognize faces by processing them as holistic patterns, which doesn’t seem to be true for other objects. What does that mean, exactly? To be blunt, we’re still working that out, but the effects I’m about to describe give us some vital clues.

The part-whole effect

First, let me tell you about something called the part-whole effect, which was first reported by my friend and colleague Dr. Jim Tanaka and Dr. Martha Farah. In their study, they presented participants with a recognition task that required them to tell faces apart that only differed in one facial feature (let’s say the nose - see below). People tended to do quite well at learning to tell these faces apart, and you can test this by showing them the faces a little later and asking them to choose the one that matches a name they learned (”Which one of these is Larry?”).

A figure from Tanaka and Farah (1993) depicting two faces that differ only in the appearance of the nose. If you learn to tell these faces apart from the whole images in the bottom row, how do you do if I ask you to recognize whose nose is whose?

But here’s the interesting part: Those two faces only looked different because they had different noses, right? Given that, what happens if we don’t show our participants the whole face to test their recognition, but just show them the noses instead? (”Which one is Larry’s nose?”). The first interesting surprise in this study is that telling just the two noses apart after learning about them from whole faces turns out to be extremely difficult! Even though this was the crucial part of the picture for distinguishing the faces, people need to see the nose in the context of the whole face to reliably recognize who is who.

The second interesting surprise here is that if you play exactly the same game with another kind of object, it works very differently. Instead of faces that only differ by a nose, let’s use houses that only differ by a door. We can use the same kind of learning task and ask the same kinds of questions afterwards (”Which is Mike’s house?” or “Which is Mike’s door?”) and in this case, it’s not a problem to pick the right house OR the right door. What’s going on? One account of these different outcomes is that objects like houses may be broken down into lots of parts (like doors, windows, etc.) for recognition, but faces are not broken down into parts like the eyes, nose, and mouth. Instead, faces may be recognized by using the whole face pattern at once, making it difficult to make good recognition judgments about any small parts of the pattern.

Tanaka & Farah’s (1993) results demonstrating that while it’s tough to recognize face parts in isolation if you learned them from a whole face, you don’t have the same problem with houses and their parts.

This is one of those results that I still think about a lot, even though it’s now more than 30 years old. In particular, it still makes me wonder about this thing people do all the time when they judge family resemblance and appeal to specific parts of the face: “She’s got her mother’s eyes!” people will say, or “That’s her Great-Uncle Kevin’s nose!” Anecdotally, at least, people seem remarkably confident about these despite a lot of data that indicates we’re awful at judgments like this. If you want a fun way to experience just how tough these judgments are, I’d encourage you to try out the brilliant Eye-Q game developed by Didi Dunin, a PhD student in Psychology at the New School in NYC. In Didi’s game (pictured in the thumbnail for this post) you hold up a card with a celebrity’s eyes in front of your own, placing these familiar eyes in the context of your face. Because of whatever is happening that drives the part-whole effect, it is MUCH tougher to figure out whose eyes you’re looking at!

Didi showing off the Celebrity Eye-Q game she developed. It’s a little difficult to recognize celebrity eyes in isolation, but even more difficult if you put them in the context of someone else’s face!

The composite face effect

The Eye-Q game sets the stage nicely for talking about one more effect that we need to understand the face-chess connection being made in our target article. I say this because something you’ll experience if you play the game is that putting the celebrity eyes in the context of a new face makes the eyes look sort of...different. That is, you may see the shape or spacing of the eyes a little differently if they appear in the context of one person’s face vs. another and this obviously complicates things for you. But what’s going on with this? The part-whole effect suggested we’d be bad at recognizing parts on their own, but it didn’t say anything about parts looking different in different settings!

Enter the composite face effect (or CFE), another classic behavioral effect in face recognition that has been studied for years and suggests that we’re doing something to process face images that involves measuring the whole pattern at one go. The basic phenomenology of the CFE is as follows: Imagine I give you two faces to look at, and I ask you to only worry about the top half of each one. That is, your job is to completely ignore the bottom half of these pictures and only tell me if the two top halves I’m presenting you with look the same or different. The tricky bit is that I am going to play around with the bottom half of these images to try and complicate things for you. In particular, the crucial result that more or less defines the CFE is that identical top halves of face images tend to look different if I give them different bottom halves. You can see an example of this below - on top, the left and right images are exactly the same, but the different bottom halves may give you the sense that they look a little different. In the lab, we can measure this effect by comparing your accuracy for judging whether those top halves are the same or different under these circumstances and compare that to your accuracy when there is either no bottom half messing with you at all, or a misaligned bottom half.

Figure 1 from Watson (2013). All the top halves in these pictures are identical, but the first two in the top row probably look different to you due to the differing bottom halves. Misaligning the bottom half or turning the faces upside-down tends to reduce this illusion.

What we think is happening here (and in the part-whole effect) is that your visual system is doing something to measure facial appearance that is either holistic (using the entire pattern) or at least at a larger scale than these small parts of the face. The result is that the appearance of the bottom half ends up tangled up with the appearance of the top half in a way that can’t really be undone without some kind of external cue like mis-aligning or inverting the image. This is also what we think is happening in the Eye-Q game of course, which relies on this illusion to make an already-difficult party game even more challenging. The appearance of your face similarly gets entangled with the appearance of the eye, making it harder to match the celebrity eyes you are seeing in the game to your recollection of what those eyes look like in the context of the original face.

Together, all these results help make a strong case that faces aren’t just interesting because they’re socially important or because you have a lot of skill at recognizing them. Faces are also important because they are a nice example of how some objects and patterns may be perceived and recognized differently. Whether or not these kinds of differences are solely confined to faces or may apply to other objects you either gain extensive expertise with or have to recognize at particular levels of granularity has been an ongoing debate in the field for some time. These results I’ve described that are measurable for face images and the possibility that large-scale or holistic processing may be applied to other images leads us very nicely into our target article, which is an attempt to look for evidence of holistic processing for chess positions.

Seeing the board holistically

The question that the researchers behind this study wanted to answer was whether or not strong chess players might also be processing chess positions holistically in the same way that the rest of us (who are strong face recognizers) process faces holistically. At first glance, this might seem like a somewhat odd connection to make, but remember: One perspective on why we see holistic processing for faces is that it may be a type of visual processing that shows up for other patterns you recognize at an expert level. Chess masters looking at chess positions would certainly qualify in that regard, and I’ve written before about the evidence suggesting that stronger players are capable of processing larger regions of the board at once than weaker players.

To look for evidence of holistic processing in chess masters, the authors of this study went looking for a version of the CFE with face images and with images of chess positions. In both case, the images were cut in half with a horizontal line, and the top and bottom halves of both kinds of images could be independently manipulated. On congruent trials, the status of the bottom halves that participants were asked to attend to would match the status of the top halves of the images they were considering. On incongruent trials, the status of the bottom would be different from the status of the top - note that this is a slight variation of the CFE as I described it, but the overall idea is the same. An observer who is not processing these images holistically should be able to “tune out” the irrelevant part of the images and do about as well on congruent trials as incongruent trials. By comparison, a holistic processor should find incongruent trials harder to deal with.

Figure 1 from Boggan et al., (2011). Both chess positions and faces were manipulated so that top and bottom half appearance could either be congruent or incongruent. Holistic processing of either stimulus should lead to greater difficult on incongruent trials.

The task the authors asked their participants to carry out is a little more complex, too, and makes it possible to ask some interesting questions about possible interference between chess processing and face processing. Participants were asked to view interleaved chess and face images, responding on each trial about the match or mismatch between the image they could see now compared to the image they saw two images ago. By constantly interposing an image of the other type between the two images you were comparing on each trial, the authors were hoping to see evidence that holistic processing of one category might get in the way of holistic processing of the other one. This would mean that chess experts might show a very strong CFE for chess boards, but a weaker one for faces.

The first big question though, is whether or not there is a “composite chess effect” at all! Do players of any strength process chessboards in a manner that seems holistic, at least as measured by this effect? In the figure below, you can see the average accuracy across player groups for congruent and incongruent same/different judgments (indexed here with a quantity called d-prime) as well as the difference between congruent and incongruent performance. That latter difference score is a quick way to see if performance with each type of image seems holistic: A tall bar in this graph means that the incongruent trials were harder than the congruent trials, which should only happen with greater holistic image processing.

Figure 2 from Boggan et al. (2012). The size of the composite effect for face and chess images is evident in both the raw d’ scores (top row) and the difference scores between congruent and incongruent accuracy (bottom row). Strong players have a clear composite effect for face AND chess image, while novices do not.

The punchline from these results is that the Expert players (ELO ~1900-2700) do indeed appear to process faces and chess positions holistically, but weaker players don’t! By itself this is a compelling result that indicates a way in which stronger players see the board in a different way than weaker players. What is that difference, though? I find it a little easier to think about this in terms of what the weaker players (like me) may be seeing, or perhaps more importantly, what they are NOT seeing. The lack of a strong composite effect for chess positions in the weaker players suggests that rather than visually processing the board at a larger spatial scale, these players are limited to taking in visual information in relatively small pieces - a bit like “tunnel vision” if you will. In other visual tasks, that failure of larger-scale processing means there is a sort of additional step you need to take to integrate or combine the parts that you see into a larger whole, which is one more error-prone process that can be inaccurate or slow. This all reminds me very much of my recent conversation with CM Can Kabadayi about “Sniper Bishops” and blunders of misperception - holistic processing of the board almost certainly helps a player notice long-range threats over the board, and the lack of it makes us patzers much more prone to nasty surprises.

There are a number of other interesting results we could talk about, but I want to highlight just one more for you that I think is particularly neat. Remember that part of the idea behind the interleaved faces and chess positions that the researchers showed their participants was to look for evidence of interference between chess and face processing. Another intriguing way that they looked for evidence of some sort of competition between visually processing faces and chess boards holistically was examining the correlation between experts’ starting age for chess and the magnitude of their composite effects. While this did correlation was NOT significant for a composite chess effect, it did reach significance for the composite face effect! Their data shows that starting chess at a younger age led to a smaller composite face effect, perhaps indicating that increased expert (holistic) processing of chess boards incurred a bit of a cost to expert face processing.

Figure 4 from Boggan et al., demonstrating that starting chess earlier in life leads to a smaller composite effect for faces! Chess may compete with face processing for holistic processing.

What does a position look like?

These results suggest that strong chess players may “see” chess positions differently than weaker players, and gives us a word (”holistic”) to describe what the difference is. That gives us some idea of how board vision works differently with increased expertise, but what I’m still interested in (and what these data don’t speak to) is what that actually means in terms of the experience of looking at the board. What does a master subjectively experience that’s different from what I see? What (if anything) actually appears different to them? To be fair, these are questions that are always hard for a vision scientist - we measure things like accuracy, speed, and the effects of image manipulations on these data, but we don’t know how to work from those back to experience itself. For now, this study (and others like it) give us a tantalizing hint that there’s something different there worth trying to understand more fully.

Support Science of Chess posts!

Thanks as always for reading! If you’re enjoying these Science of Chess posts and would like to send a small donation my way ($1-$5), you can visit my Ko-fi page here: https://ko-fi.com/bjbalas - Never expected, but always appreciated!

References

Boggan, A. L., Bartlett, J. C., & Krawczyk, D. C. (2012). Chess masters show a hallmark of face processing with chess. Journal of Experimental Psychology: General, 141(1), 37–42. https://doi.org/10.1037/a0024236

Murphy, J., Gray, K. L., & Cook, R. (2017). The composite face illusion. Psychonomic bulletin & review, 24(2), 245–261. https://doi.org/10.3758/s13423-016-1131-5

Sinha, P., Balas, B., Ostrovsky, Y., & Russell, R. (2006). Face Recognition by Humans: Nineteen Results All Computer Vision Researchers Should Know About. Proceedings of the IEEE, 94, 1948-1962.

Tanaka, J. W., & Farah, M. J. (1993). Parts and wholes in face recognition. The Quarterly journal of experimental psychology. A, Human experimental psychology, 46(2), 225–245. https://doi.org/10.1080/14640749308401045

Thompson P. (1980). Margaret Thatcher: a new illusion. Perception, 9(4), 483–484. https://doi.org/10.1068/p090483

Watson T. L. (2013). Implications of holistic face processing in autism and schizophrenia. Frontiers in psychology, 4, 414. https://doi.org/10.3389/fpsyg.2013.00414

“If Hal’s there,” I told my wife, “that’s kind of a sign from the universe that we should bring him home.”

Hal as he was featured on the shelter’s adoption page.

We got to the theater and sure enough, there he was in the lobby. He was a tiny boy - the site listed him as a 1-year-old 5-pound cat, but seeing him in person he was even smaller than that sounded. He was draped in the arms of his foster mom looking like he had no worries in the world, and when she asked if I wanted to hold him he gladly settled in with me in much the same manner. He was soft, fluffy, and an absolute purr machine.

We put in the adoption paperwork online as soon as we got home from the theatre and then we waited. After a few days we were starting to lose hope: He was listed as “adoption pending” but we had received no phone calls. Oh well, I thought. He was a nice guy to meet.

Ah, but then we got the call and made plans to meet him on the 19th. He was just as cuddly and calm at the shelter as he had been at the theatre and I was sold. We bundled him up and took him home, getting him all set up in our upstairs office with a soft bed, a comfy cat yurt that Clementine has always ignored, food, water, and his own litter box. He had an accident that night right as we brought him up for bed - a little bit of pee on the comforter that he seemed very upset about. I gave him lots of reassurances, we cleaned up the bed, and put a waterproof liner down just in case. I stayed with him for a little more than an hour reading some books and cuddling in our papasan chair - every time I turned a page he waved his forepaws at it, but never tried to claw or grab.

Hal hanging out in his Day One room.

We had a nice calm weekend with him after that, introducing him slowly to the house and making tentative introductions to Clementine. She was mostly just a giant grouch about it, but her hissing and low growling seemed perfunctory to us. “Her heart’s not in it” Erin said, “She likes him and is a little mad that she likes him.” By Sunday, they were sitting in the same room without incident and by the time we left for work on Monday morning we were feeling good about letting them hang out all day. It worked just fine and it didn’t even take long before we saw them sneak some actual playing in along with some mutual grooming. Our work week was busy with our usual schedule, but Hal was waiting for us when we got home, eating like a champ (we worried he was malnourished) and drinking a ton of water. We played with his favorite ribbon toy so he could show off his pouncing skills (a ground game was better than aerial tricks) and head scratches and lap-sitting sessions were still a favorite.

Enjoying sunbeams on our living room chairs.

On Saturday morning he seemed maybe a little quiet, but was cheerfully watching birds out our back window with Clementine right next to him. He wasn’t eating as much, though, and seemed a little on the quiet side. We didn’t think a thing of it until dinnertime, when he started to pee frequently outside the box in small amounts. Erin looked up what this might imply and made the decision to call the emergency vet. We thought maybe we were looking at the onset of a urinary tract infection - something that needed attention right away, but something manageable.

After dropping him off it was a few hours before we got the first call from the doctor. His abdomen was “doughy,” she said, and he had a high fever. He was laying on cooling mats for his comfort and an ultrasound had indicated he had fluid in his belly. It could be a lot of things, she said, and none of them were good. She asked to test the fluid and his blood and said she would call back.

Around 1am she called and the news was about as bad as it could be. The fluid in his abdomen was full of white blood cells fighting off septic peritonitis. She suspected some kind of GI tract rupture (“Did he eat something?”) but we couldn’t think of anything that could have caused such a thing. He hadn’t been a very nosy cat at all and mostly seemed to gravitate towards the best sunbeams the could find in the rooms we’d given him access to (the two brown chairs by the back living room windows were perfect). Whatever the cause, our options were to put him through an uncertain and dangerous exploratory surgery (one that would end at best with an abdominal drain, the ICU, and resected intestine or stomach) or to say goodbye.

Just a week. Just a day for things to turn south. Given that we’re only talking about a very short length of time, I almost feel silly about how much it hurt to hear the doctor say this and know what the right answer was. We hung up. My wife and I talked it over, and then we went to our daughter’s room to tell her what was going on. We all made the decision to go say goodbye and piled in the car after telling the doctor we were ready to let Hal go.

I don’t have much to say about his actual passing save that the staff was very kind, Hal was very sick, and I held him and did my best to tell him how much we loved him right up to the end. I’m someone who tries to deal with a lot of the problems I run into in my life by finding ways to communicate and that makes situations like these where you aren’t sure how much you can get across very hard. I thanked him for being my friend and told him I was so sorry we didn’t get more time.

Now, here I am (and here is the rest of my family) feeling terribly sad and trying to work out where to put that sadness. I’m writing to put some of it here, but I’m also writing because I think that Hal’s life deserves some kind of record or memorial, as short as it was. I knew him for about 9 days and he still made enough of an impression on me that I find myself grieving on a beautiful autumn morning, wondering what it was about this little cat that keeps making me cry.

I think it is as simple as this. I have owned cats since I was in college and while I do enjoy their company a great deal, I have to admit that I am also something of a selfish owner. I gripe if my cat keeps me up at night. I get annoyed when my cat stands up in front of the television to paw at the screen while I’m trying to watch something. Often, I wish that Clementine the cat was more inclined to snuggling than she is and I get frustrated that she always seems to prefer darting off my lap to chase a dust mote or a bit of light on the wall. Maybe this is all ordinary, but when I see people who dote on their cats to a higher degree than I am generally willing, I worry that I am not the best cat owner.

For whatever reason, however, Hal came home and absolutely delighted me. He was scared and lonely his first night and sitting up with him was the only thing I wanted to do. He was a little uneven with his litter box aim, but I felt deep reserves of patience about this even while cleaning up after him. I left work a little early during the week so I could get home to play with him and thought about books he might like to read with me. Something about this little guy made we want to be a better owner to him and to Clementine and I was having such a good time making him happy. What I think Hal reminded me of very powerfully is how nice it is to be kind and how the work of kindness hardly feels like work at all once you decide to do it.

Maybe that sounds like a lot for a little cat to do. It may be, but I say that he did do these things and I would know because he was my friend. We hung out, we chased the ribbon toy, we read books, and we appreciated sunbeams together. For that, thank you, Hal the cat. I wish we could have been friends a lot longer.

My buddy Hal, curled up with me on his first night at home.

Hal speaking his mind. Not a loud man, but one with clarity of expression.

One key question we have to return to multiple times during the term is the question of how best to characterize different aspects of cognition and behavior. How do you measure the mind so that you can try to link mental processes to the brain? That question can seem easy to answer when we want to measure the limits of someone's memory, but it quickly becomes apparent that a lot of intuitive, everyday experiences and tasks are more challenging to turn into a number. To get my students thinking about these issues early in the term, I usually introduce them to some tools for measuring the mind that are informal, but also a lot of fun.

Random Internet Quizzes, Psychometric Validity, and Personality

I am a total sucker for a good internet quiz. I would love to find out what country I should visit next, which movie sidekick would be my ideal brunch buddy, and the dessert that most closely matches my aesthetic. (In order, Japan, Chewbacca, and Ebelskivers). These quizzes aren't serious, of course, but they're as fun as they are because we sort of care about whether or not the answers match whatever intuitions we have about these things. If I take a "Which Disney Princess Are You?" quiz and get any answer besides Belle, I am MAD.

By http://static3.wikia.nocookie.net/__cb20120627060108/disneykilalaprincess/images/5/53/Belle.png, Fair use, https://en.wikipedia.org/w/index.php?curid=41755941

For my students, I use these to get them thinking about a key psychometric concept called validity. A simple way to think about what this means in the context of trying to measure some aspect of the mind is that the validity of a test or scale refers to how well we are measuring what we intend to measure. Measuring your memory by asking you to jump rope clearly has poor validity, for example, but measuring your spatial cognition by asking you to do a mental rotation task seems alright. A question I ask them to think about with regard to these quizzes is how they'd try to establish the validity of the sillier ones: Can you come up with a way to find out if a quiz came up with the right Disney Princess? What exactly would it mean to be wrong?

These frivolous quizzes are a gateway to talking about other aspects of the mind that we have strong intuitions about, but that are also difficult to measure. Specifically, it's not a big leap at all from Disney Princesses and brunch buddies to something most of my students are both very interested in and have strong feelings about: Their personality. Look, we all know what personality is, right? People are the way they are and that "way that they are" is what we mean by the word personality. Some people are nice, some people are not-so-nice, some people are "quiet types", and so on and so on. But what if we want to get serious about measuring personality? How do we turn these intuitions about the people we interact with into something quantitative? Once we do, how do we decide if what we've done has any validity?

What does it mean to have a chess personality?

As usual, chess is a fascinating domain for considering these questions. We have a lot of good quantitative data about players, we have a rich vocabulary for describing different aspects of chess positions, and we also have a lot of intuitions about how players differ from one another. I know I've brought up Fischer's "My 60 Memorable Games" here before, but it really deserves some special attention in this regard due to how well it describes how players like Keres, Petrosian, Spassky, and Geller each pose their own unique challenge due to their approach to the game. Part of what I enjoy about playing through classic games or watching current tournament play is seeing different chess personalities on display: Which players seem absolutely fearless and which ones are more willing to slowly squeeze the life out of their opponents? Perhaps more importantly, which one of those truly great players do I think I'm the most like?

This guy - it's definitely this guy.

Chess players clearly love thinking about that last question. There are many online opportunities to try and estimate your own chess personality, including offerings from chess.com (you can go check out chesspersonality.com), chessiverse.com (https://chessiverse.com/personality-test/), and a very nice recent blog post by Julian at Chess Engine Lab (aka @jk_182 on lichess) adopting a data-driven approach to measuring the style of GMs. What I want to do here is say a little bit about how we might try to estimate something intuitive, but complex and multidimensional like chess personality by unpacking how psychologists developed the tools we use to estimate personality. This will be just a brief overview of a rich and complicated topic that experts still argue about, but I hope it gives you some interesting food for thought with regard to how we can try to capture what's different about how different people play the game.

Chess personality categories from chesspersonalitytest.com

The OCEAN of Personality - Why are there 5 things in the "Big 5?"

While personality psychology is not my area of expertise, I feel pretty confident in saying that the most widely accepted model of what we mean by personality is the "Big 5" Personality traits. The name refers to the 5 factors that make up the model, each of which represents some aspect of personality that individuals are scored on through the use of established surveys. Those factors are often taught to students using the acronym OCEAN, which stands for the traits of Openness to Experience, Conscientiousness, Extroversion, Agreeableness and Neuroticism.

By Original: Anna Tunikova for peats.de and wikipedia Vector: EssensStrassen - https://peats.de/article/big-five-die-personlichkeit-in-funf-dimensionen, CC BY 4.0, https://commons.wikimedia.org/w/index.php?curid=113609961

But wait a second: When I say "established surveys," how did these get established? And why are the OCEAN traits more widely accepted than other models? You might have come across a personality test called the Meyers-Briggs test, for example, which will give you a personality type based on 4 binary traits (intro/extroversion, thinking/feeling, sensing/intuition, and judging/perceiving). Why does that test have four things in it while the "Big 5" has, well, five? Come to think of it, how did the people who came up with either model (or any other alternatives) settle on a specific number of personality traits and specific descriptions of those traits? The answers to these questions have more to do with data science than they do with psychology, at least in my estimation. I say this because at the heart of these personality models (and many other similar attempts to measure something complicated about the mind) is the idea that there is some useful structure, or shape, in the data we can collect from different people when try to measure how they are different from one another. Briefly, here is a look at where the Big 5 personality model came from, which I think serves as a useful starting point for thinking about how we could try to define a chess personality.

A recipe for building a personality model

The first thing we have to do if we want to assign personality traits or categories to different people is come up with some things to measure to get us started. A model of personality begins with a set of measurements (in this case usually survey questions we ask people to answer about themselves) that we think will have something to do with the different ways that people act. If we're really building a model of personality from scratch, we have to assume we don't know a great deal about what we're likely to find, so we should make sure to include a lot of different questions so we don't miss anything. In terms of the history of personality psychology, this meant asking people to either describe themselves or others using a long list of over 150 different words that could describe different ways people tended to act: Would you say you are a lazy person? Are you funny? Do you often feel nervous?

Stock Image.

Here is the hard part about this step: When can you be sure you haven't left something out? Those 150 words or so sound like quite a lot, but that doesn't mean there isn't some collection of words we didn't think to include that might matter a great deal (maybe bookish should be in there, for example, or how about vengeful or delulu?). There is no especially satisfying answer to this question, unfortunately, so at some point we simply have to make a choice.

Armed with that data, the next step is to go looking for structure in the data we collect from many different people. But what do we mean by structure? What we often mean by this from a data science perspective is that we're interested in finding out how our different measurements vary together (formally, their covariance)- if someone says that they are lazy, does that have anything to do with how funny they said they are? Are there specific combinations of scores across these words that we tend to see over and over again, leading to "clumps" in our data? To the extent that this is true, it tells us that those qualities may refer to the same thing and so (1) We don't need to measure both of them (2) We might have learned something about core elements of personality.

Covariance examples from geeksforgeeks.com

Thankfully, there are many different statistical tools for asking questions about covariance and "clumpiness" across measurements, even when we have dozens and dozens (or more!) of variables we want to think about at once. These include tools like Principal Components Analysis, Independent Components Analysis, Manifold estimation techniques, k-Means clustering, and many others. In general, these tools give us a way to turn a large set of measurements into a smaller set by taking advantage of covariance relationships.

But here is the hard part about this step: How do we know what kind of structure to look for, especially with so many measurements? Most techniques for finding covariance, clumpiness, or whatever words you want to use to describe structure in rich data do their work by making some assumptions about the structure that might be in there. Let me show you one famous example below: If we asked people to respond to just two questions and plotted the two values each person gave us on an xy-axis, let's imagine it makes the very interesting shape below.

If you were to try and find a simple correlation in this data, you'd find out that the two variables either weren't correlated at all, or were correlated only weakly with one another. If you stopped there, you'd have a nice, tidy result you could report, but you also would have missed the beautiful spiral (or "Swiss Cake Roll" as this is sometimes called) that truly captures the clumpiness of this dataset. Again, there is not a great solution to this other than to try some different things out and ultimately make the best decision you can. Another decision you may have to make will be how many pieces your final structure will have. If you look at the data I've plotted below, how many clumps do you see? When I ask students about this in the Machine Learning unit of one of my classes, I usually get a range of opinions: Maybe 6 or 7? Maybe as many as 12? It all depends on how big you're willing to let the clumps be and there isn't necessarily a clear right answer for complex data sets with many more variables.

In the case of the Big 5, examining participants' answers about dozens of different personal qualities with a technique called factor analysis led several groups (in some cases using different questions as a starting point) to the same conclusion that 5 "factors" captured the covariance between survey questions. By looking at survey questions that contributed most to those factors led us to the OCEAN model that has seen steady academic use ever since.

So can we do this for chess?

That is a very brief (but perhaps too long for a blog post!) history of how psychologists built a model of personality. Now, the real question that is the focus of this article: Does this suggest a way to construct a good model of what a chess personality? To talk through what I think is interesting and challenging about doing so, I want to consider each step in the recipe for a personality model I described above to see how this would work (or maybe not work) for chess.

What should we measure?

I said that personality models start with decisions about what to measure and chess presents with a range of interesting possibilities. In @jk_182's recent post, "playing style" was determined by a range of different game statistics that captured how specific pieces were moved during a player's games. Are you a player who is willing to forego castling in favor of a strong attack? If so, that might be something we can measure the frequency of throughout your playing career as part of your chess personality. This sounds like a decent start, but with the usual caveat that we'll want to include as broad a range of moves as possible to be sure we don't leave out important behaviors.

How often you move each pawn and where they tend to end up is one way to try and describe your playing style.

But there is something I can't help but worry about and it has to do with one of the big differences in chess personality between myself and (for the sake of argument) let's say GM Hikaru Nakamura: Part of Hikaru's "chess personality" is that he is very, very, good and I am comparatively very, very bad.

One of us is all the way to the right of this distribution and the other...isn't.

But wait? Is playing strength actually a part of chess personality? This is something that I suspect may be tricky about determining a chess personality relative to psychological models of personality. In chess (unlike personal qualities) there are objectively better or worse moves for a player to make. Some of my playing history of making various kinds of moves may reflect my chess personality, but some of them also reflect my lack of titled-player skill. This might mean that when we try to learn about the factors underlying chess personality, some of those factors are actually about being good at the game, which doesn't seem quite like what we mean.

How do we separate chess skill from chess personality?

There are a few ways we might try to avoid this problem and get to a way to determine your chess personality that isn't dominated by your playing ability. One possibility is to develop different models of chess personality for different classes of players (GMs get their own model, IMs get their own, and patzers get still another). But here are some ideas for thinking about chess personality without separating players into these kinds of skill classes.

1) Use openings!

If we want to separate skill from personality, one way to think about this is to assume that personality affects how you choose between moves that are objectively equal. If one player chooses the best move in a position and another tends to choose a worse one, it's not obvious that we should call that a personality difference. The trouble is that in many positions there is a relatively narrow set of possible moves that are of equal worth. But when are there lots of options of approximately equal value? On the first move!

We all know that there are e4 people and d4 people.

Opening choice turns out to have some neat clumpiness to it, too. In a relatively recent paper by De Marzo & Servedio, the authors went looking for structure in data describing the opening play of a large database of Blitz players. Briefly, their goal was to identify families (or clumps) of openings based on player co-occurence: If lots of players like to play the same two openings, that's a good hint that these might be part of a meaningful opening family. Below, you can see the clusters they identified in their data, which could be used to identify players' personality based on opening repertoire across these groups.

Figure 1 from De Marzo and Servedio showing the different families of opening clumps they discovered from player-opening Blitz data.

2) Use a forced-choice survey

This next idea is a little bit tougher, but might also be a way out of the skill/personality conundrum. While there is often an objectively best move in a given position, it can also be the case that there are at least 2 or 3 moves that all keep the evaluation at about the same place. Which one should you pick? Perhaps the decisions people make in those situations are down to taste, or personality. My suggestion here is that one could possibly build a short survey using positions like this and present players with the set of equally good moves to choose from. This isn't far removed from collecting moves from a huge library games as we're still interested in which moves players tend to favor, but we'd be constraining that data to focus on distinguishing between equally-good choices.

3) Use puzzles!

This last one is perhaps a little dubious as a way to disentangle skill from chess personality, but it's something I've been curious about for a while. Rather than trying to avoid situations where there is a clear best move, maybe those are exactly the situations to consider! Specifically, performance across different puzzles may be a useful means of working out what kinds of ideas tend to occur to players more readily as a way of characterizing their style. Yes, stronger players will be better at puzzles in general, but perhaps there will still be room to see what kinds of moves they tend to find the hardest to come up with when they know there is a best path.

Interestingly, Lichess (like other sites) has an existing classification of puzzle types and also incorporates these into a Puzzle Dashboard graph I can't help but find intriguing. Using different players' performance across these different categories would potentially be a neat way to determine a "chess personality" we could compare to other data sources (an idea I raised in my blog post about predicting puzzle difficulty). Now this does raise the question of how meaningful those puzzle types are in the first place (its own neat data analysis question), but still a potentially fun sandbox to play in.

How many pieces are there to chess personality?

Once we settle on a strategy for measuring personality with a lot of variables that we think are meaningful, there are still a lot of questions to think about. In particular, how do we go looking for structure in those measurements? Do we think chess personality is meaningful to talk about in terms of continuous dimensions (traits like we find in the Big 5) or categories (like the pseudoscientific Meyers-Briggs test - the scale isn't valid, but there isn't anything inherently wrong with categories!)? In either case, how many types of personality should we go looking for? If we're lucky, the data may give us some big clues, but in analytic problems like this we're often not especially lucky.

This is a setting where I'd say we may have to rely a lot on some domain knowledge from people who have known a lot of players. This also applies to the problem of deciding what to call the various factors or clusters we identify in our data. It's lots of fun to label ourselves as "Romantic" or "Defensive" but the data may present us with some combination of playing choices that are harder to pin down. In such cases, landing on a good interpretation is as much about finding good ways to communicate what you're seeing to others as it is about thinking about co-variance, clumpiness, and information criteria.

Closing thoughts

Like so many interesting questions about the mind, pinning down what a "chess personality" might be is a little maddening and potentially a lot of fun. It also has the potential to start a lot of good conversations, maybe a few solid arguments, and make room for a lot of neat data science projects. If you find these topics interesting, I'd encourage you to browse the ever-increasing library of available games and player data (including the remarkable Lichess Open Database) to see what you can see. While perhaps we shouldn't take the idea of a chess personality especially seriously, it's also a nice excuse to try and understand one little corner of human behavior quantitatively.

I know this one was a long one, but if you've made it through to the end I hope you enjoyed it! I'm hoping to have some more posts soon focusing on some smaller topics, so stay tuned for more Science of Chess in the near future.

Support Science of Chess posts (and others)!

Thanks as always for reading! If you're enjoying these Science of Chess posts and would like to send a small donation my way ($1-$5), you can visit my Ko-fi page here: https://ko-fi.com/bjbalas - Never expected, but always appreciated!

References

De Marzo, G., Servedio, V.D.P. Quantifying the complexity and similarity of chess openings using online chess community data. Sci Rep 13, 5327 (2023). https://doi.org/10.1038/s41598-023-31658-w

McCrae, RR; John, OP (June 1992). "An introduction to the five-factor model and its applications" (PDF). Journal of Personality. 60 (2): 175–215. doi:10.1111/j.1467-6494.1992.tb00970.x. PMID 1635039. Retrieved 2025-03-17.

Link to "The Personality Brokers" by Merve Emre, a fascinating account of the development of the Myers-Briggs Personality Test https://app.thestorygraph.com/books/687a7765-da8c-4ddc-b6f8-2ae616417dae

A much younger NDPatzer looking at some EEG data from the Balas Lab at North Dakota State University.

Those dreams always feel loaded with promise and potential - this is when we think through how to use new tools for changing the appearance of faces or textures in clever ways, or consider how to get new kinds of data out of behaviors like drawing, coloring, or even just looking at things. Our discussions about what to do next always start with some idea we're excited about - some possible story about how human vision works that we think might be important. If we're lucky enough to be right, maybe this is when we'll find something out that really pushes the field forward.

"Kill your darlings."

That's the fun part.

The difficult part of all of this is that in the lab we can't just let ourselves keep dreaming of the exciting stuff that might happen in our experiments. To be more specific, the trap we have to avoid is getting too carried away by the possibility that we might be right. In the lab, it's critically important that we think through what should happen not just if we're right, but also if we are completely mistaken. Sure, those neat image manipulations and clever tasks often arise from thinking through how people should behave if we're right about the human visual system, but to ensure that our experiments can tell us anything meaningful we have to invest a lot of time poking holes in our own best ideas. What is something that should only happen if we're wrong? Hopefully we can come up with a few different ways we could be mistaken and a few corresponding ways to check. You can't hope to rule out every competing account, but you've got to give it a real try.

In the context of writing, there's a similar principle that is sometimes expressed via the phrase at the beginning of this section: Kill your darlings. For an author, this phrase is intended as advice to be merciless when reviewing your own work. Maybe you loved a particular turn of phrase or were dying to start your story with an introduction that you thought was surprising and creative - that's great and all, but when it's time to edit your work, you can't let your love for the first draft stop you from rejecting those ideas if they lead to a better second draft.

It's tough. It's REALLY tough sometimes. As a fairly regular blogger, I can't tell you how many times I've had to really force myself to change passages (or reject entire posts!) that I felt invested in but that ultimately weren't very good. Besides the challenges of facing up to sunk-cost reasoning, killing your darlings requires you to make peace with the idea that even if you really liked something it still might not be all that great. Difficult, to be sure, but often necessary to produce work that's worthwhile.

Confirmation Bias and trying to Minimax OTB

As in the lab and on the page, so also on the chessboard.

While there plenty of obvious differences, coming up with a good next move has a lot in common with designing a good experiment and writing good prose. In all three cases, you need to be willing to kill your darlings to really get ahead! What I mean by this is that over the board, it's crucial to avoid playing pure "Hope Chess" and imagining only the Best of All Possible Worlds for yourself. Sure, it's possible that your opponent will take the piece you're intending to sacrifice (brilliantly) and walk right into the 3-move mate you've calculated - but what if they don't? What if you're wrong about the next move they will make? To be a really strong player, you need to consistently think about the move your opponent could make that will cause you the most trouble and plan accordingly.

In the first class I took on Artificial Intelligence as an undergrad, we learned an algorithm for implementing this kind of reasoning called the Minimax Algorithm. Minimax is a decision-making procedure that uses a game tree (a branching structure capturing the full set of choices each player could make on each turn of a game) and an evaluation function (a way to measure the value of a position to a player) to determine which choices lead to the best outcome for one of the players. I'll walk you through a short example from geeksforgeeks.org in a second, but the underlying logic of the algorithm is that while the player wants to maximize the value of the position for themselves, decisions about the best next step need to include a minimizer who will always choose a next step that worsens the player's outcome as much as possible. To see how this works in a simple tree, take a look at the image below:

Image credit: Geeksforgeeks.org

If you're the player in this scenario, your job is to work out if it's better to take the path labeled L or the path labeled R. After you make that choice, your opponent will also get to choose which path to take at the next level of the tree and your "score" will be the number you end up on (the higher the number, the better it is for you). If we assume you're playing a game with perfect information, the idea is that you have access to those values to help you make a choice. Now here's the thing to pay attention to: That 9 at the bottom right looks REALLY tempting. It's by far the best outcome on the tree and you can only get there if you try the path labeled R. Maybe it's a good idea? You want to win, right? Why not aim for the outcome that's the best thing that could happen to you? After all, the opponent is one slip=up away from you winning big.

This, dear reader, is the maximizer talking - if we're only thinking about the best outcome we could reach, this sure seems like the right answer. The obvious (I hope) problem is that the maximizer is essentially very good at hope chess: You could end up at the 9, but your opponent could also choose to go left after you chose to go right and now you'd end up with a score of 2 - the worst outcome on the tree. To avoid getting stuck there, we have to consider the minimizer's relentless drive to punish us as well as the maximizer's sunny optimism. If we choose L, the minimizer will be sure to pick the 3 instead of the 5, so we have to consider that score to be the best we can do on that side of the tree. Likewise, if we choose R, we have to imagine that the minimizer will pick the 2 instead of the 9, making that the best we can hope for. Faced with the choice between the minimizer's 3 and 2, the maximizer may sigh a little, still dreaming of the 9 that might have been, and choose (correctly) to go to the left. In a small game tree like this with a reliable evaluation function, minimax is something that human players more or less implement well: If you've played Tic-Tac-Toe, Nim, or the Dot Game, you've almost certainly used some version of minimax reasoning fairly successfully.

Image credit: The original uploader was Yonidebest at Hebrew Wikipedia. - Transferred from he.wikipedia to Commons., Attribution, https://commons.wikimedia.org/w/index.php?curid=2237158

What about chess, however? How do we do at minimaxing when faced with a much larger tree and an evaluation function that engines have access to, but that humans can generally only guess at? The study I want to tell you about next takes a look at exactly this, with an emphasis on trying to measure how willing players of different strength are to really consider killing their darlings on the chessboard.

Chess calculation as hypothesis testing - Cowely & Byrne (2004)

The authors of this study recruited two groups of 10 participants each: A group of experienced novices with a mean ELO of about 1500 and another group of expert players (mean ELO over 2200). Both participant groups were asked to consider a set of 6 different positions, all of which had equal chances for White and Black and were middlegame positions with 20 or so pieces still in play. These positions thus still provided lots of options for candidate moves, none of which were especially forcing (see below for an example). For each of these positions, participants were asked to not just choose the move they would like to play next, but also to do their best narrate their thought process while making this decision. This is referred to as a Think Aloud protocol, and their goal in using it was to answer the following question: Do experts do a better job than novices at thinking through the worst possible outcomes?

Figure 1 from Cowley & Byrne (2004)

What I quite like about this approach is that it has the virtue of being quite simple for the participants to understand (they're mostly doing what they would usually do during regular play) but yields very rich data about the decision-making process. The tough part about data like this is working out what to do with it all, however! These days a lot of research groups are exploring how to use tools like LLMs to analyze natural language data like this, but those tools were still 20 years away when this study was conducted. Here, the authors used each bit of transcribed chess monologizing to come up with what they call a problem behavior graph. This is essentially the portion of the game tree that a player came up with in their transcribed speech, including both different move sequences they considered and their sense of whether this was going to lead to something good or something bad. The dashed lines you see here also capture an intriguing feature of this Think Aloud protocol: Players sometimes skip the opponents' moves! Though it doesn't seem like they use this data for much in the conference paper, I thought it was neat that it was part of the transcription.

Figure 2 from Cowley & Byrne (2004). This problem behaviour graph organizes the move sequences a player thought through, ending with the player's determination of whether this was a positive or negative outcome for them.

The authors consider the graphs generated by just 10 players (5 experts and 5 novices), which to be honest is something I don't love about this paper. To be fair, this is challenging work to do (the transcription process is a real time sink - ask me how I know) and recruiting any special population (like the Expert group) is always tough. Still, I do want to advise you to take these results with some serious grains of salt because our statistical power is quite low.

That said, the authors coded the various move sequences in each problem behavior graph according to how the players' evaluation matched up with an engine's evaluation (the venerable Fritz 8 here) of the position. They defined confirmation bias sequences as those where the player evaluated the sequence as positive but the engine ruled it negative. Similarly, they defined falsification sequences as those where both the player and the engine thought the sequence was negative. The question is, how often did players in each group generate each kind of sequence?

Table 1 from Cowley & Byrne (2004). The authors classified players' move sequences according to the relationship between their evaluation of the outcome and the engine's evaluation.

First, it's worthwhile to point out that experts produced more sequences than novices in general. While novices came up with about 6 move sequences that they considered on average, the experts came up with just over 8. In general, experts also tended to agree with the engine's evaluation more often than novices, generating about 6 correctly evaluated sequences on average compared to the novices' average of about 2.5. Experts think through more options and tend to be right about the outcome, too.

The potentially more interesting result, however, has to do with the coding of these sequences according to confirmation bias and falsification. What kinds of sequences are experts considering and how different does it look from the novices? It turns out that according to the authors' coding scheme, expert players engage in more falsification than novices and less confirmation bias. Again, be a little mindful of the stats here, but the figure below shows you the average number of sequences of each type generated by participants in the two groups. Modulo the worries about the small sample size, etc. this looks like evidence that stronger players spend much more time thinking about the minimizer than novice players do.

Cowley & Byrne's (2004) data comparing falsification and confirmation bias behavior in their Expert and Novice groups.

Some wishful thinking about the study and possible next steps

There are a lot of things I quite like about this study - enough of them that I when I was asked by NM Ben Johnson to describe 3 studies I thought chess improvers should know about, this was one of the ones I named! What stands out to me about this one is both the willingness to wade into complicated behavioral data (coding the Think Aloud data, specifically) and what feels like a more direct relationship to real OTB chess. I try to be careful in these posts to not present scientific research about chess as some kind of ticket to improvement, mostly because these studies aren't focused on those outcomes. What I think is neat about this paper, though, is that it does feel to me like it provides some insight into something novices could stand to work on. Kill your darlings. Remember that your opponent isn't going to be kind to you, so don't let yourself off the hook when you calculate. If you see a good move, wait, it may turn out to be a bad one.

The thing is, though, that this is also a study where I think the rivets show rather a lot. I really wish there were more participants. I wish the authors had reported how often players in each group skipped moves in the sequences they generated! I also would kind of like to see (my AI skepticism notwithstanding) what modern tools for summarizing natural language could do to augment this coding scheme. Besides these fairly simple ideas for things to do to expand on this work, I also can't help but think about their definitions of falsification and confirmation bias. In particular, it doesn't feel quite right to me to say that the novices are engaging in more confirmation bias when part of the problem is that they're also just wrong about the eval function more often! It's interesting to see (as the authors report in the text) that novices tend to come up with positive impressions of positions a little more often than experts, but that's a subtler point than the one they're trying to make.

Figure 3 from Cowley & Byrne (2004) - Here, the authors consider how often both groups tended to agree with the engine's evaluation of positions they calculated (Objective outcomes) as opposed to thinking positions were better (positive) or worse (negative) than the engine did. Novices tended to be a little more optimistic overall, but not by much.

I hope the above discussion isn't taking you too far into the weeds, but given the big ideas behind this paper I think it's especially important to remember that constructive skepticism matters in chess and in science. This stands out to me as a compelling attempt to measure something quite complicated: How do different players reason differently during attempts to calculate? There are hints of something interesting here, but also room to do more - after all, what if we're wrong? Compared to some of the research I've discussed here, this one feels especially ready for some replication and some extension. If you've got ideas about this, I'd love to hear them in the comments. Regardless, hope you enjoyed reading about this study and hope to see you next time!

Support Science of Chess posts!

Thanks as always for reading! If you're enjoying these Science of Chess posts and would like to send a small donation my way ($1-$5), you can visit my Ko-fi page here: https://ko-fi.com/bjbalas - Never expected, but always appreciated!

References

[Michelle B. Cowley-Cunningham](https://philpapers.org/s/Michelle B.%20Cowley-Cunningham "View other works by Michelle B. Cowley-Cunningham") - 2004 - In K. D. Forbus, D. Gentner & Regier, Proceedings of the Twenty- Sixth Annual Conference of the Cognitive Science Society. pp. pp. 250- 255..

Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis:verbal reports as data. USA: MIT Press.

Newell, A., & Simon, H. A. (1972). Human problem solving. Englewood Cliffs, NJ: Prentice-Hall.

Wason, P. C. (1960). On the failure to eliminate hypotheses in a conceptual task. Quarterly Journal of Experimental Psychology, 12, 129-140.

Then I played some more the next day.

Friends, I had not ascended. Maybe it was the weak hotel coffee or maybe it was the unsettling dreams I’d had the night before while the in-room air conditioning unit rattled and chugged away in the corner. Whatever the reason, nothing worked. My opponents were only playing openings I hated to face and I was getting stuck in all the awkward lines I hadn’t yet taken the time to study. Now I was the one continually under pressure and all the forks, sacrifices, and skewers I’d seen just the day before escaped my notice in game after game. As I watched my rating regress back to where I’d just ascended from, I wondered which day was the fluke: Was I secretly much better at chess than I might have though, or secretly much worse?

Hot and cold streaks are an inescapable fact of chess, baseball, basketball, and most any other endeavor where we measure performance in successes and failures. A long run of successes can make you feel like you’re “in the zone,” while a long run of failures raises the spectre of being “on tilt.” How much should we read into hot and cold streaks, though? Do they really reflect anything meaningful about the state of an individual player during either a winning streak or a losing streak, or are they sound and fury that signify only the statistics of binary outcomes? I've read a few different posts lately about the relative contribution of luck vs. skill in chess: To what extent is our performance about our cunning plans in this game of perfect information and to what extent are we all taking random walks around the true value of our skill level?

Whether winning or losing streaks are “real” in the sense that they represent meaningful departures from randomness has been examined with a few different datasets, and in this short post I wanted to share a recent analysis along these lines using a large database of online chess community data. The analysis I’ll share with you is just one part of a bigger study with other interesting results I’ll write about later, but I thought this piece of their work was a nice look at chess expertise, losing slumps, and winning streaks via a quantitative analysis that’s fairly easy to describe.

Streaks are (probably) inevitable

When you're in the middle of a streak, good or bad, it can feel like you're being swept up in something that demands an explanation. Surely 10 wins in a row must mean something is afoot! Likewise, losing 12 or 13 in a row must be some kind of hint that there is some reason you're spiraling deeper and deeper into the ELO abyss. The thing is, if we're thinking about binary outcomes like winning or losing a game (we'll forget about draws for now), a series of outcomes is bound to have some runs of either 1's (wins) or 0's (zeros). There's an exercise I give students in one of my computational classes to illustrate this point - the goal is to build intuitions for stochastic systems and demonstrate the value of simulating random processes to learn more about their behavior, but it's also just fun to watch them get surprised.

One group of students gets an actual quarter, and I ask them to flip it 150 times and write down what they get in order. Another group of students does NOT get the quarter, but I ask them to imagine flipping one 150 times and to record the imagined outcome of the flip each time. In both cases, we get a list of Heads and Tails outcomes, but every time I've done this exercise with students there is an immediate difference between the two lists as well. The first one, with the real quarter, usually has a run of either Heads or Tails that's 7 (sometimes more) in a row. The second one, that students just imagined, tends to top out at streaks of about 4 or 5. If you ask the second group why they didn't put in bigger runs of Heads or Tails, they'll usually tell you that it "didn't seem random enough" to have the same thing come up again and again. There's a nice cognitive science point here about the Gambler's Fallacy and a nice point about probability: What appears improbable to the human mind may in fact be quite likely if you know how to calculate!

Image credit: Joshua Hoehne

In this case, you can see why a 7-streak in 150 coin-flips isn't so strange fairly easily with a quick-and-dirty approximation. If we were thinking about getting 7 heads in a row or 7 tails in a row from EXACTLY 7 flips of the coin, that is quite a long shot. How long? There are 2^7 possible series of heads and tails we could end up with, and just 2 of those (7 Heads or 7 Tails) meet our criterion. That means that we've got a 1 in 64 shot of getting our streak. What about our 150 coin flips? This is the quick-and-dirty part, but we can think of this series as a bunch of overlapping 7-flip sequences - 144 of 'em to be exact. If you've got a 1 in 64 shot of something happening, it's not terribly likely if you only get one try. If I give you 144 tries, though, it's much more likely to actually happen.

We've played with these kinds of problems in my classes with other stochastic systems. One of my favorite assignments was asking students to analyze Joe DiMaggio's MLB record-setting hitting streak of 56 games. This one's particularly fun because at least a few years ago, it was the case that you should expect a streak of that length just about once in the history of baseball, so on one hand it maybe wasn't terribly surprising that someone did it. The even better catch, though, is that it turns out to be pretty unlikely that a player with DiMaggio's batting average (lifeitme average of 0.325) would do it - Ty Cobb probably should have been the guy (l0.366). Modeling random systems carefully and playing with the parameters in your model is a great way to understand what is and isn't likely in a concrete and computable way, and helps you get some intuition for thinking about what should and shouldn't surprise you.

Public Domain photo. Ty Cobb may not hold the record for MLB's hitting streak, but if there's a record for sliding "spikes high" into home, I bet he's your guy.

Chess streaks depend on skill - Chowdhary et al. (2023)

Compared to coin flips and baseball, chess may not feel random to you. You don't roll the dice on your turn, after all - you and your player get to see everything on the board and make your best decision. Where's the luck or the randomness in that? Well - hang on. Do you have a headache? Does your opponent? Is your side of the board exposed to a particularly annoying beam of light? What color did you draw, by the way? Is your opponent completely prepped for your favorite pet line?

I imagine you get the point, even if you still want to argue a bit! Yes, chess is indeed not a game of chance, but there are also random factors that affect how we play that vary from game to game. The difference between your rating and your player's rating is essentially a way to arrive at a probabilistic (NOT deterministic!) estimate of who's likely to win: You can go check out some ELO calculators online to see how different rating gaps predict different success rates for the higher-rated player. That makes analyzing time-series data of wins and losses (draws, what draws?) a means of evaluating chess as a stochastic system and comparing real outcomes to different models of how wins and losses might pile up. This is one part of the results Chowdhary et al. (2023) report in their paper, in which they offer a number of different analyses of how elite players differ from beginners in their approach the game and the results they achieve. There are a lot of intriguing results here that I'm planning to talk about in future posts, but for now I want to highlight their investigation of how beginners, intermediate players, advanced players, and experts exhibit "streaky" behavior online.

Given a time series of an individual player's wins and losses during their playing history, the simplest thing we could do is count up streak of wins and losses to see how often we find runs of 5, 6, 7 or more wins or losses in a row. This is where the authors start in this analysis, separating players by skill level according to Glicko-2 quartiles in their dataset of lichess games. Critically, however, they can't stop here! Remember the coin flips and Joe DiMaggio's hitting streak from our previous discussion - streakiness can play out differently just as a function of the bias in the data favoring one outcome over another. To the extent that beginners may not win as often as experts, they may exhibit different streak probabilities just because of their different win-loss proportions. To put it another way, even after we count up streaks of different lengths, do hot and cold streaks just reflect the inevitability of randomly packing that many wins and losses into a sequence? To find out, the authors compared the streak counts they observed for different lengths to what they counted up in shuffled versions of the players' time-series data. That is, what's the ratio of streaks of each length that we see in the real player data to what we see in mixed-up data that only preserves the proportion of wins and losses?

Chowdhary et al. (2023), Figure 1. At the top, an example of one player's timeseries data depicting wins in red and losses in blue. Below left, the ratio of hot and cold streaks of each length in the real data relative to shuffled series. Below right, winning streak ratios of each streak length separated by player ability.

What they found is that in general hot and cold streaks happened more often in the real player data than in the shuffled data, suggesting that such streaks are NOT just a by-product of players' win-loss records. Streakiness can't be explained by modeling players as random win-loss coins. If it were, these ratios should be very close to 1, but instead you can see that longer streaks become increasingly more frequent in the real data for players of all skill levels. Also, note in the lower left of the figure above that slumps occur more often than winning streaks! We shouldn't jump to a firm conclusion based on this, but it may suggest that losing streaks are possibly more clearly due to some deterministic factor than winning streaks.

All of this changes as you get better, however - beginners are MUCH streakier than stronger players, which you can see in the lower right figure. What's this about? The authors are able to rule out a few accounts of this difference: Long temporal gaps between "consecutive" games doesn't explain their data, for example, and the rating difference between opponents during a streak is only weakly correlated with streak length. They were also able to rule out consecutive games with the same opponent, so what's left? Maybe there really is something to being on a bit of a tear, or to being on tilt: Whether that's a motivational effect, a sign of environmental conditions that are affecting your play, or some other aspect of individual behavior or social interaction, hot and cold streaks seem to be more meaningful than random chance.

OK, but...

I feel the need to be a little careful here, however. The "hot hand" has been examined in a bunch of different sports and the results of these analyses have been picked over and argued about for long enough that there is every chance there's more to this story. If you're interested in such statistical arguments, check out the references below for some of my favorite papers about streakiness in different sports. I haven't had the chance to think through the analysis I described above relative to some of the bias issues raised in these papers, but the Chowdhary paper is a great example of open data making it possible for you to play around with these numbers as much as you like, If you find something neat, let me know!

Support Science of Chess posts!

Thanks as always for reading! If you're enjoying these Science of Chess posts and would like to send a small donation my way ($1-$5), you can visit my Ko-fi page here: https://ko-fi.com/bjbalas - Never expected, but always appreciated!

References

Chowdhary, S., Iacopini, I. & Battiston, F. Quantifying human performance in chess. Sci Rep 13, 2113 (2023). https://doi.org/10.1038/s41598-023-27735-9

Gilovich, Thomas; Tversky, A.; Vallone, R. (1985). "The Hot Hand in Basketball: On the Misperception of Random Sequences". Cognitive Psychology. 17 (3): 295–314. doi:10.1016/0010-0285(85)90010-6

Miller, Joshua B.; Sanjurjo, Adam (2016). "Surprised by the Gambler's and Hot Hand Fallacies? A Truth in the Law of Small Numbers". IGIER Working Paper (552). doi:10.2139/ssrn.2627354

Raab, Markus; Gula, B.; Gigerenzer, G. (2011). "The Hot hand Exists in Volleyball and Is Used for Allocation Decisions". Journal of Experimental Psychology: Applied. 18 (1): 81–94. doi:10.1037/a0025951

A few weeks ago, I finally found the time to sit down with the book "Blindfold Chess" by Eliot Hearst and John Knott, which I'd seen referred to several times as the best resource for learning more about the history of blindfold play and the techniques blindfold players use to manage multiple games in their mind's eye. It's a wonderful book with tons of little asides about noteworthy blindfold players and their approach to playing simultaneous exhibitions that could easily keep me busy in the lab for a while. For example, many players relied on imposing a sort of spatial organization on their games during a large exhibition by blocking out games opened with 1. d4 and 1.e4, sometimes with "anchor" games separating these with an opening like 1. f4 or c4. A few players throughout the history of blindfold play have also noted that having an empty physical board in front of them during a blindfold match was hugely disruptive! This is consistent with some standard interference paradigms for measuring visual imagery in the lab, and made me think of modern players who are known for looking up and away from the board while calculating.

Photo by the author.

Blindfold exhibitions give no rest for the weary

For this post, there is another item that came up several times throughout the book may seem less interesting from a scientific perspective, but stood out to me right away. It turns out that many players who have attempted large blindfold simuls over the years have reported an unfortunate side-effect of all this mental effort: Terrible, terrible insomnia. Harry Pilsbury, for example, noted that "his mind was so occupied with unplayed variations" that sleep was impossible for several hours. Miguel Najdorf claimed that after his famed 45-game blindfold exhibition it was days before he could really sleep, finally nodding off during a movie. While not all of the blindfold players in the book reported such aftereffects, it appears that persistent and intrusive chess imagery - seeing positions in your mind's eye and feeling stuck in loops of calculation - is fairly common. Some players found ways to mitigate these symptoms with other activities (card games, for Pilsbury) while others either toughed it out or used various sleep aids to overcome the problem.

On one hand, you might chalk this post-game sleeplessness as nothing more than a bit of leftover adrenaline from an intense experience. Keep in mind that we're talking about players who are juggling dozens of games for 8-12 hours in public, often in attempts to break world records, or at least to impress people with their prowess. Maybe that just means sleep is tough to come by because you've been pushing yourself to your limits for such a long time. I think it's a little more interesting than that, though. These observations about blindfold play and insomnia reminded me of a few different phenomena that are all related to the important role that sleep plays in learning and memory. Here, instead of walking you through a specific research study in depth, I’ll highlight these various effects in an effort to make some connections between blindfold play, sleep, and learning. Our first stop is another game that's known to lead to similar outcomes, but that has been studied by cognitive scientists and neuroscientists more thoroughly.

The Tetris Effect: Clearing lines in hypnagogic images

First of all, my guess is that you've very likely experienced visual imagery when you are just on the edge of sleep. Something like 3 out of 4 people report seeing things ranging from simple phosphenes of greenish or violet light to full-fledged visions of natural scenes with people moving around. These visual phenomena are referred to as hypnagogic images, and the first thing that came to mind when I read about blindfold players seeing the pieces and squares as they were trying to sleep was a well-known hypnagogic experience called the Tetris Effect. I'm usually more of a phosphene guy myself at day's end, but I have fallen victim to the Tetris Effect many, many times over the years as well.

I’ve mentioned in other posts that I wasn’t the most diligent chess player when I was a kid, but I more than made up for it in video games. I loved going to the arcade, my Sega Genesis still sees a fair amount of use, but the real gaming console of my childhood was my Game Boy. *A sad personal note: I managed to lose my Game Boy and all of my games during our move from Boston to Fargo, and I still have days where I consider going back to that apartment on the off-chance that it’s still sitting in the basement. Surely there’s a chance, right?

The original Game Boy came bundled with Tetris and I think in some ways the game and the console were made for each other. Tetris is addictive in any format, and the Game Boy version seemed especially so. I played for hours, eventually rewarded with the full-size shuttle launch you could only see by getting past level 25. If you played the game as much as I did, you may have seen the same things I did on the edge of sleep after long sessions: Visions of blocks falling through space, or sometimes just sort of floating there, crying out to be rotated and fit together into rows and rows of perfect rectangles.

Image Credit: Zyliskiai, CC BY-SA 4.0 <https://creativecommons.org/licenses/by-sa/4.0>, via Wikimedia Commons

Here's the thing: Playing Tetris requires that you make use of a broad range of cognitive processes that depend on coordination of multiple parts of the brain: Mental rotation (pre-motor cortex and parietal lobe), working memory (frontal lobe and sub-cortical structures like the hippocampus), and basic visual processing (occipital lobe) are all required to be good at the game, and continued play can strengthen connections between these areas. As you play more Tetris, the increased efficiency of communication between these different parts of the brain helps you recognize key spatial forms more accurately and more quickly and also helps you work out how shapes can be transformed by rotation to fill critical gaps in the playing field. That part probably seems intuitive, but where does sleep and hypnagogic imagery fit into all of this?

The replay of waking activity Tetris players experience just before sleep or during dreaming is more than just a quirk of focusing pre-adolescent attention on a tiny screen for hours. Instead, there is substantial evidence that “playing through” recent experience while you’re sleeping is a reflection of important processes that help ensure learned information sticks around in the mind and brain and strengthen your ability to carry out these tasks later. That is, forming those connections that make you a better Tetris player appears to depend in part on what's happening in your dreams. When I was an undergraduate, a particularly exciting result in this area came from electrophysiological recordings n rodents trained to navigate mazes, which is our next stop on this brief tour of hypnagogic imagery and learning.

Replaying maze navigation in hippocampal "place cells."

To tell you about a particularly cool connection between sleep, dreaming, and learning, we need to dive into the brain a bit. Specifically, we need to spend sometime in a subcortical structure called the hippocampus - so named because it supposedly looks like a sea-horse, at least if you look at it from the right angle and don't quite remember what a sea-horse looks like (maybe it's just me). Anyway, the hippocampus (pictured below) plays a critical role in memory across species. Bilateral loss of the hippocampus in human leads to the so-called amnesic syndrome, in which the ability to form new memories is disrupted, and the left and right halves of the hippocampus appear to support verbal and spatial memory respectively. This makes it a key target of many, many studies designed to understand the basis of learning and memory in the brain.

Henry Vandyke Carter, Public domain, via Wikimedia Commons

In rodents, the hippocampus helps to encode information about spatial navigation through neurons called “place cells.” These cells got this name because they are sensitive to an animal’s location, becoming active for example when a rat enters a specific part of a maze that is has learned to navigate to get some reward. These place cells offered an opportunity to examine how learning to traverse a maze during the day (an experience experiments could control) might have implications for activity in the hippocampus during memory consolidation processes researchers thought may happen during sleep. The remarkable result I learned about in class years ago was that recording activity from these place cells while a rat slept led to sequences of activity that were consistent with the rodent replaying their journey through the maze. As one place cell after another became active in sequence, researchers could see how they corresponded to the correct path from the beginning of the learned maze to the end.

Stuartlayton at English Wikipedia, CC BY-SA 3.0 <https://creativecommons.org/licenses/by-sa/3.0>, via Wikimedia Commons

By itself, this is really neat - rats are dreaming and we can see what they're dreaming of! There are subsequent studies replicating this result and demonstrating that place cell activity can even reveal a sort of wish-fulfillment in rodent dreaming: Rats that knew a food reward was in a part of the maze they couldn't get to will show place cell activity for the location of the food while asleep. If you can't get there while you're awake, at least you can get to the reward in your dreams! More interesting though is research showing that interrupting this replay of the path through the maze disrupts performance the next day. This suggests that working through waking experiences again and again either hypnagogically or during sleep isn't just about the imagery, it may help you consolidate what you learned so you'll be better at the task later.

Targeted Memory Reactivation - Nudging the sleeping brain towards learning

So rats dream of the mazes they learned to run during the day and that seems to be a crucial part of them retaining what they've learned. How sure are we that the persistent and intrusive visions our Tetris players (and maybe our blindfold masters) experience also have something to do with learning and memory, too? The last phenomenon I want to highlight is one I was fortunate enough to hear about a few weeks ago from Dr. Michael Scullin, a faculty member at Baylor who visited NDSU to give the keynote address at our annual Red River Psychology Conference. During his talk, Dr. Scullin told us about a bunch of exciting work his lab has done examining sleep and cognition, some of it using a technique called Targeted Memory Reactivation (or TRM). The results from his lab and other research groups that have used this technique help connect memory, dreaming, and learning in a way that I think helps make a compelling case that blindfold chess insomnia may be more of a feature than a bug.

The gist of Targeted Memory Reactivation goes something like this: If I really think that replaying waking experiences during sleep is an important way to consolidate memory and reinforce learning, it would be great if I had a way to promote dreams about tasks and other learned material that I wanted to retain. That is, I'd like to be able to sort of nudge the sleeping brain into dreaming of the content I really want my learner to hang on to. One way I could try to do that is by pairing the activity they do during their waking hours with some sort of cue (let's say a sound) that I know I can deliver to them while they're asleep. If the association between that cue and the activity is strong enough, playing the cue to them while they're asleep may provide the nudge I need to persuade their brain to replay their memories of the activity I want them to learn. That, in a nutshell, is how Targeted Memory Reactivation is supposed to work.

Image Credit: Backyard Brains. (https://docs.backyardbrains.com/retired/experiments/targetedmemoryreactivation/)

And it does seem to work, by the way! If you play music while participants do some task, or maybe fill the room with a particular fragrance, playing that sound or providing that smell during sleep does lead to replay of the activity. One simple way to measure this relies on one of the most annoying experimental paradigms I can think of: Measuring sleep stages with polysomnography to work out when a participant is dreaming just so you can wake them up and ask them what was going on. Though participants are probably justifiably irritated by this, their reports (though probably a bit bleary) do suggest that delivering the cue is enough to reactivate the memory researchers are targeting.

Now for the part that I think makes all of this come together. In Dr. Scullin's lab, his team carried out a version of this paradigm in which they played classical music to students who were asked to learn quantitative material about microeconomics. Next, those students either heard that music again during slow-wave sleep or were part of a control group that heard a different auditory stimulus. Rather than try to measure replay specifically, the question they were interested in was how students fared the next day when they were tested on that microeconomics material again. You can see the outcome below - hearing the music during slow-wave sleep led to better retention of the material from the lecture.

Figure 1 from Gao et al. (2020). Students in the TRM group who heard the classical music cue during slow-wave sleep outperformed students in the control group on a test of learned material from the previous day.

Hearing about these results during our conference immediately put me in mind of the sleepless blindfold players I had read about in Hearst and Knott's book. The difference is that there isn't any obvious cue driving the reactivation of chess memories (and this is probably true for those Tetris players we talked about too), but this may still be a brain on the edge of sleep working on reinforcing cognitive effort so that learning sticks. Perhaps it's the sheer amount of effort that leads to such persistent replay even in the absence of a clear cue - if you've been pushing this hard on a task, perhaps the brain simply defaults to memory reactivation. Alternatively, maybe visuospatial processing is special in some way - the sleeping brain may be biased towards reinforcing activity related to space, mental rotation, and navigation. Regardless, I hope I've connected enough dots here to give you a sense that the dreams of blindfold players may be an intriguing window on how the chess player's brain learns the game both on- and off-line.

A few scattered bits to wrap up

Like so many of the topics I write about here, I think there are a ton of interesting questions to ask about this phenomenon - finding out that blindfold play consistently leads to insomnia-inducing chess visions is a starting point rather than an end. For my part, the first thing I wondered about was whether regular simultaneous exhibitions led to similar outcomes. That's still an awful lot of chess, but without the immense demands on working memory blindfold chess requires. In terms of the neural basis of hypnagogic chess visions, there are some intriguing results regarding the nature of the Tetris effect in amnesia patients that I found compelling too: Patients with damage to their hippocampi still experience Tetris-Effect dreams without explicit memory of having played the game! This work (from Dr. Robert Stickgold's lab) suggests that Tetris replay doesn't happen in the hippocampus, but may instead depend on areas in the cerebral cortex. Is this the case for chess as well?

Finally, I feel like I'd be remiss if I didn't mention that this whole body of research (of which I've only skimmed the borderlands of here) concerning sleep, memory, and learning almost universally points towards one very concrete result: Sleep matters. I try to steer away from anything that sounds like an improvement hack of any kind here, but this one is so general and supported by so much work that I feel like it's worth saying. Sleep is critical for learning and memory, so if you (like me) would like to get better about chess by learning more about the game, managing your sleep health is important. So: Practice openings if you like, solve puzzles too, but get a good night's rest. Sweet dreams, friends - and may you dream of brilliant moves.

Chessable Research Awards - Submit your proposals by May 15th!

Some of you that read my Science of Chess posts may be scientists yourselves and may have your own interesting questions about the game that you'd like to research. The latest round of the Chessable Research Awards is now open for submissions and there is plenty of time to put together a proposal before their May 15th deadline. One of my students (the intrepid Alex Knopps) was awarded one of these grants last year and we've been excited to be able to pursue his ideas about collaboration in chess analysis with this support. If you'd like to chat about our experience with the CRA, feel free to drop me a line or check in with the Chessable Science team.

Support Science of Chess posts!

Thanks as always for reading! If you're enjoying these Science of Chess posts and would like to send a small donation my way ($1-$5), you can visit my Ko-fi page here: https://ko-fi.com/bjbalas - Never expected, but always appreciated!

References

Carbone, J., Diekelmann, S. An update on recent advances in targeted memory reactivation during sleep. npj Sci. Learn. 9, 31 (2024). https://doi.org/10.1038/s41539-024-00244-8

Cellini, N., & Capuozzo, A. (2018). Shaping memory consolidation via targeted memory reactivation during sleep. Annals of the New York Academy of Sciences, 10.1111/nyas.13855. Advance online publication. https://doi.org/10.1111/nyas.13855

Gao, C., Fillmore, P., & Scullin, M. K. (2020). Classical music, educational learning, and slow wave sleep: A targeted memory reactivation experiment. Neurobiology of learning and memory, 171, 107206. https://doi.org/10.1016/j.nlm.2020.107206

Haier, R. J., Karama, S., Leyba, L., & Jung, R. E. (2009). MRI assessment of cortical thickness and functional activity changes in adolescent girls following three months of practice on a visual-spatial task. BMC research notes, 2, 174. https://doi.org/10.1186/1756-0500-2-174

Hu, X., Cheng, L. Y., Chiu, M. H., & Paller, K. A. (2020). Promoting memory consolidation during sleep: A meta-analysis of targeted memory reactivation. Psychological Bulletin, 146(3), 218–244. https://doi.org/10.1037/bul0000223

Louie, K. & Wilson, M. A. Temporally structured replay of awake hippocampal ensemble activity during rapid eye movement sleep. Neuron 29, 145–156 (2001)

Ólafsdóttir, H. F., Bush, D., & Barry, C. (2018). The Role of Hippocampal Replay in Memory and Planning. Current biology : CB, 28(1), R37–R50. https://doi.org/10.1016/j.cub.2017.10.073

Stickgold, R., Malia, A., Maguire, D., Roddenberry, D., & O'Connor, M. (2000). Replaying the game: hypnagogic images in normals and amnesics. Science (New York, N.Y.), 290(5490), 350–353. https://doi.org/10.1126/science.290.5490.350

Reading through and playing these games was a bit of a challenge for me when I was a school-age patzer. The games in this edition are written out in descriptive notation (1. P-K4, for example) rather than the algebraic notation (1. e4) I was learning to use. This made reproducing the game in my head or on a board a little more like a code-breaking exercise for me, but I liked the challenge and the mildly archaic look to the text. Besides the slightly odd notation, the moves are also richly annotated with marks signaling blunders, questionable moves, intriguing moves, and brilliant moves. This notation also made the text come alive and posed new challenges for me as a young player: What made the bad moves so bad? I often looked at them and thought they looked perfectly natural until I kept going to see what Fisher had in store for his opponent. Maybe more puzzling to me in some cases were the good moves: What made some of them so noteworthy? Again, I could always keep playing and read the annotations to find out, but often the real brilliance of the moves remained obscure to me.

The picture above illustrates one of Fisher's most legendary brilliant moves. His sacrifice with15 ... NxP! (or if you prefer, 15 ... Nxf2!) sets off a beautiful combination that led to this game against Robert Byrne from the 1963/64 US Championship winning a Brilliancy Prize. As a student in chess classes at the Murrysville Chess Club , we analyzed this game one week (guided by the very patient Jay Griffin), so I ended up with a decent understanding of why this move was so impressive. On balance, however, Fisher's brilliance (and his opponents' in a few cases) was something I often just had to take his word for.

Brilliant moves online

When I first started playing chess online a few years ago, the prospect of reviewing my games afterwards with an engine was very exciting. From a learning and improvement perspective, seeing the moves that were labeled as blunders and inaccuracies is probably more useful, but I couldn't (and still can't!) help being excited to see the occasional moves that were labeled as being exceptionally good. If you play at chess.com, their game review algorithm assigns labels to your moves that include "Great" and "Brilliant," so you can imagine your own games marked up like the ones I read about in Fisher's book. For ordinary players like me, these brilliant moves ought to be fairly rare, and in my case they certainly are! Here's my breakdown of various move types from the CC "Insights" panel of my Blitz performance.

As much as I like seeing those rare !! symbols tacked onto my own game, I don't take them too seriously. After all, my overall level of play hardly warrants adjectives like Great or Brilliant in general, so these annotations probably ought to be considered in the context of my skill and my opponents' typical skill. But are they? How exactly does the engine decide which moves are Great and which are Brilliant? Here is an example of a position from one of my games in which I managed to make not just one, but two "Brilliant" moves back to back. See if you can spot the first one (Black to move).

I doubt that this would have impressed anyone if it happened in a game between titled players, but I was glad I saw Nf3! My opponent followed up with h3, making my Bxh3! response another "Brilliant" move. Both of these moves have the characteristics that it turns out chess.com uses to assign the Brilliant label: The algorithm pretty much requires that you sacrifice a piece, and that this sacrifice be "good" in the sense that it does not weaken your position. There are a few more bells and whistles per their support page, namely that it also needs to be the case that you wouldn't still be winning convincingly without making this move. Another way to say that is that a Brilliant move has to be a sort of narrow path to a winning position: You can't just sacrifice stuff when you're overwhelmingly ahead and get credit for it.

The project of figuring out how to reasonably assign brilliancy to players' moves raises an interesting cognitive question: What makes moves seem brilliant to human players? When Bobby Fisher decides that a move he or one of his opponent's made deserves to be considered remarkable, I know that his decision was based on a deep understanding of the game. Part of that understanding is a subjective sense of which moves are both very good (which is easy to define with a modern engine) and difficult to see (this one is much harder to define). We don't call an obvious winning move Brilliant, after all - part of what makes these moves seem worth noting is that they are easy to miss (for a human). The rough definition I gave you in the previous paragraph is one attempt to make those two qualities concrete: Maintaining a winning position makes a move good and the algorithm assumes that piece sacrifices are generally tough to see. Who thinks of throwing a piece straightaway, after all?

GM Mikhail Tal, master of the sacrifice. Image by Croes, Rob C. for Anefo, CC0, via Wikimedia Commons

OK, maybe some players think of that first. All joking aside, this is the tricky and fascinating part of trying to automate annotations like this. You essentially need a model of what players will and won't think of easily and such models are not easy to develop! The target article I'll tell you about next is an attempt to tackle this challenge head-on by comparing human annotations of brilliancy to those assigned by different computational models. How can we best capture the subjective impressions of surprise, strength and creativity that humans use to call a move Brilliant with an objective computational criterion?

Modeling perceptions of move brilliance

In this study, the authors needed to do two things: (1) Collect a large-ish number of human ratings of move brilliance (and other annotations like Great, Bad, etc. to serve as a comparison class) from real games. (2) Compare models designed to guess those human labels using stuff we can calculate from the position itself.

Human judgments of move aesthetics

I'm going to refer to the assignment of labels to moves as being judgments of move aesthetics from now on. This is a useful catch-all for these kinds of evaluations that a few different researchers have used (see Osborne 1964 and Humble, 1993) and that highlights the idea that we're really talking about judgments of something like beauty, creativity, and surprise. At any rate, to get a decent sample of judgments of move aesthetics, the authors collected about 7000 moves from studies available on Lichess that were labeled as Brilliant by human annotators (820 moves), Good (1637 moves) or Other (4518). From now on, this is the problem they're going to try and solve - can they work out which of these categories a move belongs to? Perhaps more importantly, can they provide insight into what's different about the way a model evaluates the moves in these different groups?

An example of a Lichess study with clumsy annotation by the author added for illustration purposes only!

Building a model of aesthetics from chess engines

The next part of the authors' work is much harder: How do you decide to calculate something from the position itself that might have something to do with human perceptions of aesthetics? The goal here wasn't just to try and get the answer right (or as right as they could) but also to try and understand what measurements or features supported good computational guessing about human judgments. To try and arrive at these insights, the authors used two different chess engines, Leela and Maia, as the basis for separate models. There is a lot one could say about these two models, but the key distinction the authors talk about in their work is the difference in how the two engines learned to play: Leela learns through self-play (over 1 billion games) while Maia learns from games humans of different skill levels have played with each other.

Using both of these models, the authors could calculate a number of computational features from a data structure called the game tree. If you're not familiar with this term, it refers to the branching structure of possible move sequences away from a starting position. That starting position is usually called the root of the game tree and we can draw branches that extend away from it for each possible next move in the game. Each extension can then serve as the next node from which more possible moves can branch away, eventually yielding a hierarchy of everything that could happen from the position we started with. For a simple game like Tic-Tac-Toe, the game tree is not terribly large. For chess, the game tree is enormous - so large that engines generally don't try to examine the whole thing, but adopt some way to sample promising branches and select the best option from there.

A portion of the Tic-Tac-Toe game tree - not too intimidating, right? By Traced by User:Stannered, original by en:User:Gdr - Traced from en:Image:Tic-tac-toe-game-tree.png, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=1877696

Using the trees generated by Leela and Maia, the authors were able to obtain a collection of measurements describing how the starting position (before the critical move was made) was being evaluated. I don't want to take you all the way into the weeds here because it does get rather technical, but these included things like whether the critical move was even included in the 10 trees each engine opted to examine, and how making the critical move was evaluated: How often did the engine decide that making the critical move improved the position vs. weakening it? How often did the engine think the move led to an actual advantage for the player? What was the winning percentage for the player that the engine estimated in different parts of the game tree? Together, these and a number of other measurements intended to describe the actual shape (width and height) of sub-trees inside the larger game tree were the vocabulary the authors used in each engine to make good guesses about human move aesthetics. They extracted these features from both models and varied the number of nodes in the tree that the engines could inspect from 10 all the way to 100,000 by factors of ten.

Brilliant moves might be good moves that look bad

I'll start with the answer to the main question: Can the authors accurately classify moves according to the aesthetic categories humans assigned them to? Briefly, yes! In fact, the authors found that they were able to do this at above-chance levels (70% or so) using just the engine estimation of win probability before and after the move, the difference in win probability before and after. Including more of their features improved their classification further, demonstrating that describing the nature of the game tree more thoroughly (including some of those descriptors width and depth of sub-trees) made it easier to guess which moves seemed brilliant and which ones were not.

Besides looking at the accuracy of guessing human move aesthetics, though, the authors also included some interesting analyses of how the two models under consideration evaluated moves in the Brilliant, Good, and Other categories when fewer nodes were included (10 is the low bound, remember) and when more are included (10^5 is the upper bound). First, let me show you how the Winning Chance of these three kinds of moves changes on average when Leela gets to use fewer nodes or many more.

Adapted from Figure 2 of Zaidi and Guerzhoy, 2024 - Win Chance increases for each move type as the number of nodes increases.

I know the axes might be a little tough to read here, so let me tell you the key things to notice here. As you move from left-to-right across each panel, the number of nodes that Leela gets to inspect increases by powers of 10. These squeeze-y shapes are a Violin plot of the data, which is intended to give you a sense of how the data is distributed across y-values at each stop on the x-axis. A skinnier part of the shape means that there are fewer data points there, while a wider part means lots of data is concentrated there. I hope you can see that it looks like a meaningful difference between Brilliant moves (leftmost panel) and the Other moves (rightmost panel) is that the red line connecting the mean values of these shapes has a steeper slope on the left. What does this mean? It suggests that the thing about Brilliant moves is that they start to look better when the engine gets to look at much more of the game tree. A shallow look isn't so exciting, but a deeper look makes a Brilliant move start to shine.

Here's what the same analysis reveals about Maia.

Adapted from Figure 2 of Zaidi and Guerzhoy, 2024 - Win Chance increases for each move type as the number of nodes increases.

The Maia data might look much the same to you at a casual glance, but there's something even more compelling here. Not only is that red line steeper for the Brilliant moves, but check out just how dramatically different the violin shapes are in that leftmost panel. Compared to the data from Leela, the neat thing about this is that the Brilliant moves are not only evaluated as weaker when Maia only looks at 10 nodes, they actually appear to be losing moves on average. Good moves have a weaker version of this problem, but not so dramatically as the Brilliant ones: The thing about brilliancy as far as Maia is concerned is that a Brilliant move looks like a terrible one at first, but reveals itself as a winner when the engine has the resources to evaluate it more thoroughly. Even the step up from 10 nodes to 100 shifts the distribution of values for Brilliant moves from the absolute floor to at least equal chances. While the authors offer evidence that understanding more about the game tree helps us model move aesthetics more completely, this is a neat way to think about a simple computational definition of what it means for a move to look Brilliant.

Conclusions and next steps

I'm still really interested in understanding how we can try to understand subjective human judgments about the game using computational models, and I think this was a great addition to this literature. I can't help but think back to my book of Fisher's memorable games and wonder how these engines would do at guessing his annotations. The dataset considered here has the benefit of being compiled by a wide range of players, which means more variability about move aesthetics was built into the sample. I think that's a good thing, but it could also be a neat historical project to consider how aesthetic judgments about moves made during different eras might be more or less predictable. The game has changed many times over the centuries, after all, and what looks brilliant and unexpected to one generation may be part of what a new generation considers straightforward. For now, this study is an intriguing look at how we can make humans' subjective impressions of the game more concrete.

Now, please bear with me for a short announcement that I think some of you may be interested in...

Chessable Research Awards - Submit your proposals by May 15th!

Some of you that read my Science of Chess posts may be scientists yourselves and may have your own interesting questions about the game that you'd like to research. The latest round of the Chessable Research Awards is now open for submissions and there is plenty of time to put together a proposal before their May 15th deadline. One of my students (the intrepid Alex Knopps) was awarded one of these grants last year and we've been excited to be able to pursue his ideas about collaboration in chess analysis with this support. If you'd like to chat about our experience with the CRA, feel free to drop me a line or check in with the Chessable Science team.

Support Science of Chess posts!

Thanks as always for reading! If you're enjoying these Science of Chess posts and would like to send a small donation my way ($1-$5), you can visit my Ko-fi page here: https://ko-fi.com/bjbalas - Never expected, but always appreciated!

References

Humble, P.N. (1993) Chess as an art form. British Journal of Aesthetics, 33, 59-66.

McIlroy-Young, R., Sen, S., Kleinberg, J. & Anderson, A. (2020) Aligning superhuman AI with human behavior: Chess as a model system. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1677-1687.

Osborne, H. (1964) Notes on the aesthetics of chess and the concept of intellectual beauty. British Journal of Aesthetics, 4, 160-164.

Zaidi, K. & Guerzhoy, M. (2024) Predicting User Perception of Move Brilliance in Chess. Arxiv, https://arxiv.org/abs/2406.11895

Many Japanese logic puzzles (like my favorite Herugolf puzzles) have a beautiful structure that's revealed to the solver as they work through the constraints that lead to the correct answer.

Among all the different kinds of puzzles I do, chess puzzles occupy a unique niche. The mathematician G.H. Hardy referred to them as "the hymn tunes of mathematics," dismissing them as unimportant in the grander scheme of understanding number, shape and space. Though they may be among the least important examples of mathematical reasoning and theorem-proving, they are of obvious importance to people who play the game and want to improve. We don't solve chess puzzles to arrive at some deeper truth about the universe! Instead, most of us solve chess puzzles because they're fun and also (perhaps) in the hope they'll help us get better at playing full games. Puzzle Rush, The Woodpecker Method - all of these various types of puzzle drill are intended to support the acquisition and/or refinement of patterns, or "chunks," of chess knowledge that will help us see brilliant moves over the board. It's this presumed pedagogical value that I think makes chess a bit different than most of my morning puzzle routine and helps motivate the question we're going to consider here: How do tell how hard a puzzle is?

How do you compose a good chess puzzle?

The thing about a good puzzle of any kind is titrating its difficulty. It's very easy to write a puzzle that is so difficult that it is impenetrable, and it is also straightforward to write one that is trivial. The tough thing about puzzle composition of all types is to give the solver something that isn't obvious at first, but will give way after careful and clever enough thought. Writing the kinds of puzzles that make up a good Mystery Hunt , for example, is all about finding this sweet spot, and locating it depends on understanding the problem domain of the puzzle AND having a good sense of what a solver will do when confronted with it.

In cognitive science, the term of art I'd use here is "Theory of Mind," which refers to the ability to estimate what someone else might be thinking or feeling based on their experience of the world. Below, I've included an image of the classic "Sally-Anne" problem used for years as a means of measuring individuals' Theory of Mind capabilities. In this case, answering the question correctly depends on correctly understanding what each person does and doesn't know about the events depicted below. That same kind of understanding is necessary for writing good puzzles - what does my solver know and what will they try to do? What steps might be immediately apparent to them and what ideas will they have a hard time arriving at? It can be tough to take another person's perspective, but composing a puzzle that will be satisfying to solve depends on it.

By Simon Baron-Cohen, Alan M. Leslie, and Uta Frith - Does the autistic child have a “theory of mind”?, Cognition, Fair use, https://en.wikipedia.org/w/index.php?curid=53202493

The way that understanding puzzle difficulty relies on your own understanding of the kind of problem you're presenting AND your understanding of other people's minds makes this a fascinating topic for cognition research. Chess puzzles are also especially convenient for thinking about these issues of puzzle design and difficulty because we have a great deal of data about how hard individual chess puzzles are. Unlike most of the puzzles that people solve recreationally, chess puzzles don't just vary in difficulty in colloquial terms like "easy," "intermediate," and "hard" (or if you're an NYT Crossword buff, Monday vs. Friday). Instead, chess puzzles can have an ELO rating just like people do: Each attempt to solve a puzzle is a "match" between the puzzle and the solver. If the solver works it out correctly, they win and their puzzle-solving ELO increases! If they get it wrong, the puzzle is the victor and it's rating increases. Just like regular tournament play, applying ELO this way should yield ratings that support good predictions about what solvers can succeed against which puzzles. Here's a look at my own ELO puzzle trajectory during my time over at chess.com.

As I've mentioned in other posts, these statistical tools used to characterize chess offer researchers a valuable resource for asking specific questions about the game, and by extension about the cognitive processes that support playing it and thinking about it. In this case, chess puzzles provide a unique opportunity to ask some particularly neat questions about puzzles, domain expertise, and Theory of Mind: How easy is it to determine how difficult a puzzle is? Tand How does puzzle difficulty estimation relate to playing strength? The quantitative data we have about chess puzzles' difficulty makes these probems the perfect sandbox for examining how people assess task difficulty based on their own expertise. Once again, chess turns out to be a very useful proxy for expertise considered more broadly, making it possible to address a specific question about cognition precisely.

Trying to predict puzzle difficulty in the lab

The target article I want to tell you about is really quite straightforward in a number of ways. At its core, there is a simple question that the authors try to answer: Do higher-rated players do a better job predicting how hard puzzles are? How good is anyone at guessing how hard a puzzle will be? The design and analysis of the study is similarly fairly easy to get your head around, which makes it an easy experiment to understand. I thought the results were surprising, however, and maybe hint at some compelling ideas for future work.

Study Design

First, the basics: The authors of this study recruited a group of just 12 chess experts whose FIDE ratings ranged between about 1850 and 2300. These participants were then presented with 12 chess puzzles from Chess Tempo grouped into Easy, Medium, and Hard categories based on their rating. Below, you can see some solving statistics from the manuscript: The higher the puzzle rating, the more time it tends to take people to solve it and the lower the solve rate. Those stats are important to see because they mean there is a clear difference in puzzle difficulty between these three categories that the experimenters constructed for their task.

Table 1 from Hristova et al., 2014. Puzzle statistics for the 12 problems selected from Chess Tempo for the current study.

Participants were asked to solve these puzzles in a randomized order (to avoid sequence effects) and were given three minutes to come up with the best move for each position. The entire testing session took about 30 minutes, and I should note that the researchers were also collecting eye-tracking data throughout the task. To me, the eye movement data didn't add a great deal to the study, so I'm going to focus on the simpler behavioral outcomes instead. If you're interested in some studies that used eyetracking in service of some neat questions about chess and visual perception, however, I'll refer you to some of my other articles on that topic.

Measuring difficulty estimation with puzzle rankings

We're not just interested in how well these players solve puzzles, however, but how able they are to estimate how challenging a puzzle is. This means that the last important detail we need about this study is how they asked participants to rate puzzle difficulty. Their approach is about as simple as you can get: They just asked each participant to rank the puzzles in order from easiest to hardest. Then they used the formula below to measure the agreement between the real difficulty ranking and the ranking provided by each player.

I'm showing you this formula for Kendall's Tau because I want to highlight something neat about their analysis of participants' rankings that makes the results especially interesting to me. In this formula, you're basically counting up pairs of puzzles in the rankings to arrive at the value you need to measure accuracy. In particular, you're counting how many discordant pairs you find in the data and how many concordant pairs you find. A discordant pair refers to a pair of puzzles that's in the wrong order in the participant's rankings: If the puzzle rated 1492 is ranked as more difficult than the one rated 2230, for example, that is one discordant pair. On the other hand, a concordant pair refers to a pair that's in the correct order.

Here's the really important thing: If you take a look at those puzzle stats up above, you'll notice that within the Easy, Medium, and Difficult categories there are puzzles with very similar ratings. If it seems a little much to expect that the participants might be able to tell an 1878.1 puzzle apart from an 1878.6 puzzle, you're not alone in your concern! The researchers thought this was too much to expect as well, so they only counted discordant and concordant pairs from different difficulty categories! That is, the only thing they are really looking at is whether participants said Difficult puzzles were harder than Medium and Easy ones, and whether they thought Easy puzzles were simpler than Medium ones. That's it! A high value of tau means you got that order right more often than not, and a low value means you didn't. So what happened?

It's really tough to tell how hard a puzzle is, actually

I've got a couple neat things to show you, but the bottom line is this: This group of experts wasn't great at ranking the puzzles by difficulty, even though the researchers were only worried about Easy, Medium, and Difficult differences. Let me start by showing you the raw data, which in this case is small-scale enough for you to look at yourself. The table below shows you the rankings for each of the participants in the study. In each column, the right answer should be 1, 2, 3...all the way to 12, but the scrambled orders you see in each case demonstrate how hard it was for players to come up with that order.

Table 3 from Hristova et al., 2014. Raw puzzle difficulty rankings provided by all 12 participants.

I'm sure you can find interesting and surprising errors of your own here, but check out how often some of the most difficult puzzles float to the top while much easier ones end up at the bottom! Clearly there is a lot of misperception going on in terms of how hard these puzzles are at the population level and how hard they appear to be to these players. In the bottom row of the this table, you can also see the value of Kendall's Tau for each player, which is our quantitative tool for talking about how close players got to the right category-based ordering of the puzzles. I think its easier to look at this in the graph below, however, to ask another question: Do stronger players do a better job of estimating puzzle difficulty?

Figure 5 from Hrisvota et al., 2014. A scatterplot of Kendall's Tau (ordinal ranking accuracy) and FIDE rating.

I hope you can see from this graph that the answer to that last question is a big ol' nope. In general these players aren't doing a great job at the task according to Kendall's Tau, and there isn't a significant relationship between these two sets of values to support the claim that you do better at difficulty estimation if you're a higher-rated player. The authors have a few more analyses in the full paper to look at this a few different ways, but the punchline stays the same. Even when you ask how performance with these puzzles relates to difficulty estimation, there just isn't a meaningful relationship there. It's just hard to guess how difficult a chess puzzle is.

So what does this mean (and what comes next?)

Though it might seem disappointing that these players weren't great at this task, I think these results are rather exciting and lead to some cool ideas for related work. To motivate what I'm about to say, I'd like to show you one more figure from the paper - this time a table that shows off how the players did at the 12 Chess Tempo puzzles they were presented with during the study.

Table 2 from Hristova et al. Solving statistics for the puzzles presented to participants in the current study.

What you're seeing here is the percentage of people that got each puzzle correct ("Success" column) and the average time it took them to do so in the last column. The thing I want to call your attention to are some funny discrepancies between these values and the Chess Tempo stats for these puzzles I discussed previously. Puzzles 4 and 5 are supposed to be Medium puzzles, but fewer than half of these expert players got them right! On the other hand, Puzzle 10 is pretty challenging, but this group fared pretty well (and quickly to boot!). I really don't want to delve into the details of these puzzles in an attempt to get very granular about the design and the results, but I think these features of the data hint at an important possibility: Chess Puzzle performance might be very idiosyncratic.

You probably have a sense of what kinds of puzzles you're better at, and if you don't, you can get that kind of information pretty easy with the analytics tools on lichess or chess.com. Here's a look at my Puzzle Dashboard (which is pretty sparse at the moment - gotta do more drill!): I'm apparently atrocious at finding good defensive moves, but not so bad at exploiting pins. Your mileage may vary, however, and that is very much my point.

NDPatzer's Puzzle Dashboard stats. I'll probably find the right sacrifice a fair amount of the time, but don't count on me for good defense.

Puzzle difficulty may not be especially meaningful as a population-level construct. Instead, it may vary so much between players that the more relevant thing might be how difficult a puzzle is for a player like you. You may have taken one of those "Chess Personality" quizzes at some point (I really wanted to be Capablanca, but that is not at all in the cards it seems), but I suspect there is a lot of good work to be done to try and establish exactly what that might mean quantitatively, especially for beginner/intermediate players with variable skills. What kinds of clumps are there in dashboard data like I've displayed above? Is perceived difficulty intimately tied to the kinds of puzzles you struggle with vs. the ones you fly through with ease?

I think these are exciting questions and ones that can probably be answered with the data available through sites like this one. So while the current study doesn't give us an easy answer to the question of how we estimate the difficulty of a chess puzzle, it highlights the need to think about how population-level data and more granular analytics might be necessary to fully understand how different players see different positions.

Support Science of Chess posts!

Thanks as always for reading! If you're enjoying these Science of Chess posts and would like to send a small donation my way ($1-$5), you can visit my Ko-fi page here: https://ko-fi.com/bjbalas - Never expected, but always appreciated!

References

Baron-Cohen, Simon; Leslie, Alan M.; Frith, Uta (October 1985). "Does the autistic child have a "theory of mind"?". Cognition. 21 (1): 37–46.

Chuzo Iwamoto, Masato Haruishi, and Tatsuaki Ibusuki. Herugolf and Makaro are NP-complete. In 9th International Conference on Fun with Algorithms (FUN 2018). Leibniz International Proceedings in Informatics (LIPIcs), Volume 100, pp. 24:1-24:11, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2018) https://doi.org/10.4230/LIPIcs.FUN.2018.24

Hardy, G. H. (2012) [1st pub. 1940, with foreword 1967]. A Mathematician's Apology. With a foreword by C. P. Snow. Cambridge: Cambridge University Press. ISBN 9781107295599.

Hristova, D., Guid, M. & Bratko, I. (2014) Assessing the difficulty of chess tactical positions. International Journal on Advances in Intelligent Systems, 7(3), 728-738.

Introduction to a possibly bad idea

As a university professor, I've found myself on the receiving end of a lot of messaging regarding how I could (or maybe should) find ways to incorporate AI into my teaching. To be blunt, I can think of few things I would like to do less. Without getting up on my soapbox for too long, I put a lot of value on hands-on lab work, open discussion, and working on your own thinking and writing without LLMs by your side. Your mileage may vary, but yeah - I'm a bit of an AI skeptic.

NDPatzer very much doesn't want his classroom to look like this. Illustration Andrzej Olas/Svensk biblioteksförening, CC BY-SA 4.0 <https://creativecommons.org/licenses/by-sa/4.0>, via Wikimedia Commons

Or am I?

Not entirely, of course, because I play chess and routinely use chess engines to support my analysis, my opening training, and to enrich my experience while watching events like the recent World Rapid and Blitz Championships. As I've watched my colleagues wrestle with how to deal with the accessibility of various forms of AI, I've frequently been struck by how chess engines have become an integral part of the chess community. The kinds of existential issues academics are having right now played out 30 years or so ago for chess players, with a lot of time since then to figure out how to use engines rather than fret about their presence.

Chess players have been worried about AI since this guy made his debut. The Mechanical Turk was a false alarm, but a portent of things to come! Public Domain Image

How can we work (and play) meaningfully with AI?

One thing the academy shares with the chess community with regard to AI, however, is the ever-present worry about cheating. What if someone uses a chess engine (or LLMs) when they aren't supposed to? How do we deal with the reality that incentivizing good grades or tournament wins enough inevitably means someone will try to find a way to guarantee success by using these tools unfairly?

I've been prepping my Spring course and thinking through the academic version of these concerns as I go: How do you write an assignment that de-emphasizes AI use as much as possible? I take a few different approaches to this, one of which is the old "If you can't beat 'em, join 'em" strategy. If I think students are going to use AI, either let them do so or require them to do so. Writing my assessments with that in mind means I have to work out how to make an assignment interesting and useful by accepting that it will be completed by a combination of human and machine effort. Where can we go in the classroom if that's the context for coursework?

Should we just welcome our new robot overlords? Bovee and Thill, CC BY 2.0 <https://creativecommons.org/licenses/by/2.0>, via Wikimedia Commons

Advanced Chess as an approach to human-machine play

Like I said above, chess has been ahead of the academy for a long time with regard to these issues and this idea is no exception. As computers like Deep Blue began to demonstrate that they could outpace the strongest human players, those human competitors started to think about whether and how chess engines could enrich human play rather than stand in opposition to it. Specifically, Kasparov proposed a version of the game he called "Advanced Chess" (sometimes referred to as "Centaur Chess") in which each human player could consult directly with the machine of their choice throughout the game. If you follow that wiki link, you'll read that this idea had been floating around the sci-fi literature for some time before Kasparov's proposal in the 90's, but the core idea is intriguing regardless: What does chess look like if machines are allowed to participate? Advanced Chess tournaments have been held by a number of different organizations ever since and continue to explore the nature of the game when a human-machine hybrid takes the place of human competitors.

Eros Riccio, Advanced Chess Champion - By Cinmad - Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=27934082

There are a wide range of opinions about how exciting Advanced Chess tournaments are to watch (and to play in). On one hand, it makes it possible for almost anyone to have a fighting chance against anyone else, so long as they're armed with a fairly strong engine. On the other hand, pitting computer line against computer line can also seem dissatisfying to some compared to the drama of human minds searching for the strongest moves while time ticks away. In my reading about Advanced Chess, my overall impression of it is that I wouldn't really enjoy playing this way. I like coming up with my own moves, after all! Sure, it's occasionally crushing to miss something obvious and lose, but that's offset by the joy of finding something creative, or daring, or just plain accurate and winning. Using an engine against an opponent who is also using an engine would take that experience away from me, I think, so I'm not excited to tie my chess playing to an engine so directly.

A (possibly) new idea: Advanced Chess on a budget?

But here's where I want to make a proposal for a different kind of human-machine OTB (or online) chess that I think could be a lot of fun. I'll preface this by saying that its entirely possible that this is something people have thought of and/or implemented before, but I wasn't able to find examples anywhere. Please let me know if I've overlooked something, though! That said, here is the idea.

Step 1: Increase the points available for wins/draws

Typically, a game of chess in a tournament is a battle for a single point that is up for grabs between the two players. A win gets you the point, while a draw means the players share a half-point each. The first modification I want to make to tournament play is to change that single point to 10 points, for reasons that will become clear in a moment. A win could mean getting all 10 of those points, while a draw could mean splitting that pot up into 5 points each.

Step 2: Allow players to spend points on engine consultation

Here's where the second part of the idea comes into play, however. During the game, either player may "spend" their share of the 10-point pot on consultation with an engine. Let's say asking for the strongest move in a given position costs you two points - this means if you ask for help once and win the game, you only get 8 points instead of 10. Ask twice and win, now you only get 6 points - pretty close to what you'd get from a humans-only draw! Besides asking for the best move, you could also potentially opt to get weaker information during your consultation at a lower cost: Maybe asking for the eval bar value (Is it +3.8, -1.2 or 0.0?) only costs a single point and asking who has the advantage (but not by how much!) only costs a half-point. The idea is to permit limited use of an engine at the player's discretion, but at a cost! Ultimately, the goal is to create a variant in which players' decision-making would incorporate analysis of the position and the cost-benefit of whether to use the engine and how to do so.

My thinking is that this format would allow players to make use of engines in crucial situations, but disincentivize extensive use due to the dilution of your tournament points. A player who can win consistently without the engine will rack up more valuable wins than a player who needs to get bailed out more often! Ideally, this folds AI into the game without completely compromising the human element.

There are a lot of things to figure out about how to make this work in practice, but I'd love to give it a try sometime. What do you think about this kind of Centaur Chess? I'll be honest, this is absolutely a half-baked idea on my part and there are a bunch of parameters to work out. The specific costs of different kinds of engine consultation probably need to be titrated carefully and there are likely other neat ways to think about using information from an engine during play. If you've read this far, let me know what you think, suggest your modifications of the framework I'm proposing, and if by some chance you try it out, let me know how it goes!

Screen shot of Sargon III gameplay on the Apple IIc.

It wasn't long after I quit competitive play that computer chess took a huge leap forward. In 1996, Deep Blue managed to take a game from then World Champion Garry Kasparov, and the next year it would outright beat him under tournament conditions. Though I had mostly left chess behind, the news that a computer could beat the best player in the world was equal parts exciting and disappointing. Part of me couldn't help but want Kasparov to win, but part of me was intrigued and inspired by the idea of building an algorithm that could play chess at the highest levels.

Kasparov facing off against Deep Blue - Chessbase.com

Fast-forward to 2022 and I decided to give chess a try again. I was prepared to have to re-learn a ton of what I had forgotten during years of not playing, but what I wasn't prepared for was just how thoroughly incredibly strong chess engines had become incorporated into the game. The idea that I could analyze my games with the help of an engine more powerful than Deep Blue was startling. The more I started to consume online chess content, the more I was amazed by the way commentary worked now: Streamers and commentators didn't have to speculate about a position the way I remembered, but could now look at an eval bar or an engine line in real time. Gone were the days of wondering if a certain line was best - now the question was whether or not a player would make the move we all knew was strongest with the benefit of the engine. Could a GM see what the machine saw?

What are "computer moves?"

The more chess content I watched, the more I kept hearing a phrase that fascinated me. That phrase was computer move. I'd watch a recap video, for example, and the commentator might point out the best line by saying something like, "But that's such a computer move - there's no way a human is going to play that." I also saw discussions like this in videos exposing chess cheaters: In many of these videos, a player with a rating not far from mine would uncork a move that seemed impossibly strange, but ended up putting enormous pressure on the opponent. A "computer move" strikes again!

Really? h4 is best?

From a cognitive science perspective, I thought this was fascinating. What this idea of "computer moves" suggested to me was that chess engines aren't just better, but that they also play chess differently than people. What could that mean? Why would some computer moves seem "hard to play" or "impossible to see" when they're right there on the board? One possibility is that while people bring a lot of cognitive biases to their decision-making (including confirmation bias, which I've written about elsewhere), computers have no such constraints on their thinking.

A classic example of Confirmation Bias: If I tell you that every even-numbered card is red on the other side, which cards do you turn over to check if that's right? Most people will say "8" and "Red," but it's really "8" and "Blue" that you need to check!

This idea of "computer moves" raises an interesting question, however: Can chess players really tell the difference between a machine opponent and a human opponent? Those examples I gave you above are compelling, but also aren't quite enough to provide a firm answer. To really find out if we can tell a person from a bot, we need to know if chess engines can pass the Turing Test.

What is the Turing Test?

Alan Turing (pictured below), is widely considered one of the founders of computer science and made foundational theoretical contributions to the field (like his work on computability) and practical applications of computer algorithms (like his war-time work on cryptanalysis). His work was wide-ranging, including topics like pattern formation in biological and chemical systems alongside his more mathematical work. Perhaps his most enduring idea is the concept of the Turing Test, which he suggested as a simple criterion for deciding whether or not a machine could be considered "intelligent."

Public domain, via Wikimedia Commons

The key idea behind the Turing Test is that if a person cannot distinguish between a person and a machine in some setting, than we may as well say that the machine is intelligent. There is a LOT of argument about whether that's an acceptable stance to take, but nonetheless the Turing Test has been an important benchmark for AI ever since. The setting that Turing described to introduce the concept of his test was a situation in which a person and a machine would each be asked questions by a person he called the Interrogator. The Interrogator could ask whatever they liked of either the person or the machine while the machine would try to act person-like and the person would try to help the Interrogator identify who was who. At the end of the questioning, the Interrogator would have to try and guess which entity was the real person.

A schematic view of the Turing Test: Can the Interrogator (C) determine which of A or B is the machine? Schoeneh, CC0, via Wikimedia Commons

Since this initial proposal, there have been different attempts to conduct Turing Tests in widely different settings. These include conversation (like the original proposal), visual art (Daniele et al., 2021), music composition (Ariza, 2009) and even driving behavior (Bazilinskyy et al., 2021). But what about chess? This question is the subject of the study I'd like to tell you about, in which researchers examined how well chess players could succeed at an over-the-board version of Turing's "Imitation Game."

A Turing Test for chess (Eisma et al., 2024)

In this study, the researchers set up a straightforward "Imitation Game" scenario for evaluating how detectable machine play would be over-the-board. However, this included a number of important design choices to ensure that their version of a chess Turing Test wouldn't be trivially easy for their participants. For example, we all know that the best engines are simply far stronger than the vast majority of human players. Besides their strength, they also will choose good moves much faster than nearly any human, especially in difficult positions. Noticing that an opponent is playing with super-human accuracy and speed is an easy way of working out that you're playing a machine, so what can we do to make a Turing Test for chess more meaningful?

The strongest engines are just better than humans - how do we make a Turing Test about more than superhuman chess strength?

To look more specifically at the nature of computer play vs. human play, the authors of the current study set up their Turing Test to balance properties of human and computer play carefully. First, all participants were given the same set of starting positions to begin playing from, all of which were selected from games played by GM Drvitman on Lichess. Each of these positions was the result of the first 10 moves of a Rapid game, and were selected so that there were no tactical forcing moves. You can see one of these positions below:

Figure 2 from Eisma et al., (2024) illustrating one of the starting positions used in their Turing Test experiment.

The participants (24 of 'em in all) had an average Lichess Blitz rating of about 1550 and were told that the objective was NOT to win, but to try and work out who they were playing in each of 8 positions: A person or a machine? In reality, they would face one human player (with a Lichess Blitz rating of 1136) and 3 different engines across these conditions. The engines included two versions of Stockfish (one set to Level 4 and one set to Level 20, the highest you can choose) and Maia, an engine trained to play like a human at different playing strengths (McIlroy-Young et al., 2020). I need to point out that there are definitely people here on Lichess who know MUCH more about the way that Maia works than I do, so apologies if I am doing grave violence to it with this description. That said, the important feature of how Maia is trained is that it is trained on games played by people with the goal of identifying what a person will play when faced with a particular position. Maia is trained using data from games played by humans of varying skill, so users can choose different strengths for Maia that are intended to reflect specific human ratings. For the current study, the researchers selected a level meant to match an ELO rating of ~1100.

Compared to Stockfish, Maia is specifically trained to mimic human play of varying strength during play using deep learning. It achieves this through training data from human games used to establish matches to human moves rather than optimal moves. (image from Maiachess.com)

Subjective ratings of human-ness

With these human and machine opponents lined up for each participant, players were each given 5 minutes on the clock. Participants were informed that their opponent's moves in each position would be made every 10 seconds, minimizing the usefulness of playing speed as a way of guessing who was the machine. After playing through each of the 8 positions (2 for each opponent in a randomized order), the researchers asked the participants a number of questions about who they thought they were playing. These included asking whether the opponent was a human or machine, their confidence in that guess, and the estimated strength of the opponent. You can see the results of those questions below for the 4 different opponents used here - in order: Human, Maia, Stockfish (weak), and Stockfish (strong).

Figure 4 from Eisma et al., illustrating how players responded to various questions about the 4 opponents they faced.

There is a lot to see in this picture, so let me walk you through the main results. First, in general players thought both versions of Stockfish seemed more machine-like, while Maia was not rated as any less human than a real person. This suggests that even when Stockfish is relatively easy to beat (like the Level 4 opponent) there is still something machine-like about the way it plays that is different than what Maia will do. Second, players were consistently confident about their guesses across all opponents. They were also able to identify the difference in playing strength between Level 4 and Level 20 Stockfish.

Play-based descriptors of human vs. engine performance

Besides these ratings of participants' impressions of who they were playing, the researchers also collected data about how the games unfolded to compare objective performance across different opponents. You can see that bar graph below, which includes a number of clear differences across conditions in how many moves were played, the outcome of the games, and a "win rate" descriptor that the authors estimated using the engine evaluation of each game's final position (this may not match the game outcome due to time-out losses with a superior position).

Figure 3 from Eisma et al. (2024) displaying their various objective measurements of chess play across different opponents.

To my eye, the most important results here are that players fared more poorly against both versions of Stockfish (especially the stronger one, which is no surprise) and needed to take more time to plan their moves, too. Looking at this data can't help but make me a bit concerned that players' subjective impression of who they're playing likely depends a great deal on simple playing strength. Maia may appear more human-like simply because it's not as much stronger than the human opponent. I'd love to see more piloting work to help match playing strength between the human opponent, Maia and the weaker version of Stockfish as the current study seems like it has a bit of a confound built into the design. Still, it's interesting to see how these objective descriptors relate to the subjective ratings of playing strength up above, where Stockfish at Level 4 wasn't rated as drastically stronger than Maia. There is something worth studying further here, I think, insofar as it's not clear to me how subjective impressions of playing strength may relate to objective outcomes and/or impressions of human-ness - is there some threshold for relative playing strength that makes people switch their assignment of human vs. machine, for example?

Kibbitzing about who your opponent is

One analysis I thought was particularly neat is the data they have from recording what players said to themselves while playing. (this is in the last two panels of the blue bar graph up above). This allowed the researchers to measure how much players spoke when facing different opponents, but also how often they expressed surprise or confusion about their opponents' moves. Players tended to speak less when facing the stronger version of Stockfish, but express more surprise during play against the weaker version. Machine-like play appears to be more unexpected to humans and to elicit either commentary about the weirdness of a move or perhaps a bit of stunned silence as you realize you're done for.

Besides this, the authors included another natural language analysis of how participants made decisions about the identity of their opponent. What seem to be the key clues to who you're playing, both when you're right and when you're wrong? You can see their GPT-enabled summary of these comments in the table below.

Table 1 from Eisma et al. (2024) - What makes players think their opponent is a human or an engine?

I find this particularly interesting and I'd love to see a more granular look at individual moves coupled with more NLP descriptors of participants thinking aloud about their opponent at different points of the game. What you can see here is that blunders tend to be understood as a dead giveaway for humanity...except when they're not! Maia, for example, seems human because the engine makes various blunders and misses checkmates. StockfishL, however, seems like an engine because it - wait for it- because it makes blunders and misses objvious checkmates.

What are we to make of this? I think a ton of work is being done in these descriptions by words like 'cold," and 'unnatural' moves and 'strange' blunders. What exactly are those? Why exactly do they seem strange to us? What is the difference between a blunder that I can understand my opponent making and a blunder that seems like it must be the result of an engine's calculation? You may have your own intuitions about this and these may point the way for future work. My feeling is that there is a nice indication from this work that there are meaningful differences in the kind of play an engine engages in and the moves that a human makes, but the criteria we use to reach those judgments are still rather elusive.

Conclusions

Overall, the data suggest that there really are such things as "computer moves," at least with regard to how Stockfish plays. The different training used to build Maia leads to more human-like play, with participants generally being tricked into thinking Maia was a human more often than they thought the same of Stockfish. Chess competence still looms as a potential spoiler here, however, so future work would absolutely benefit from closer matching of engine and human ability.

Given that there may be "computer moves," another big question for cognitive science in this domain (at least, I think!) is why even strong players have biases away from these optimal moves that they need to unlearn to get better. Many current GMs (Carlsen and Nakamura included) have talked about studying engines with the goal of understanding why certain puzzling moves do serve a purpose and incorporating those plans into their play. What makes some moves seem more natural than others such that a machine plays like a machine and a human tends not to? For now, Stockfish can clearly beat the living daylights out of us all over the board, but doesn't pass the Turing Test.

Thanks from NDPatzer!

With 2024 nearly over, I can't help but look back on the past year of writing these articles. I first started blogging about chess and cognitive science in 2023, but this year I decided I really wanted to commit to working on my science communication skills while learning more about the science of the game. I've been very excited to see so many people reading these and your comments and questions in the forum have taught me a lot about writing for a wider audience than I've been used to in my professional life. So: Thank you all very much for a great 2024 - this will be my last post for the year, but I'm hoping to have more Science of Chess content for you in the New Year!

Support Science of Chess posts!

Thanks for reading! If you're enjoying these Science of Chess posts and would like to send a small donation my way ($1-$5), you can visit my Ko-fi page here: https://ko-fi.com/bjbalas - Never expected, but always appreciated!

References

• Ariza, C. (2009) The Interrogator as Critic: The Turing Test and the Evaluation of Generative Music Systems. Computer Music Journal, 33, 48-70.

• Bazilinskyy, P., Sakuma, T. & de Winter (2021) What driving style makes pedestrians think a passing vehicle is driving automatically? Applied Ergonomics, 95, 54968-7.1

• Daniele, A., Di Bernardi Luft, C. & Bryan-Kinns, N. (2021) What is Human? A Turing Test for Artistic Creativity. Proceedings of EvoMUSART 2021.

• Eisma, Y.B., Koerts, R. & de Winter, J. (2024) Turing tests in chess: An experiment revealing the role of human subjectivity. Computers in Human Behavior Reports, 16, 100496.

• McIlroy-Young, R., Sen, S., Kleinberg, J. & Anderson, A. (2020) Aligning superhuman AI with human behavior: Chess as a model system. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1677-1687.

The Greek Gift is a chunk a lot of beginner and intermediate players acquire early in their training.

There is a fair bit of evidence for template theory by now, which has motivated chess improvers and coaches (both real and digital) to emphasize tactics training. Drill those tactics, and you'll assemble your own library of chess chunks! With a healthy collection of templates, you too may be able to immediately spot key configurations during play and make the right moves with ease. When I'm not playing too many Blitz games, I do my share of puzzles too in the hopes that I can take advantage of this formula.

I can't decide whether spending 5 full 24-hour days on chess puzzles is a point of pride or not.

Expert players who presumably have stored lots of chess patterns in their long-term memory (or LTM) have a bunch of advantages over weaker players that I've described in previous Science of Chess posts. They tend to be better able to remember chess positions accurately and can reconstruct them from memory (De Groot et al., 1996), for example. They also seem able to correctly perceive where pieces are on the board without needing to move their eyes very much, an ability that suggests a greater perceptual span than novices (Reingold et al., 2001). Stored templates almost certainly contribute to successful blindfold chess, too: The more you can reference long-term memory to keep a position in mind, the less likely it is that the bottleneck of your short-term memory (or STM) limits your ability to calculate, etc.

Presently, I want to introduce you to another interesting by-product of having lots of chess knowledge stored in templates, patterns, or "chunks" if you prefer: Making chess judgments without awareness. Besides the various ways experts outpace novices that I described above, it turns out that experts also seem capable of making meaningful judgments about chess positions without being conscious that they are doing so. But what does this mean and how would we ever measure such a thing? The answer to both of those questions lies in understanding a cognitive science phenomenon called priming.

What is Visual Priming?

To understand what priming is, there are a few facts you need about how your mind and your brain process visual information. The first fact may be one you have some intuitions for: It's possible to show you an image so quickly that you don't really see it. To be just a little technical, we can use a technique called visual masking to get this done. Masking works by showing you one image for a very short amount of time (maybe 16ms or so) and then immediately showing you a second image, usually visual noise of some sort. Even though both patterns were visible, that second picture effectively cancels your experience of seeing the first one. This means that though your retina might have had that pattern of light land on it for a moment, your mind never gave you awareness of it. You can see a schematic view of how masked stimuli are deployed in an experimental setting, in this case, including both a pre-stimulus mask and a post-stimulus mask.

Presenting images of static or clutter before and after a target image (here, the word "radio") can make the target image invisible to an observer. This is called visual masking. MattF, CC0, via Wikimedia Commons

The second fact I need to give you is the more counter-intuitive one: An image you weren't aware of seeing can still affect your behavior. I'll give you my favorite example of this, which I think is so much cooler than so-called subliminal advertising - the phenomenon of blindsight. Blindsight refers to a circumstance in which individuals with extensive damage to their visual cortex (critically, NOT their retina!) are blind, but nonetheless prove able to avoid obstacles while walking, point towards visual targets, and in some cases identify letters, all without a subjective experience of seeing anything. I've put a link to a short video of a patient with blindsight in the references in case you want to see what this is like.

A still frame from the video linked in the references in which a patient with blindsight can walk down this cluttered path without any experience of seeing. This is one example of how visual information can affect your behavior without conscious awareness.

A blindsight patient's ability to step around an obstacle they can't see is a remarkable example of how visual information can end up being used by your nervous system without reaching visual awareness. The phenomenon works by virtue of some visual information getting to a part of the brain that can use that data to guide where your feet go next. That part of the brain doesn't provide any kind of conscious awareness of the information, however! The primary visual cortex (at the posterior end of the occipital lobe, labeled V1 in the figure below) appears to be crucial for providing you with a subjective experience of what you are seeing. In cortically blind patients with blindsight, this is the part of the visual cortex that is extensively damaged. The result is that the patient takes the right step around a dropped object, (supported by intact retinae and functioning areas outside of V1) but still doesn't see that it's there.

Patients with blindsight lack the neural pathway at right that leads to conscious visual experience in area V1. They retain the pathway at right that delivers visual information to other sites that can guide motor behavior and other processes, leading to "seeing without seeing."

Together, these two facts are what we need to understand how visual priming works. In priming experiments, the idea is to (1) use visual masking as a tool for showing people images they don't "see" and then (2) see if the information in those images was used by the mind and the brain.

A priming example

Imagine this situation, for example: I'm going to show you some pictures of objects that are either red or green. Your job is just to look at each picture and tell me the color you see by pushing one button or the other. Easy, right? The tricky part is that I am secretly going to show you a masked image right before each picture you'll actually see. That masked image might be congruent, meaning it is the same color as the one that will be visible, or it might be incongruent, having the opposite color. It turns out that even though you won't see the masked images, you'll be a little faster to push the right button when the masked image is congruent than when it is incongruent. Like our blindsight patient, an image you never saw has the potential to change the way that you respond.

A diagram of how visual priming is intended to work: The red prime images on the left may be presented too quickly for someone to see, but the fact that the prime would be called "Red" makes the "Red" response to the first target image a little quicker (top row) than if the prime doesn't match the target (bottom row).

With that in mind, let's return to our chess experts with lots of stored templates in their LTM and our chess novices who are stuck using their STM and working memory more heavily. Kiesel et al. (2009) decided to see if the differing libraries of templates these two types of players stored meant that expert players could effectively play chess without awareness. That is, could they evaluate chess positions that they weren't really seeing? To test this idea, they used a priming design very much like the red/green experiment I described above, only they had to make the task a little harder.

Can you prime chess players with checking attacks?

Specifically, instead of asking players to make a basic judgment about something like object color, they asked their participants to do a check-detection task on a 3x3 mini chessboard. Each board had a Black King and a White piece that was either attacking the King (Check condition) or not (Non-check condition). Players would get to see this target image for 250ms (a quarter of a second) and were asked to report if the Black King was in check or not. By itself, this is already a fairly tough task given just how quickly these images are being shown! Nonetheless, players tended to be good at this part of the task.

A check-detection example on a mini-board from Reingold & Sheridan (2023): Is the White King in check from either of these two pieces?

But now the tricky part again: Before each visible mini-board, a masked mini-board was presented for just 20ms, sandwiched between two 70ms pictures of visual static. (see the figure below) This masked board couldn't be seen by the participants, but it was either congruent or incongruent with the 250ms image that they could see. Half of the time, the Black King's status was the same in the masked image and the target image. The rest of the time, it would be different. The question is: Does unseen visual information about check/noncheck affect judgments about boards you can see? If so, that suggests that the masked board is not only being delivered to parts of the visual system, but that the information on the board is being used to make a judgment about a checking attack.

Figure 1 from Kiesel et al., (2009). Mini-boards that either match or don't match target images according to the presence of checks serve as masked primes. Can players detect the checks in those boards quickly enough to affect their speed with the target images?

The interesting result here is two-fold. First, those masked images do turn out to affect how quickly players responded to the visible images - congruent masked images led to faster responses to the visible target images. Second, however, this effect depended on players' expertise. Only experts showed a clear affect of priming, while novices were not affected significantly by the masked images. You can see the actual response times in the table below, where I've highlighted the numbers you need to see these two features of the data.

In the table above, the yellow boxes show you the time it took experts (left box) and novices (right box) to detect checks in target images after non-matching and matching primes. That small difference in the response time of the experts in those two conditions means "unseen" boards were being understood without awareness.

You may feel a little skeptical about this result given how small the difference in response time is between the congruent and incongruent conditions: We're talking about an effect of ~10ms on average, after all. There are two things I have to say about this. First, the size of the response time difference isn't what we're really interested in here. Instead, the presence of that significant difference is taken as evidence that the masked image was indeed interpreted as a "check" or a "non-check" - else there should be no effect on the response to the target image at all. Now for the grain of salt: These kinds of small effect sizes are very much the kind of thing that my lab (and my colleagues' research groups) tend to look at more critically these days. The replication crisis in psychology made a lot of us start thinking more carefully about best research practices including how we determine our sample sizes, how we plan to analyze our data, and how precisely we measure the effects we hope to draw conclusions from. There are things in the Methods section that I like to see (the direct test of the interaction between participant group and priming, for example) but other things that make me worry (only 12 expert players!). This is all to say that while this is an intriguing result that's consistent with other research about expert play vs. novice play, this is also one of those things I'd love to see replicated and extended.

Still, to the extent that this result holds up, what this suggests is that chess experts' library of templates and patterns may make it possible for them to get information about checks, attacks, and captures from images that they are unaware of. Another way to put this is to say that useful information about chess positions is measured automatically by expert players and influences their behavior even if they aren't conscious of the data. While a poor patzer like me usually resorts to carefully working my way through a checklist of things to look for over-the-board, expert players' vast experience gives them the capability of playing chess even when they aren't conscious that they're doing so.This has implications for how various cognitive processes, including distinct kinds of memory, may be recruited in players of different strength.

The CHREST model from Smith et al. (2009) includes interactions between various kinds of memory during chess play - experts likely rely on these different kinds of memory in ways novices can't.

There are a lot of interesting questions that follow from this result, too. Just how much information does an expert get from a masked image like the ones used here? Sure, subliminally "seeing" check on a 3x3 mini-board is cool and all, but what could a player get from a full board that was masked? Also, while templates are useful for a lot of things, they also have their pitfalls: The Einstellung Effect refers to errors players make when relying too heavily on stored chunks and patterns. Are there circumstances where a prime could be detrimental to accurate board judgment? Finally, how do effects like these develop as a player gains more experience?

Conclusion

As always, I hope that this discussion highlights some interesting things about how our brains and our minds play chess. For me, this study is a great example of how techniques from visual cognition can reveal things about expert play that aren't easy to observe from what happens during a game. Thanks as always for reading, and look for more Science of Chess soon.

Support Science of Chess posts!

Thanks for reading! If you're enjoying these Science of Chess posts and would like to send a small donation my way ($1-$5), you can visit my Ko-fi page here: https://ko-fi.com/bjbalas - Never expected, but always appreciated!

References

1. de Groot, A. D., Gobet, F., & Jongman, R. W. (1996). Perception and memory in chess: Studies in the heuristics of the professional eye. Van Gorcum & Co.

2. Kiesel, A., Kunde, W., Pohl, C., Berner, M. P., & Hoffmann, J. (2009). Playing chess unconsciously. Journal of experimental psychology. Learning, memory, and cognition, 35(1), 292–298. https://doi.org/10.1037/a0014499

3. Reingold, E.M.. & Sheridan, H. (2023) Chess expertise reflects domain-specific perceptual processing: Evidence from eye movements. Journal of Expertise, 6(1), 5-22.

4. Reingold, E. M., Charness, N., Pomplun, M., & Stampe, D. M. (2001b). Visual span in expert chess players: evidence from eye movements. Psychological science, 12(1), 48–55. https://doi.org/10.1111/1467-9280.00309

5. Smith, Ll., Gobet, F., & Lane, P. C. R. (2009). Checking chess checks with chunks: A model of simple check detection. Proceedings of the Ninth International Conference on Cognitive Modeling.

6. Blindsight video:

White King on e5, White Queen on g4. Black King on h8.

...and here's a few blank rows to not spoil the experience! (the position is visible below).

.

.

.

.

.

.

.

Sure, you could open up a board editor and set this up, but I want you to try and solve it without doing so. While you're doing that, my real question isn't about whether or not you solve the puzzle but what your experience is of trying to do so. What are you thinking about? What's happening in your mind while you try to figure out the answer?

"Seeing" while blindfolded and imagining what's to come

This puzzle, by the way, is the first of Martin B. Justesen's 1160 Blindfold Chess Problems which is a wonderful e-book filled with increasingly challenging puzzles all to be solved without a board. I also won't keep you in suspense any longer in case you (like me) find blindfold problems very difficult. Here's the position I just described.

I'm guessing that this helps a lot. In case you really hate spoilers I won't post the answer here, but I'm confident that most readers will work it out very quickly. By contrast, I suspect that trying to think this through without a board to look at was quite challenging, even with this sparse position. Despite this apparent difficulty, many past and present chess players have demonstrated remarkable abilities to work through complex positions without looking at a board: Philidor (pictured in the thumbnail image), Alekhine, Morphy (pictured below) and a host of other famous players were (and are!) capable of winning multiple blindfold games in simultaneous exhibitions.

www.academicchess.org/.../morphy2.shtml, Public domain, via Wikimedia Commons

Blindfold chess at this level is a particularly startling demonstration of an ability nearly all chess players try to use in normal play - keeping a position in mind and imagining what comes next. During a game, we all try to think through the consequences of candidate moves so we can decide whether particular next steps are good or bad. Like playing blindfold chess, this can be quite difficult. I tend to feel confident about calculating about two or three moves ahead, but beyond that I start to lose some kind of clarity about exactly what is where. This is an imperfect analogy, but it feels a little like the "Fog of War" variant in which some of the board is obscured during play.

Chess.com "Fog of War" example

A key part of my experience of trying to calculate is that I do feel like I'm picturing something*,* though. I describe my estimate of the board as "clear" vs. "foggy" because it seems like an image I either can or can't see very well. That's just my intuition, though - Is that really what my brain and my mind (or yours) are doing when I calculate?

You might wonder what the alternative is - doesn't everyone have a mental picture they use when they try to think ahead OTB? The answer, which has become the focus of a lot of interesting visual cognition research lately, is no! Different people report very different experiences of visualizing things in the "mind's eye," with some individuals reporting no visual imagery at all*.* This condition is called aphantasia and is the extreme end of a spectrum of visual imagery that researchers usually assess with simple surveys.Below you can see an example of the kind of question researchers might ask participants to respond to on such a questionnaire: When you picture an apple, how vivid is that image you hold in your mind's eye? If you're interested in assessing your own imagery, you can try out the Vividness of Visual Imagery Questionnaire (VVIQ) for yourself.

By Composition by Belbury, original image components by Mrr cartman, Caduser2003, Bernt Fransson and IconArchive.com - This file was derived from ilhuette.png:Apple 001.jpg:Äpplen 001.jpg:Red Apple Icon.png:, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=142324043

The VVIQ has important limitations, but I think it's fair to say that it provides meaingful evidence that visual imagery does indeed vary widely across different people. What does that potentially mean for how chess players try to maintain and transform a given position to evaluate their next moves?

Verbal and visuospatial modules in working memory

If you (like me) are someone who has the subjective experience of seeing a mental image of some sort when you imagine things, it probably seems difficult to consider what it's absence would be like. My experience of trying to calculate during a game makes it tough to understand what I would be doing if I didn't have a picture in mind. What is imagination without an image? To be clear here (and mindful of different people's experience), I want to emphasize that I don't mean that as a rhetorical question or to imply that varying vividness of imagery is a problem. Instead, it's a question about what that subjective experience is like. Individuals with aphantasia report using a lot of different strategies, but one I want to focus on is the use of verbal descriptions rather than pictorial descriptions. If you've ever watched Hikaru Nakamura calculate on his stream, you've certainly seen images like the one below, suggesting a highly pictorial (or at least spatial) representation of the board.

Arrows!

On the other hand, the same streams also often include him saying moves out loud in rapid sequence! "Nf3-g4-Nh5-Kh7-captures-captures-and-mate" for example, though please be aware that I just made that one up! That kind of quick description of a combination in words might be evidence of an equally strong verbal representation of the board that can be expressed in words or symbols. So here's the interesting question if you're someone who thinks about vision, cognitive science, and chess - what kind of imagination, verbal or pictorial, do players tend to rely on the most?

A classic model of human working memory (the kind of memory we use to keep information in mind for short periods of time) suggests people use a cognitive module we call the visuospatial sketchpad to keep track of pictorial and spatial information and a different module we call the phonological loop to keep track of verbal content. If you've ever tried to remember a 2AFC code or something similar for a few moments, you may have found yourself internally repeating the letters or numbers in the code so you wouldn't forget. That kind of phonological rehearsal is a process that memory researchers would attribute to the phonological loop. On the other hand, keeping a picture in mind or a sense of where things are in space during a brief interval would be attributed to the visuospatial sketchpad. Given that these modules appear to be meaningful to talk about cognitively, my question about verbal or pictorial approaches to representing and transforming a chess position turns into a question about specific cognitive processes. How are the phonological loop and the visuospatial sketchpad engaged during play?

FrozenMan (talk) (Uploads), CC0, via Wikimedia Commons

Using task interference to measure how modules contribute to complex tasks

This is a hard question to answer. The problem with trying to understand and measure someone's imagination is that it all happens inside the mind without obvious outward signs. There are some interesting candidates for objective measurements of imagery, but many of these have proven difficult to replicate and are thus hard to rely on. For example, there are some results suggesting that eye movements while "scanning" an internal image or pupil size while picturing a bright light may reflect the vividness of an individual's visual imagery. A number of these results have proven difficult to replicate however, so it pays to be careful. But in the absence of something like that, what do we do? Do we just ask people and hope that they have good insight into their own mental processes? Unfortunately, we can't just rely on that kind of introspection because metacognition (thinking about your own thinking) is also not guaranteed to be accurate.

There is, however, a clever way to estimate how verbal and pictorial imagination each contribute to a complex task like chess calculation. The big idea is to use task interference to measure how much one can interrupt the main process we're interested in (here ,chess calculation) by asking people to do something else at the same time. The logic behind this approach is that if the main task and the interfering task rely on the same processes and cognitive resources, it should be tough to do both at the same time. On the other hand, if they rely on different things, doing both shouldn't be so difficult.

This is the strategy reported in Saariluoma (1992), in which the researchers worked with 8 chess masters (average ELO over 2200) and 8 intermediate players (average ELO under 1750) to test how visuospatial and verbal interference affected chess calculation. The main task both groups were asked to complete was "Mate-in-1" detection: Given a position with White to move, can Black be checkmated on the next move? For half of the trials the answer was yes, while in the other half White would need 2 moves to win.

A trickier Mate-in-2 than those used in the target article. Jan.Kamenicek, CC0, via Wikimedia Commons

But now here comes the interesting part: While players were trying to work these puzzles out, the experimenters asked them to do one of two secondary tasks at the same time. In their Verbal task, each player was asked to repeat a novel word over and over again (the word was TAIKURI) to occupy their phonological loop. In their Visual task, each player was asked to imagine tracing a path around the letters of the word TIIKERI and deciding if the turn they would have to take at each corner they arrived at was a left or right turn. This task was intended to occupy their visuospatial sketchpad. If this second visuospatial task sounds strange to you, I get it - however, this is a variant of an established neuropsychological assessment called the Money Road-Map test that has been successfully linked to visuospatial processing in the human parietal lobe.

With these two tasks in hand, the critical idea is that if either of these interfering tasks makes solving Mate-in-1 puzzles harder, that's good evidence that you need that cognitive module (and presumably that kind of information about the board) to help you. You can see the results (graphed here as the time it took to answer the Mate-in-1 question correctly) depicted below. Because this is response latency data, a taller bar means that participants were slower and we assume that the task was more difficult. Shorter bars mean faster and we assume easier responses.

Figure 2 from Saariluoma (1992).

What these bars show you for the Masters (at left) and the Medium (or Intermediate) players at right is that repeating a word over and over again doesn't make you any slower to detect Mate-in-1. The black "Control" bar shows you the time it took to solve these mating puzzles with no secondary task interfering with you, so the hatched bar being about the same height as this one means this secondary verbal task didn't make the main task any tougher. However, the secondary visual task does interfere with both groups' speed! That dark-grey bar being higher than the other two in each case means occupying your visual imagination with another task made chess puzzle-solving harder. The conclusion is that even though verbal and auditory information seems like it could play a role in how you calculate and imagine a board, ultimately, chess appears to be more visuospatial. than verbal.

Next steps and further questions?

This is one of those results that may strike some readers as obvious - if chess seems very visual to you, you may wonder if the researchers had to go to all this trouble to test that idea this way. The thing is that this very easily could have turned out differently in ways that would have led us to different conclusions: Doing ANY kind of secondary task might have made the main chess task more difficult for example. That could either imply that players rely on both working memory modules to play or that chess is sufficiently demanding that any additional cognitive load makes things tougher to do over-the-board. You could also certainly have had verbal interference incur more of a cost than spatial. Because introspection and metacognition aren't perfect, doing tasks like this to check our intuitions is important for getting more evidence about how different mental processes contribute to complex tasks.

That said, there are a lot of important limitations here that are worth considering. For one, this is a case where modern training about statistics makes this result look a lot less convincing - 8 participants per group is just not a fantastic number and I'd love to see this result replicated (and maybe extended) with larger samples. Also, while tasks like check-detection or Mate-in-1 puzzles are easier to implement in an experimental setting, I'd also like to see how these results play out with a more naturalistic version of playing chess. Finally, there are definitely different forms of visual and verbal interference that could be worth considering. Compared to just repeating a word over and over again, Speech Shadowing in particular can be quite demanding, Other kinds of verbal overshadowing are known to disrupt a range of processes, including visual tasks like facial recognition, which may mean another look at visual and verbal processes during chess play would be worth considering.

Something that I also think is an interesting question related to this topic has to do with training. To what extent can we improve our visual imagery? What are the most successful kinds of training to do so, if any? I'm about to embark on Aidan Rayner's suggested visualization training regimen at some point in the next few weeks, and I'm very curious to see what it's like and how well it helps me see a little more clearly through the fog of my own visualization. As always, there are remain a number of intriguing next steps to take from here, but this study is a neat way to see how we can try to peek "under the hood" with careful cognitive testing to understand how the brain and the mind help us play the game.

Support Science of Chess posts!

Thanks for reading! If you're enjoying these Science of Chess posts and would like to send a small donation my way ($1-$5), you can visit my Ko-fi page here: https://ko-fi.com/bjbalas - Never expected, but always appreciated!

References

Baddeley A. (1992). Working memory. Science (New York, N.Y.), 255(5044), 556–559. https://doi.org/10.1126/science.1736359

Marks D. F. (1973). Visual imagery differences in the recall of pictures. British journal of psychology (London, England : 1953), 64(1), 17–24. https://doi.org/10.1111/j.2044-8295.1973.tb01322.x

McKelvie, S. J. (1995). The VVIQ as a psychometric test of individual differences in visual imagery vividness: A critical quantitative review and plea for direction. Journal of Mental Imagery, 19(3-4), 1–106.

Saariluoma, P. (1992). Visuospatial and articulatory interference in chess players' information intake. Applied Cognitive Psychology, 6(1), 77–89. https://doi.org/10.1002/acp.2350060105

Vingerhoets, G., Lannoo, E., & Bauwens, S. (1996). Analysis of the Money Road-Map Test performance in normal and brain-damaged subjects. Archives of Clinical Neuropsychology, 11(1), 1–9. https://doi.org/10.1016/0887-6177(95)00055-0