What has changed is the tolerance for waste. Six months to build an intervention, launched into the business with no expectation of provable impact, is no longer a accepted in many organisations. Timescales need to be reasonable, which is a contextual judgement, and impact needs to be demonstrable, which is not. Neither has anything to do with whether the thing you built was a course or a resource; both are questions of working practice, and apply to a commercial content library just as to an in-house team. Being asked to show the work was worth the investment is a good development, and if you or your team are finding it uncomfortable, it probably indicates a need for you to seriously assess your working practises.

Now, this isn’t a wholesale defence of every course out there. There are plenty that are, quite frankly, terrible!

When I look back in my career, there are certainly some I wish I could scrub from existence. Courses, when compared to, say, a job aid or a resource or directing people to a knowledge management solution, are much more time-consuming in terms of creation and putting people through them. They require a greater level of financial and resource-based investment, but they can also provide things that none of these other solutions can, most notably, if designed well, an opportunity for practice. Notably, that is well-designed practice, not just “hey, go do a thing.”

Put simply, I would argue that a big part of our job is figuring out when the additional investment required for a course-based solution is worth making from an organisational standpoint (i.e., what will deliver the required performance uplift in the most efficient way). There will be times when the answer to that question is, knowledge management, a resource, an environmental change, coaching, mentoring, and yes, there will be times when it is a course. Hey, sometimes it might even be all of them, almost like some kind of blended solution… crazy I know!

Every few years something is announced as the successor to everything else: mobile would see off the desktop, gamification would absorb all other forms of training, VR was the future of corporate training. Claims of this kind are often made by people with something to sell, usually the technology they are telling you is the future, and the replacement they describe tends to never quite arrive.

Each of those technologies found a place and changed what our work looks like without displacing what came before, and generative AI looks likely to follow. Human beings do not develop at the speed of technology. Thank goodness.

Ultimately, the best way for us to adapt is not to swing wildly for these things as they’re talked about, or to abandon things wholesale because the talking head consensus is that they’re dead, but instead to experiment broadly, rapidly, and loudly so that we can all learn together how to harness new things without surrendering what’s already been proven to work.

Leave a comment

July’s research could best be defined as confirmation and nuance-building, nothing groundbreaking and nothing massively new, but still worth thinking about.

As always, before we get started, two quick pieces of housekeeping:

1. A single study proves nothing on its own, however large the sample and however neatly it confirms what we already believed; each one adds a bit of weight to an idea or takes a bit away, and where a study challenges what you think you know, the useful response is to ask what questions it has opened up rather than to throw out the study or the belief.

2. Industry white papers need reading differently from research. White papers have a commercial agenda behind them the publisher is usually selling something adjacent to its findings; that doesn’t make them worthless, but it does change how we should think about their findings.

   Instructional Design Tips is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

   Get 7 day free trial

Build and recall, or recall and build?

Two kinds of activity dominate the practice phase of most well-designed interventions. Either people build something out of what they’ve just encountered, an example of their own, an explanation in their own words, a worked case; or, they recall what they’ve encountered from memory, which is what most of us mean when we talk about retrieval practice.

The first is helping people construct an understanding and the second helps it stick. As you might expect, using both tends to be better than either one alone. But does the order we use these in matter?

Obergassel et al. (2026) ran an experiment with 231 university students, comparing a build-then-recall order against a recall-then-build order. They also included a restudy-then-recall version for comparison, and varied between these activities, coming straight after initial study and two days after the initial study period.

Apart from a difference in how well people did on the recall activity itself when it came after the two-day gap, the orders produced the same results.

If you already combine building activities with recall activities, good news, you don’t need to agonise over which comes first, and the design effort you might have spent on that question can be saved and used to make sure both are created well.

Researchers in this area have pointed out before that construction activities are often run without feedback and without any chance for users to correct what they got wrong.

One thing to note is that this single experiment looks only at when these are used immediately after study and after a two-day gap. Whether the order stays irrelevant over longer gaps, and with procedural content rather than conceptual, would be an interesting next study in this area and something I’ll be keeping an eye out for.

References

Obergassel, N., Renkl, A., Endres, T., Nückles, M., Carpenter, S.K. and Roelle, J. (2026) ‘Generative and retrieval tasks: does the sequence matter and do sequence effects depend on learning task delay?’, Applied Cognitive Psychology, 40, e70188.

Typed answer or answer in your head?

Pre-testing is pretty widely used, whether it’s to offer people a shortcut through a repetitive course that they have to do every year or to enable those with greater prior experience to not waste their time covering ground they already know well. Have you ever wondered whether or not it matters if they actually record an answer or just think about one?

Rivers et al. (2026) had undergraduates read informational articles, preceded either by questions they answered by typing or by questions they answered in their heads. Both versions improved how much people got right on a later test compared with just reading the article, and neither version came out ahead of the other. A second experiment produced the same pattern.

So, a question asked out loud in a webinar, or sitting on a slide before a demonstration, appears to be as effective in supporting learning, as a typed response.

But, the picture looks different for recall practice after instruction. Some classroom work has found that having people write or say their answers may be more effective than having them think it, possibly because users who aren’t required to produce anything don’t always bother doing the recalling (Witherby, Northern and Tauber, 2025).

What this means for us is that if you’re not pre-testing during an intervention because it was going to be tough to capture the answers, maybe in a workshop context or perhaps a webinar, it might be worth adding in some pre-testing and simply allowing users to think about their answers.

References

Rivers, M.L., Berdelis, A.J., Pan, S.C. and Tauber, S.K. (2026) ‘Does covert prequestioning enhance learning? Implications for the roles of attention and retrieval in the prequestioning effect’, Applied Cognitive Psychology, 40(1), e70165.

Witherby, A.E., Northern, P.E. and Tauber, S.K. (2025) ‘Does covert retrieval benefit adolescents’ learning in 8th grade science classes?’, Behavioral Sciences, 15(7), 843.

Goals about technique, or goals about the result?

We tend to define the success of our interventions in behaviours, tasks, and standards; what someone should do, how they should do it, and to what standard it should be done. And there’s a solid case to say this makes perfect sense, but it’s not the only way.

When I worked in product, the goal always represented the outcome we wanted to see. We didn’t define success or failure as being tied to one specific route to that outcome. Now, this is not inherently better or worse. It’s simply a different approach.

Tsuda et al. (2026) had a hundred people practise Boccia, a Paralympic Precision Ball Sport similar to lawn bowls or curling, while wearing a soft robotic exoskeleton on their arm, split across five groups. One group worked to goals about technique, one to goals about their score, one moved from technique to score partway through, one had goals adjusted as feedback came in, and one acted as a control.

People in the score, shift, and adjusted-goal groups came out of it more confident in their own ability at the task than the group working only on technique.

For clarity, this study is set in a rehabilitation environment, and the task is highly physical. We therefore have to think carefully about where we may or may not want to consider applying what we can learn here. That doesn’t, however, mean we should ignore it completely. Whether in coaching conversations or capability frameworks, the objectives we write for development programmes do tend to lean heavily on process language.

The two most interesting groups for our purposes are the ones where the goal moved: from technique to result partway through, and in response to feedback. Both describe a goal that changes as the person progresses, you can probably see how this would align with the work of a coach or line manager.

For now, this is something I’d be inclined to test or experiment with. If you have the volume, try it across two cohorts and watch whether confidence and actual application come apart, feeling more capable and being more capable aren’t the same and this study only measured the first.

The version of the paper available at the time of writing is an early-access manuscript that hasn’t been through final editing.

References

Tsuda, A., Manalo, E., Teramae, T. and Noda, T. (2026) ‘Unveiling the impact of goal types: enhancing self-efficacy in rehabilitation robotics’, npj Science of Learning.

Does uncertainty make people curious?

There’s a pretty well-established link between uncertainty and curiosity, and this often forms part of the discussion in which people claim that periods of organisational upheaval and change are fertile ground for developmental growth.

That’s a convenient thing to believe when you’re trying to launch something during a restructure. Is it true?

Ali, Yang and Poh (2026) analysed around 570,000 posts and comments from about 55,000 students talking to each other on Reddit between 2019 and 2023, using the questions people asked as a visible sign of curiosity. They separated two kinds of uncertainty: the near and specific sort that comes with an exam period, and the vague, background sort that came with the pandemic.

Month by month, the number of questions people asked rose sharply during exam periods, across every topic being discussed. Pandemic uncertainty barely registered. Looking hour by hour at how conversations behaved, the changes were smaller and varied by topic.

In our context, this leads me to wonder if attaching social learning to something specific and imminent will be more effective than a general campaign to build a learning culture. It also suggests that a period of organisational ambiguity is not, on its own, a moment when people get more inquisitive.

Communities of practice, discussion channels, and cohort programmes come alive around a current project, a deadline, or an assessment, and go quiet when the only thing on offer is an invitation to discuss things. This certainly matches much of what I’ve seen in organisations where seemingly vibrant communities and knowledge-sharing spaces have died for lack of a clear and defined purpose.

One big consideration when we read this study is that these are students on a public forum rather than employees in an organisation, and what was counted was questions asked rather than anything learned. What this really got me thinking about was where the design effort goes when launching social learning programmes. Perhaps we should spend less time on launching the chosen platform, be it technological or environmental, and more on understanding how the initiative attaches to the real world of work. Is it permanent or temporary? Is it attached to a specific task or a goal? How do these decisions define what we do or do not do?

References

Ali, F., Yang, Y. and Poh, J.-H. (2026) ‘Curious behaviors in times of uncertainty: dynamics of student question-posing in online educational discussion’, npj Science of Learning.

Share Instructional Design Tips

Applying training and passing it on

For as long as I’ve worked in the field, it’s been a known reality that when people don’t know what to do, they are far more likely to ask a colleague perceived as being a high performer or knowledgeable than to look up the right answer in your perfectly indexed knowledge base. A useful question to consider, then, is whether or not our goal of learning transfer within the initiative is sufficient to enable the employee not only to learn what we need them to learn, but to pass that information on accurately to a colleague.

Mehner, Rothenbusch and Kauffeld (2026) set out to answer that across two studies: 537 German-speaking employees who’d completed job-related training in the previous two years, and 507 US employees surveyed twice, with the conditions measured first and the outcomes measured a month later.

Both outcomes ran through the same chain. People need to want to do it, they need to stick with wanting to do it when something gets in the way, and then they do it.

Training that had been deliberately built to support application made people keener to apply it, but made no difference to whether they passed it on. Training that people saw as relevant to the actual job made them keener to pass it on, while its link to their own application was marginal. Probably most interestingly of all, conversation between participants during the training made people keener to do both.

On the organisational side, support from colleagues and getting feedback on your work mattered for both. Having the time and capacity to apply things mattered for application. And support from your manager was linked to passing knowledge on, but showed no meaningful link to applying it yourself.

That last point runs up against a large body of transfer research in which manager support is one of the more dependable predictors, and this study measured whether people wanted to apply their training rather than whether they did. The same goes for an odd result where people more confident about applying their training were slightly less keen to share it, which the authors themselves flag as possibly a quirk of the statistics rather than anything real.

What I’d take from it is that a programme built entirely around individual application is leaving the multiplier effect on the table. If you want to support people helping their colleagues back on the job, this study suggests that we ensure:

• they recognise the training as relevant to their work,

• they are provided with materials that they can take back into the workplace that are genuinely useful,

• we provide them with opportunities to converse and exchange ideas during the training initiative rather than trying to introduce social elements once they’re back on task.

The most immediately useful thing to come out of the paper is the measures themselves, which give you something short and tested to put in a post-programme survey in place of the reaction sheet.

The caveats are the usual ones for this kind of research:

• Everything is self-reported, including whether people did any of it.

• Participants were casting their minds back to training they’d done several months earlier.

• The authors are explicit that they have found correlation and not causation in this data.

References

Mehner, L., Rothenbusch, S. and Kauffeld, S. (2026) ‘When formal training leads to informal learning among colleagues: antecedents of training transfer and knowledge sharing’, Frontiers in Organizational Psychology, 4, 1705256.

Final thoughts

July proved to be a fairly interesting month. None of the studies I’ve included here are ground-breaking. They don’t tear down or lift up any new insight, but they do provide us a set of interesting questions to ask ourselves about how we do things, especially in the field of social learning.

As always, I strongly encourage you to take the time to check out the original articles and give them a read. Taking the time to get comfortable with research makes us better able to inform our practice and challenge one another with new ideas and things that we might need to consider.

There will be another Month in Evidence article early in September looking back at the month of August, but in the meantime, feel free to share any research papers you’ve come across and found interesting down in the comments and remember, these insights are just one of the three key considerations we must take into account as evidence-informed practitioners. We must take the findings of research and combine them with our professional experience that we’ve earned on the job and with the organisational and audience insights we gain from our specific work in contexts. Only by bringing together all three sources of insight can we properly inform our practice.

Leave a comment

The reality is that unguided observation is going to cause problems. Asking line managers to observe how people are doing the work, record it, and send it back to you is a high-friction activity. It’s a task that lacks clear parameters and is very difficult to build a system around. So, some don’t do it, some do it occasionally. Some do it all the time, but looking at all the wrong things. The data you gather is inconsistent, incomplete, and probably not particularly useful.

Get 7 day free trial

Because of this, I often recommend some kind of observation scale when trying to introduce line manager observations at significant scale. Of these, I regularly end up talking about behavioural observation scales, developed by Latham and Wexley back in 1977.

You start by writing behavioural statements. A behaviour on its own is a vague thing to ask anyone to look for; “communicates well” or “gives feedback” could describe a wide range of activities that many line managers might struggle to notice, and two managers watching the same conversation could score them differently and both be right. A behavioural statement describes one complete action, in context, in enough detail that there’s minimal interpretation required.

Some examples include:

• “conducts performance reviews on time,”

• “conducts the performance review as a dialogue with the employee,”

• “completes GDPR confirmation process.”

The line manager, or observer, either saw that happen or not.

The statements are built from critical incidents, which are accounts gathered from SMEs of specific occasions where something somebody did made a notable difference to how the work went, for good or bad. Latham and Wexley collected over 1,200 of them and reduced them to 78 statements. Sorting those statements into groups is where factor analysis comes in: they looked at which statements tended to be scored the same way by the same observers, on the assumption that statements moving together are picking up the same underlying quality. Each of those clusters becomes a heading on your scale, so the categories are grounded in how the statements behave in practice and not in somebody’s assumption about what the categories ought to be.

The manager then records how often they have seen each behavioural statement on a five-point scale:

1. Never: 0%

2. Seldom: 1 to 20%

3. Sometimes: 21 to 40%

4. Generally: 41 to 99%

5. Always: 100%

The scales are summated, which means the individual scores are added together. Every statement sitting under a heading gets a score, those scores are totalled, and the total becomes the score for that area of performance, which is what you report on and track over time.

This works at scale because you’re asking managers for something specific and bounded: a defined list of behaviours, defined scoring criteria, and a few minutes on whatever cycle you’ve set, weekly, monthly, or quarterly. Building a reporting platform on top of it becomes far easier, because the output is a number.With this, completion tends to go up, the data stays consistent between teams, and the analysis is more easily automated.

This approach is not perfect, and the first thing you lose is any kind of totally unexpected insight. That’s why I always like to maintain an open and optional space for line managers to leave comments. This is where you can still occasionally find a nugget of gold that might explain something that’s not showing in the numbers.

It’s worth remembering that none of this works without properly defined behaviours that are applicable across the organisation. What you can’t do is have a huge list of behavioural statements and then ask managers to just score those that apply to their team. If you were to do this, you would actually be adding back in the very friction and complexity that we were running into at the start. Not only will you need to define the behavioural statements for the organisation, you will also need to build a system that allows you to send just the right statements to each line manager.

As with all observation, you’ll also need to take into account that you’re not receiving a perfect account of the facts, and instead what will be reported is each line manager’s impression of frequency. Bernardin and Kane (1980) argued that recall of behaviour in this format is closer to a judgement built from a general impression of the person, and Murphy and Balzer (1986) found that memory-based behaviour ratings are systematically distorted in ways that reduce their accuracy. As always, observations are guiding insight, not absolute fact, and I would always caution against rapid and radical change underpinned only by line manager observation.

No approach to observation is perfect. Behavioural observation scales are highly effective at rolling out observations in a very quick and practical manner at scale, but when used alone, severely limit the depth of insight and, to some extent, the accuracy of insight. If you’re wanting to roll out a line manager observation programme, I would strongly suggest considering the blend of techniques that is right for your organisation and the level and accuracy of insight you’re looking to achieve.

Subscribe now

Leave a comment

References

Bernardin, H.J. and Kane, J.S. (1980) ‘A second look at behavioral observation scales’, Personnel Psychology, 33(4), pp. 809-814. (PAID)

Latham, G.P. and Wexley, K.N. (1977) ‘Behavioral observation scales for performance appraisal purposes’, Personnel Psychology, 30(2), pp. 255-268.

Murphy, K.R. and Balzer, W.K. (1986) ‘Systematic distortions in memory-based behavior ratings and performance evaluations: consequences for rating accuracy’, Journal of Applied Psychology, 71(1), pp. 39-44. (PAID)

The criticism tends to come from two camps.

The first is made up of people who have spent a lot of time going to conferences and have realised they aren’t getting much from them. The second is made up of people trying to sell you access to their own conference by slagging everyone else’s off, and their reasons for doing that are obvious enough that I don’t want to spend much time on them here.

The first camp is worth discussing. There are a lot of conferences available in our industry, and a lot in other industries that we might be able to learn from as well, and to say, or to suggest, that big conferences are inherently not viable, or don’t work in terms of making us better at what we do, is an oversimplification stretched to the point where it stops being accurate. That’s not to say they’re right for everyone, or right for every problem.

Get 7 day free trial

What I’ve gained

At different points in my career I have gained varying amounts from different types of event. When I started, I gained a tremendous amount from attending big conferences such as Learning Technologies; I had no map of the profession, no sense of who was doing what or why, and no way of telling which of my assumptions about the job were widely shared and which were quirks of the organisation I happened to be sitting in. Two days and a large exhibition hall gave me all of that at once.

As time has gone on, those same events have provided me with less and less value, whereas smaller, niche and specific events have become of increasing value to me as I have found the areas in which I want to deliberately focus and develop my practice.

Variation between events

I’ve also found that not all big events, and not all small events, are uniform.

• Some events are more commercially focused.

• Some are more practice-focused.

• Some focus on getting as many people as possible in the room.

• Some focus on having the right people in the room for a discussion.

• Some focus on matching format to speaker and to audience.

• Others run a set format that you buy into by attending.

None of this is right or wrong, though it may be more or less right or wrong for you at a given time. It’s personal and temporal as opposed to absolute.

The isolation problem

One thing that stuck in my mind while I was thinking about this is that I’m increasingly having conversations with people in our industry who feel incredibly isolated: people working in-house, people who are freelance, people who run businesses, people who work remotely full-time. I speak to people who are in the office often, or even full-time, and who still feel isolated because what they do is so specific compared to everyone else around them; the rest of their team is counting down the days until retirement while they’re trying to press ahead in their career, and finding little support for that within the organisation.

We know that isolation is not good for the way in which we think, and it isn’t good in terms of progressing our practice. Perceived social isolation is associated with poorer cognitive performance and with a confirmatory bias in social thinking that protects us in the moment and works against us over time (Cacioppo and Hawkley, 2009), and workplace loneliness has been linked to lower job performance (Ozcelik and Barsade, 2018). The World Health Organization now treats social connection as a public health priority, having found that around one in six people worldwide experience loneliness (WHO, 2025).

We stop being exposed to new ideas, new connections, and new approaches, and we inherently become institutionalised. We fall into the pattern of thinking that if I think it and everyone around me agrees, it must be right, and this must be the best way forwards. We stop looking to improve. Opinion and behaviour are more homogeneous within groups than between them, which is why the people who connect across groups tend to have sight of options that nobody sitting inside a single group is likely to see (Burt, 2004).

These situations can also have significant impacts on mental health and on our ability to focus more generally.

Whether we are introverted or extroverted, human beings are still fundamentally social beings. We benefit from being around others, especially when we can engage in a constructive and meaningful way to address a challenge, and a well-run event is one of the few ways of arranging that on purpose.

My bias

I say all of this as someone who is incredibly biased. I run events. They are not right for everyone, and I will be the first to say that the Evidence-Informed Practice Conference is not for someone who spends every moment of their professional life developing e-learning and has no interest in the science and research behind how people learn, think, and perform. That particular person might be much better served by a trip to DevLearn, or to an a11y event focused on digital accessibility.

Choosing carefully

I’m not arguing that all conferences are amazing, or that anyone who says they aren’t getting anything from them is wrong. At a time when budgets are tight, time is tight, and effort is hard to justify, the argument is that you should choose very carefully what you attend.

Even if you have had negative experiences attending big conferences, small conferences, roundtables, or whatever else, don’t shut yourself off from the entire event space. Ahead of booking or agreeing to go to anything, ask yourself two questions:

• What elements of my practice do I want to focus on?

• What do I want to deliberately improve over the coming twelve months?

Once you know the answer to that, then and only then should you start thinking about:

• Which events in the calendar could support me on that journey?

• Who’s speaking at them?

• What format will it be in?

• What points will they be addressing?

• How will I be able to engage with this?

• Is it going to answer the specific questions I have about getting better in this field?

Then go to those events, and let go of the ones that everyone says you must go to, or that are such staples in the calendar that you feel you’re somehow missing out by not being there, even though every time you attend you walk away thinking what a waste of time that was.

At the end of the day, it isn’t always about the event. It may be about your decisions.

Leave a comment

References

Burt, R.S. (2004) ‘Structural holes and good ideas’, American Journal of Sociology, 110(2), pp. 349-399.

Cacioppo, J.T. and Hawkley, L.C. (2009) ‘Perceived social isolation and cognition’, Trends in Cognitive Sciences, 13(10), pp. 447-454.

Ozcelik, H. and Barsade, S.G. (2018) ‘No employee an island: workplace loneliness and job performance’, Academy of Management Journal, 61(6), pp. 2343-2366.

World Health Organization (2025) From loneliness to social connection: charting a path to healthier societies. Geneva: WHO.

That conversation took me back to role plays, partly because I hear the same two lines all the time:

“My employee’s hate role plays,”

and

“Role plays don’t work.”

In truth, I think the two are related because role plays definitely can work, but most that I look at are designed terribly, thus people tend to hate them.

Behaviour modelling training, which is the structure of describe the skill, demonstrate it, practise it, give feedback, and support the transfer back to work, produces skill gains of a little over one standard deviation compared with untrained groups (Taylor, Russ-Eft and Chan, 2005). This means, the average trained participant came out ahead of roughly 84% of those who weren’t. That structure is everything a training role play ought to be.

This is training like any other, so everything we would normally consider still applies. That said, there are three specific areas that I regularly see needing considerable improvement when I work with teams on the design of role play activities. The rest of this article will focus on these, the evidence behind them, and how we can start applying them in our work.

Practice conditions

I was once responsible for training a large group of contact centre agents, and the training they were being given had them practise the system. They filled out the form, clicked the buttons, and did it again, and again, until they could complete the process without hesitating. What they never did was complete that process while a person on the other end of a phone was upset with them, which was the only condition under which the task would ever be performed, since customers do not ring a contact centre because things are going well. Communicating, working out what the customer needed, explaining what could and couldn’t be done, and getting somebody through the process was the job. The form was the easy part, and the form was all they’d practised.

Note: excessive focus on policy and process is something I still see regularly in workplaces, and I understand why. These things are what internal SMEs know well and care most about. They are the things that they themselves have slaved over designing, implementing, and getting internal agreement upon. It is essential that we do our job of remembering that both policy and process will be enacted by employees within a wider context. It will almost always include some kind of external stakeholder who really couldn’t care less about internal policy and process and instead would just like their problem resolved. As always, environmental considerations in the real-world application of a task will trump internal or individual considerations.

Memory serves us best when the practice resembles the real work (Morris, Bransford and Franks, 1977), though reading that as a call for realism in the decorative sense is a mistake. Physical resemblance to the workplace does not drive learning outcomes; what matters is whether the practice demands the same functions of the person as the real task (Hamstra et al., 2014). Matching the physical surroundings does little on its own: a review of the environmental context research put the average effect at d = 0.28, which is small enough that you would struggle to see it in real-world performance, and a well known study of divers recalling words underwater failed to replicate when it was repeated with better controls (Smith and Vela, 2001; Murre, 2021). So the good news is that you don’t need to build a replica call centre to train your agents.

What does need replicating is the demand, the pressure, the stress that is inherent in the completion of any workplace task. Now, that’s not to say you should stand there screaming at your trainees to create a sense of real-world stress. Where the real task is performed under emotional pressure, graduated exposure to that pressure during practice reduces anxiety and improves performance under stress (Saunders et al., 1996).

I like to create a list of every demand present on the employee when the specific task being trained is being performed in the workplace. This should include emotional and interpersonal demands, and then we need to make sure these are present in the role play that we design.

Note: The research underpinning graduated exposure to stress comes from the medical and military worlds, and there has been no clear generalising of these findings to other industries, so far as I am aware. That’s not to say they can’t be generalised, but given the lack of evidence, it’s worth considering this a steer as opposed to something we know for a fact.

When creating graduated stress or demand, it is worth being very clear with employees about what you are doing. You do not want them to misunderstand the deliberate and gradual increase of pressure within the activity as you becoming frustrated with them. Open and clear expectation setting about what the role-play is and isn’t and why they will experience this graduated stress is essential for the employee to trust you as their trainer. Without trust, role-plays do not work.

Appropriate difficulty

Role plays can be awkward, we all know that. Sometimes this can lead us to be kind, to make things a little bit easier, to help the person out with things that never happen when they are performing the task in the workplace. This is the way in which trainers can fundamentally undermine the entire activity whilst trying to be kind.

Performance during training is an unreliable guide to whether learning has happened, and the conditions that produce the fastest visible improvement in the room tend to be the ones that produce the weakest retention and transfer afterwards (Bjork and Bjork, 2011; Soderstrom and Bjork, 2015). We also know that when the answer is available while you study, you systematically overestimate what you’ll be able to produce later without it (Koriat and Bjork, 2005).

A role play in which the trainee follows a script, has a compliant partner, or a facilitator ready to rescue them at a moments notice, isn’t going to do them much good. In fact, the people least well placed to notice their own limited understanding or ability, are the ones who most need to. In a study of physicians, Davis et al., (2006) compared twenty self-assessments against measured competence, and found that accuracy was poorest among those who were least skilled and most confident.

Note: The Davis Study looked at physicians, a pretty specific group of people, though nothing suggests the rest of us are calibrated any better. Similar findings have been replicated, though it’s important to note that all studies looking at this tend to consider relatively small sample sizes.

When I talk about difficulty here, I mean allowing people to get it wrong. Delivering a role play that actively encourages errors and exploration, rather than steering people around them. This approach produced an overall effect of d = 0.44, and around d = 0.80 on tasks that were novel and complex rather than close copies of what had been practised for Keith and Frese (2008).

Effects of that size means the average person who trained this way outperformed about 79% of those who didn’t when the task in front of them was unfamiliar. This is exactly the goal of a training role play.

So the partner pushes back, and the person fails at least once while the failure is still low risk and safe for the trainee and the business. Sadly, there’s no perfect piece of research telling us exactly how hard is hard enough and where the line is between this and unreasonable difficulty. That remains a judgement call only you can make regarding your specific audience and your organisational expectations.

Whole tasks

going back to my early example of the contact centre, we spent a lot of time practising the use of the computer system with nothing else in mind. Applying the process that everyone had learned to the computer system again and again and again until it was practically muscle memory.

In other words, we were letting people set aside parts of the task in order to concentrate on one, when the job will never offer them that courtesy. Handling the objection, operating the system, and keeping the record straight all happen at once in reality, and a role play that isolates one of them is a poor practice scenario for the real world.

Isolating elements is a legitimate technique with good evidence behind it. Novices can be overwhelmed by tasks with multiple interacting parts, because holding all of those parts in working memory at once exceeds what the person can manage before the underlying skills are habitual.

Here, we can draw on the four-component instructional design model (van Merriënboer and Kirschner, 2018) by using part-task practice as a supplement to whole-task work. By this, I mean we can initially simplify to part of the task, but must build towards practising the whole task in its entirety with real-world conditions before we consider the individual trained. This process will involve creating a sequence of classified tasks, building in complexity and reducing in obfuscation from the real-world delivery criteria. The trainee will then move from the simplest to most complex class as they develop mastery.

There is a significant weakness to this approach that we do need to consider, which is that this level of help, whilst fantastic for a novice, can become a significant hindrance to somebody who has already mastered this process once and is merely in need of a refresher (which we are somewhat addicted to providing in the modern workplace). We’ve discussed the expertise reversal effect (Kalyuga et al., 2003) in several previous articles, so I won’t go into it in great detail. The important thing is to consider the fact that this level of scaffolding and support is not appropriate for highly experienced members of staff who do actually know what they’re doing and to whom you are merely refreshing or offering optional additional support. It is therefore worth having practice scenarios that have all this scaffolding stripped away, ready to go for this specific audience.

The debrief

when it comes to the effectiveness of role plays, I found that nothing has a single more significant impact than the quality of the debrief. This is your opportunity to provide feedback to the trainee and highlight what they are getting right and where they still need to develop.

Structured debriefs improve subsequent performance by around 20 to 25 per cent (Tannenbaum and Cerasoli, 2013). That doesn’t mean there’s a single magic format that you must always follow, but that a consistent and considered approach is required to the debrief.

That said, feedback is not automatically helpful. In a study by Kluger and DeNisi (1996), 600 effect sizes were considered. Overall, feedback did improve performance, but in over a third of the interventions studied, it actually made performance worse. The worst performance damage was concentrated around feedback that turned the participants’ attention towards themselves rather than towards the work. In other words, when it got personal.

It’s important then that when we’re providing feedback in the debrief following a role play, we keep the conversation focused on what happened and what to do differently, and keep it away from character and personality.

Role plays then can be highly effective, impactful, and efficient training tools, but they don’t have to be. It’s not automatic. Like any other approach to training, role-plays require careful design processes, but when we get it right, they are a phenomenal opportunity for low-stakes safe practice, especially when we’re expecting trainees to dive into completing complex multifaceted tasks.

Subscribe now

Leave a comment

References

Bjork, E.L. and Bjork, R.A. (2011) ‘Making things hard on yourself, but in a good way: creating desirable difficulties to enhance learning’, in Gernsbacher, M.A., Pew, R.W., Hough, L.M. and Pomerantz, J.R. (eds.) Psychology and the Real World. New York: Worth Publishers, pp. 56-64.

Davis, D.A., Mazmanian, P.E., Fordis, M., Van Harrison, R., Thorpe, K.E. and Perrier, L. (2006) ‘Accuracy of physician self-assessment compared with observed measures of competence: a systematic review’, JAMA, 296(9), pp. 1094-1102.

Hamstra, S.J., Brydges, R., Hatala, R., Zendejas, B. and Cook, D.A. (2014) ‘Reconsidering fidelity in simulation-based training’, Academic Medicine, 89(3), pp. 387-392.

Kalyuga, S., Ayres, P., Chandler, P. and Sweller, J. (2003) ‘The expertise reversal effect’, Educational Psychologist, 38(1), pp. 23-31.

Keith, N. and Frese, M. (2008) ‘Effectiveness of error management training: a meta-analysis’, Journal of Applied Psychology, 93(1), pp. 59-69.

Kluger, A.N. and DeNisi, A. (1996) ‘The effects of feedback interventions on performance: a historical review, a meta-analysis, and a preliminary feedback intervention theory’, Psychological Bulletin, 119(2), pp. 254-284.

Koriat, A. and Bjork, R.A. (2005) ‘Illusions of competence in monitoring one’s knowledge during study’, Journal of Experimental Psychology: Learning, Memory, and Cognition, 31(2), pp. 187-194.

Morris, C.D., Bransford, J.D. and Franks, J.J. (1977) ‘Levels of processing versus transfer appropriate processing’, Journal of Verbal Learning and Verbal Behavior, 16(5), pp. 519-533.

Murre, J.M.J. (2021) ‘The Godden and Baddeley (1975) experiment on context-dependent memory on land and underwater: a replication’, Royal Society Open Science, 8(11), 200724.

Saunders, T., Driskell, J.E., Johnston, J.H. and Salas, E. (1996) ‘The effect of stress inoculation training on anxiety and performance’, Journal of Occupational Health Psychology, 1(2), pp. 170-186.

Smith, S.M. and Vela, E. (2001) ‘Environmental context-dependent memory: a review and meta-analysis’, Psychonomic Bulletin & Review, 8(2), pp. 203-220.

Soderstrom, N.C. and Bjork, R.A. (2015) ‘Learning versus performance: an integrative review’, Perspectives on Psychological Science, 10(2), pp. 176-199.

Tannenbaum, S.I. and Cerasoli, C.P. (2013) ‘Do team and individual debriefs enhance performance? A meta-analysis’, Human Factors, 55(1), pp. 231-245. (PAID)

Taylor, P.J., Russ-Eft, D.F. and Chan, D.W.L. (2005) ‘A meta-analytic review of behavior modeling training’, Journal of Applied Psychology, 90(4), pp. 692-709.

Van Merriënboer, J.J.G. and Kirschner, P.A. (2018) Ten Steps to Complex Learning: A Systematic Approach to Four-Component Instructional Design. 3rd edn. New York: Routledge.

I won’t lie, this wasn’t a problem I was expecting to have when I started writing, largely because this is something we tackle all the time when creating training interventions. We work hard as people professionals to avoid adding seductive detail to our work, removing this is one of the most foundational things we do when working with SMEs and operational leaders.

What fascinated me was that, finding myself effectively in the role of both SME and content creator, I was now responsible for the addition of an immense amount of seductive detail that I then had to remove. It turns out that by filling both roles, I was not doing a good job. In hindsight, this seems really obvious, as I’ve often, when asked, told people that it’s incredibly dangerous for L&D professionals to act as both internal SMEs and the person designing some kind of intervention. We need the tension between the two in order to do our best work, and yet, because I was not designing a training intervention, it hadn’t even occurred to me that the same challenge would present itself.

Eventually, I realised that the questions I needed to be asking myself were exactly the same questions I ask myself when looking at content that’s been written for a training intervention:

• What here is going to help someone do the job better?

• What delivers on the intent?

• What helps the end user?

• What is simply an interesting addition in the eyes of the author or creator?

In other words, is that interesting or useful?

Now, to be clear, things can be both interesting and useful, but to my mind, if your goal is to help people be able to do something, we should prioritise the useful. If it happens to be interesting, that’s a bonus.

The more I think about this, the more I reflect on some of the training that I’ve seen and experienced over the years, and how it’s full of interesting tidbits. I remember working in one organisation that was obsessed with having little pop-up bubbles of “did you know” facts about the company. None of them really necessary, none of them performance-focused, none of them ever really considering whether or not the end user needed this information at all. But hey, who doesn’t want some interesting facts about their favourite place in the world? Work!

The last thing that’s really highlighted to me is that no matter how long you’ve been doing this stuff, there will always be new contexts that, for whatever reason, you don’t attach your standardised best practice to. Definitely something to be aware of in our own practice, but also something to bear in mind about the people we’re here to support. When their context and environment changes, we cannot assume that their established habits and behavioural norms will transfer to the new world they find themselves working in.

I’ll hear things like, “There was an incident,” and that incident suggests that people don’t appreciate why the policy exists. We need something to make them more aware. Everyone nods, and the request is logged. Design work begins on how we can parse this incredibly technically dense information and get people to care about their 132nd policy of the month. As of yet, no one has described anything that I or a line manager or another employee could actually observe being done in the job.

Get 7 day free trial

Let’s get specific. A performance need is a gap between what a defined group of people are doing at work and what the organisation requires them to do.

A properly defined performance need should give us five things:

1. Who: a specific population identified by role and task rather than all employees or the operations team.

2. What they currently do: described as behaviour or output that somebody has observed or measured, not inferred from a single incident or made a leap of faith from simply thinking we haven’t got training on this, so we must need some.

3. What they need to do instead: described as a stated standard so that two people watching the same piece of work would agree on whether that standard had been met.

4. What the gap is costing the business in rework or error rates, cycle time, customer complaints, regulatory exposure, lost revenue, or safety incidents: something the business already cares about and is actively losing due to this current situation.

5. Why the gap exists: this should be investigated rather than presumed. A root cause analysis may feel old-fashioned, but it is essential for a completed performance need. If we make presumptions here or fail to investigate or take a surface-level answer, everything we do will be built on metaphorical quicksand.

You’ll notice that by having all five of those, we will naturally avoid things like greater awareness and better understanding of the policy. There is no way to observe these things, no standard to hold them against, no number that moves when they are achieved, and no examination of whether awareness was ever actually a constraint.

The person who breached the policy may have understood it perfectly and worked around it because the compliant route took longer than they had to complete the job. Real performance needs are inherently complex, fraught with environmental considerations and deeply human interactions that are not as simple as “Do you know what the policy is?”

One important thing to consider when building a performance need is the fact that there will rarely be any one person that can give you all the information you need. You will need to collaborate cross-functionally, not just with the project sponsor, or the designated SMEs, or even the end users. You will need to speak to people at all levels of the organisation and throughout the entire system that the workflow or process you are looking at touches. You’ll likely need to speak to:

• senior and middle management

• line managers

• employees doing the work

• support teams like IT and HR

• operations and logistics professionals

I tend to find that by the time I have fully defined a performance need, I will have spoken to 10, 12, maybe even 15 different types of employees within the organisation. It is exactly this cross-functional view of the situation that enables me to properly understand what is happening, what should be happening, and what’s causing the gap. Only once I have that information am I able to start asking the question: how do we close the gap?

This is not a complete view of defining performance problems. For that, I suggest looking at the work of Gilbert and Mager. I’ll include some links at the end of this article.

I think probably the most important consideration here is that when you receive a training request, remember that it is a hypothesis about a cause offered by someone who has usually not been asked to examine it. It is therefore coming to you with imperfect, incomplete insight in asking you to fix the problem.

Your job is to take what they give you, map it to the considerations I have spoken about here, figure out what information you don’t have, and then go and get it. That doesn’t have to be hugely time-consuming. It doesn’t have to look and feel like analysis paralysis. A great, well-defined performance need is not bounded by the amount of time it takes to create, but by the effort you put in gathering the breadth of views required to inform it.

Leave a comment

Fore more on defining performance needs

Gilbert, T.F. (1978) Human competence: engineering worthy performance. New York: McGraw-Hill.

Mager, R.F. and Pipe, P. (1997) Analyzing performance problems: or, you really oughta wanna. 3rd edn. Atlanta: CEP Press.

Get 7 day free trial

To me, unstructured learning is the stuff that happens without anybody putting any kind of framework around it. Somebody turns to a colleague next to them and asks how to handle a customer complaint or where the report template lives. They get shown and carry on with their day. Nobody schedules anything, nobody designs an experience, and nobody records that this ever happened. That is social learning in its most unstructured form, and it happens all the time, which is exactly why we can’t see most of it.

I’m a strong advocate for line manager observation, and even with a manager paying close attention, the majority of these moments will pass unrecorded, and that’s absolutely fine. There’s not actually, I don’t think, a tremendous value in a list of just how many times a colleague helped someone else out that week.

What we should, and I think in many organisations are doing, is work on the conditions: whether people feel able to ask, whether the person being asked has the time and the standing to help, and whether the organisation treats this has a distraction or interruption to work or part of doing the job well.

Those environmental considerations, in many ways, are a form of structure. Yes, they’re more cultural in nature than a strict L&D-driven framework, but it shapes what happens all of the time.

In contrast, structured learning is anything we have given shape to: a time, a format, a set of participants, a question to answer, an output to produce, or some combination of all these things. The shape may be light. When we say a piece of social learning is unstructured because we opened a space and then stepped back, the space is the structure. We chose the channel and decided who could see it. Set the tone for the first few posts and put someone in place to keep it alive. Anything we are involved in and anything we can report on has some level of structure to it, because otherwise we’d not have any way of knowing what was happening.

All of this means, to my mind, that anything that is social learning within our broader L&D strategy is inherently structured.

Lunch and learns, structured.

Communities of practice, structured.

User-generated content programmes, structured.

Peer review sessions, mentoring schemes, and knowledge sharing threads, structured.

My favourite of all structured social learning, of course, are round tables. We love attending them as L&D professionals. HR are in the same boat here, and we always come away excited about things that we picked up from our peers, whether it’s pages of notes or things we want to tell colleagues about when we get back to the office. Yet how often do you run them in your own organisation?

I’ve found that by bringing together subject matter experts, practitioners, leaders, and stakeholders, we can rapidly define problems, ideate solutions, decide on a path forwards, and agree on next steps to start early-stage experimentation and piloting much more quickly than through a series of meetings, even if we do label them agile.

To go right back to the start of all this: social learning can be unstructured, and plenty of it is, but everything we’re involved in and have ongoing visibility of is structured. All of the social learning in our strategy, the area we have control over, is structured, and that’s okay, but it’s therefore incredibly important to understand that unstructured learning and social learning are not and have never been the same thing.

Share Instructional Design Tips

Leave a comment

First, a stable internal language. If a skill, a capability, a competency, and a behaviour mean four different things depending on which team wrote the document, then every conversation starts with ten minutes of translation. Agreeing what we call things inside our own organisation, and then using those names consistently, just ensures people know what the heck we’re talking about, and we get to stop relitigating vocabulary when we should be discussing substance.

Next up, what we name our initiatives. When we put something out into a workplace, especially something larger where long-term uptake and continued engagement are required, the name is an exercise in internal marketing. It’s the earliest and cheapest opportunity we have to communicate what something is and what its potential value might be. We’re not creating blockbusters, we’re not seeking Oscars, and we’re not putting up billboards unless there’s some serious budget on hand; the name, perhaps a thumbnail, perhaps a one-line description, is often everything a person sees before they either scroll past an optional intervention or engage with it.

In a couple of my recent projects, we’ve spent a fair amount of that time thinking about what each phase of a learning journey is called, and whether the name instils in them the core point of what they’re doing at that stage. The test is whether the purpose feels obvious and natural from the title alone. None of this means great development can’t happen under a poor name, because it can and regularly does; but when the goal is getting people to take that first step of engaging when they have every option not to, good titling can be massively important.

Thinking about this in my recent work has led me somewhere closer to home. This Substack, and the YouTube channel alongside it, runs under the name Instructional Design Tips, and it made sense when we started; but the conversations we have here have expanded somewhat and now take in a lot more than just instructional design. These articles now range across the whole of L&D, into HR, performance improvement, organisational design, and leadership, and by my own argument above, the title is no longer communicating what this thing is or the value it might hold.

So I’m considering changing the name, and I’d love your input. There’s a poll below with some options; if you think they all suck (which is entirely possible) and you have a better idea, please do leave it in the comments. I’m not definitely going to change anything right now, but the more I think about it, the more I like the idea. I figured, hey, who better to ask than you? Not you individually, obviously, that would be weird, but the collective you, who somehow set aside some time to read this.

Get 7 day free trial

I expected the difficult part to be systems training, or perhaps some mindset work around AI. It was neither. The difficult part was translation: taking a tool built by data scientists and developers who had been immersed in the numbers, the insight, and the fine scrutiny of accuracy, and turning all of that into something people on the ground could understand and adopt within their existing workflow.

Whilst everything else about the project was new to me, to the team, and to the rest of the business, this part was the work people professionals were doing decades ago, and I expect it will be the work we’re doing decades from now. Simplifying complex technical detail into meaningful insight, into prompts that change real-world behaviour, without weighing anyone down with detail they don’t need, is an inherently human process; no amount of new technology changes it.

Having watched that translation work make or break adoption, I now wonder how much of our current enthusiasm for AI literacy is damaging the adoption it’s supposed to support. Are we loading people with insight that feels useful, that feels like it will serve them at some point, when they have no use for it today? I’ve seen organisations roll out external training covering several different LLMs, then tell their people, “Obviously, you don’t have access to any of these, but it’s important that you’re aware.” Aware of what, and in service of which task?

Everything about this technology is new; the discipline of working out what people need to know today, and the restraint to leave out everything else, is one of the oldest skills our profession possesses. We would do well to trust it.

Leave a comment

Below are the questions I think they should have asked.

Get 7 day free trial

Which decision will this improve, and who makes that decision today?

Be as specific as you can: who gets deployed onto a project, where to open a role, whether to build or buy a capability, who’s ready for the next level. Visibility may sound great, but an inventory of skills present in your business will not inherently improve anything. You need to know who is making the decisions, what the decisions are, and how the outputs of your new tool are going to help them do it.

Where does the data come from, and how is it kept up to date?

Self-assessment is cheap to collect and varies wildly between teams and between individuals with different levels of confidence; manager assessment is more consistent and costs a great deal of management time; inference from work systems is the most promising route and depends entirely on the quality of your integrations. Whichever mix you choose, ensure there is a clear, ongoing and well-supported plan to keep the data up to date. Skills profiles are incredibly flexible and can change radically in a relatively short period of time.

What happens to it when the operating model changes?

Roles get redesigned, functions merge, and a taxonomy built tightly around today’s structure we’ll inevitably go out of date. It’s essential that whatever tool you use is able to quickly and easily adapt when the organisation makes small or more significant changes to the operating model, including team structure and function, as well as the skills that the tool is trying to track.

What will you do about the gaps it finds?

Any skill system worth its salt is going to find gaps. That’s what it’s there for, so you need to make sure there is budget, capacity, and authority to fix those gaps when they are found. It’s no good waiting to know what they are and then trying to get the business to take this seriously. These tools are only truly valuable if there is agreement in place for a response when skill gaps are identified. Treat these tools as something to help you target initiatives rather than data points you can use to argue for the initiative to be funded in the first place.

How will you know in twelve months whether it was worth it?

Understand and set clear, organizationally linked objectives so that, twelve months after launching your skills platform, you can demonstrate clear organisational value. Any good supplier will help you do this, as it is part of good platform adoption.

Your supplier should be happy to engage with you on all five of these questions, even when their tool cannot help solve the challenge. Whatever skills platform you’re rolling out, I would always advise starting with a single business-critical function. Not only does it cost a fraction of an enterprise rollout, but it can also help you answer any of the unknowns surrounding these five vital questions. This also gives you the opportunity to run a pilot demonstrating value on a small scale before scaling your platform across the entire organisation.

Leave a comment

This is easy to agree with in principle, but once you’re sat at a desk with hundreds and possibly even thousands of observations coming in, you quickly start to realise that you are going to need a data analysis plan. In fact, you should probably have thought of one before the observations started pouring in.

Technology can help with this, whether that’s AI-assisted analysis or a well-built spreadsheet or Power BI dashboard, but only if we know what we’re looking for before.

I start with a hypothesis about success:

For a sales programme, that might be objection handling appearing in the right part of the conversation; for a compliance-heavy role, it might be GDPR questions being asked at the point of data capture rather than at the end of the call.

The second decision is how the observation gets recorded. Asking a line manager to write freely about what they saw gives you nuance, but it also signs you up for a lot of analysis. Giving them a set of structured questions, with tick boxes or a list to select from, lets you compare what you expected to see against what the data says very quickly.

Most of the teams I’ve worked with land on a combination: structured questions covering the behaviours they’ve predicted, plus an open text question for anything else the manager wants to include in their observation.

Some of the most effective teams I’ve worked with have set up small bespoke platforms, or used something like Power BI, so that managers can record an observation in ninety seconds or less, and the analysis updates the moment they submit it. This approach keeps the outputs relatively simple:

• What’s coming through positively?

• What’s coming through negatively?

• Which of your predicted themes are appearing and at what frequency?

• Which themes are turning up repeatedly that nobody predicted at all?

That last question is a great way to start building a performance backlog, as we have discussed before. A list of known performance issues within the organisation that you can proactively approach the organisation to support, rather than waiting for them to become significant organisational issues being brought to your door.

When it comes to reporting, we don’t want to be including specific observations like testimonials and instead should look to report the findings of our data analysis:

• What are the themes?

• What are the reoccurring observations?

• What are the opportunities for further improvement?

Observation shouldn’t stand alone either. It sits near the bottom of the metric chain, telling us whether the intervention landed on individual behaviour, which then licenses the next question: is that behaviour moving the performance KPI we said it would? Skip the observation layer and we’re assuming the behaviour is there, which weakens every claim we make about the numbers further up the chain.

Leave a comment

Get 7 day free trial

Choosing which decisions matter

Every process contains far more decisions than any training experience could test, and most of them shouldn’t be tested. When I’m weighing up whether a moment deserves to become a decision point, I ask three things:

1. Does a poor choice carry a meaningful consequence in the real work?

2. Is there a plausible chance a competent person would make that poor choice?

3. Does choosing well require judgement rather than the recall of a rule?

If the wrong option costs nothing, if nobody would realistically pick it, or if the right answer is a matter of looking something up, the decision doesn’t need practising as a priority. Tell people what good looks like, explain why, and move on.

Decision points are expensive in design time and in learner attention, so they should be reserved for the moments where judgement under realistic conditions is the thing being developed.

Building the moment itself

Whether the experience is a piece of e-learning or a facilitated workshop, the components of a working decision point are the same.

The cues that would trigger the decision in real work, because recognising that a decision is required is often harder than making it.

The options need to be plausible, which means drawing the wrong answers from mistakes people make in practice rather than from a designer’s imagination.

The consequence needs to be visible, so the learner sees what happens as a result of their choice rather than a red cross and a ‘try again’.

The feedback needs to explain why, addressing the reasoning behind the choice. Feedback that works at the level of the task and the employee’s processing of it is what moves performance (Hattie and Timperley, 2007).

The environment needs to make error safe; people take the interpersonal risk of being wrong when they believe it won’t be held against them (Edmondson, 1999).

Realistic difficulty; conditions that make practice harder can improve long-term retention and transfer because they demand retrieval and judgement rather than recognition, provided the difficulty is one the learner can work through (Bjork and Bjork, 2011).

None of this requires a branching scenario. A consequence can be a single beat:

• the employee chooses,

• the outcome plays out,

• the reasoning is explained,

• the experience rejoins the main path.

“That’s incorrect, and here’s what would have happened” is a consequence, and for many decisions it’s the right amount of consequence. Branching, where an early poor choice shapes everything that follows and the user has to find their way back from it, is a an expensive design choice, but can be effective when this level of detail is required.

Note: In my experience, the vast majority of decision points should be kept simple. Sprawling branching scenarios are highly effective when dealing with very complex, nuanced situations that can be successfully dealt with in different ways. However, this approach to training is costly, both in terms of your time and resource, and to the end user in terms of the amount of time they’re in training and the amount of energy you’re asking for from them. Always consider, even if you’ve created a wildly effective learning experience, if you are fulfilling your role as a servant to the wider organisation when it comes to the efficient use of time and resource.

Novices and experienced performers need different failure

Allowing people to struggle with a problem before instruction can produce deeper learning, but the benefit depends on said people having enough prior knowledge to generate ideas and interpret their own errors; without that foundation, failure is unproductive rather than productive (Kapur, 2008; Kapur, 2016).

The guidance that helps a novice can also hinder an experienced performer, and the reverse (Kalyuga et al., 2003). So for novices, I keep the loops short: decision, consequence, explanation, retry, with the safety net close to the ground.

For experienced employees, longer arcs really pay off; letting a plausible early misjudgement play out, and asking them to recognise it and recover, mirrors what their work asks of them anyway. Recovering well from a poor decision is a skill in its own right, and it’s one we rarely give people anywhere safe to practise.

The decisions worth building into a training experience are the ones where competent people plausibly get it wrong and it matters when they do; everything else is information, and should be delivered as such.

Leave a comment

References

Bjork, E.L. and Bjork, R.A. (2011) ‘Making things hard on yourself, but in a good way: Creating desirable difficulties to enhance learning’, in Gernsbacher, M.A., Pew, R.W., Hough, L.M. and Pomerantz, J.R. (eds.) Psychology and the real world: Essays illustrating fundamental contributions to society. New York: Worth Publishers, pp. 56-64.

Edmondson, A. (1999) ‘Psychological safety and learning behavior in work teams’, Administrative Science Quarterly, 44(2), pp. 350-383.

Hattie, J. and Timperley, H. (2007) ‘The power of feedback’, Review of Educational Research, 77(1), pp. 81-112.

Kalyuga, S., Ayres, P., Chandler, P. and Sweller, J. (2003) ‘The expertise reversal effect’, Educational Psychologist, 38(1), pp. 23-31.

Kapur, M. (2008) ‘Productive failure’, Cognition and Instruction, 26(3), pp. 379-424.

Kapur, M. (2016) ‘Examining productive failure, productive success, unproductive failure, and unproductive success in learning’, Educational Psychologist, 51(2), pp. 289-299.

Get 7 day free trial

Part of that comes down to the processes they’ve built, which work rather like a drive-through window; someone fills in a form when they want L&D involved, they assess it, hold a meeting, and the whole apparatus is designed around being approached rather than approaching. Nothing in the organisation obliges anyone to fill in that form, and plenty of leaders will find what they consider a faster route to change.

What’s needed is a tank on the lawn: a visible, proactive move towards where the work and the biggest organisational need sits, saying we’re here, this is what we can do, and we want to help. The tank doesn’t have to be enormous. A small one, parked in the right place, does the job.

Step one is observation. Being across performance data, organisational objectives, and whether the business is on track to meet them gives us the raw material for spotting where we could deliver value, whether that means addressing underperformance or capitalising on an opportunity that moves the organisation towards its goals more quickly or more sustainably. That’s how the performance backlog gets built: a running list of problems the business cares about and hasn’t solved.

From there:

• pick small, highly impactful projects to get involved with

• set clear goals you know you can deliver on

• build a value proposition for the organisation

New technologies have made keeping abreast of what’s happening in the business easier than it’s ever been, so the barrier to a more proactive stance is lower than most teams assume. We don’t need to go from zero to a hundred; zero to one, then one to two, will do. Gradual, prolonged change is more sustainable and more realistic than the explosive transformations described in LinkedIn posts that most likely never happened.

If we keep waiting to be asked, we forfeit the right to complain about being treated as order takers.

Leave a comment

The other half belongs to someone else.

You can build a carefully constructed practice environment, with branching scenarios, live role play, spaced retrieval, and feedback at every stage, and it will sit untouched if the people you built it for lack the time, the permission, or the reason to open it. For most employees, one person holds all three, and that person is their line manager.

Line managers decide what lands on people’s plates, where attention goes, what comes up in a one-to-one, what gets revisited a fortnight later, and what gets noticed when performance is reviewed. For most people in most roles, their line manager makes more difference to their development inside the role than any programme we design.

Supervisor support and a supportive work environment sit among the more reliable predictors of whether trained capability is observed as real workplace performance (Blume et al., 2010), and the conditions surrounding training shape transfer more heavily than much of the elements we’re absorbed in designing (Grossman and Salas, 2011).

This makes practice a cross-functional problem. We own the design; line managers own the conditions; both need deliberate attention or the whole thing falls apart.

Making the Ask

Approaching line managers about this works best as a performance conversation.

• Open on their problem. Ask what’s slowing the team down, which conversations people avoid, and what they’d fix given a spare fortnight. Practice answers some of those and is no help at all with others; be honest in what you say practice can and cannot support.

• Connect it to their objectives. Line managers are measured on team output, quality, retention, and how quickly a new starter becomes useful. Frame practice against those numbers.

• Be specific and small about the ask. “Support the programme” means nothing to someone with a full week. “Protect forty minutes a fortnight for two people, and mention it in your Monday huddle” is something a manager can agree to or decline on the spot.

• Embed conversations into existing patterns. Team meetings, daily huddles, one-to-ones, coaching conversations, performance reviews, and the weekly email are great opportunities to promote practice. Give managers the two or three sentences they need for each, so raising it costs them nothing.

• Tell them what exists and what it’s for. Managers can’t promote what they don’t know about; a one-page summary of the available practice opportunities, and what each is designed to improve enables them to promote the right thing to the right people at the right time.

• Set the follow-up expectation explicitly. If you want them noting who has practised and what they observed afterwards, say so, and keep it to something that fits into ninety seconds on a phone.

• Ask them to practise themselves. A manager who has run the scenario, found it awkward, and said so in front of the team gives that team permission to engage with and face practice challenges and discuss them openly.

Easing the Workload

Line managers are stretched in the same way everyone else is, so if you’re asking them to pick something up to support your work, you also need to look for opportunities to ease their existing workload. If we’re asking for time, we should be looking for time to hand back.

Nine times out of ten, the job a line manager is there to do is lift the performance of their team, and practice is one of the more dependable ways of doing it; the case is there for us to make, in their language, with an ask small enough that saying yes carries no risk.

Practice makes it into people’s development diets when the person who controls their diary believes it’s worth the time. Getting line managers on board with deliberate practice as part of the working week is essential if we want our initiatives to be effective in the long term.

Share Instructional Design Tips

Leave a comment

References

Blume, B. D., Ford, J. K., Baldwin, T. T., and Huang, J. L. (2010) ‘Transfer of training: A meta-analytic review’, Journal of Management, 36(4), pp. 1065-1105.

Grossman, R. and Salas, E. (2011) ‘The transfer of training: what really matters’, International Journal of Training and Development, 15(2), pp. 103-120.

What we find is a fresh set of tasks spread across the organisation: content that needs checking by someone who wasn’t consulted, outputs that turn out to be inaccurate, and analysis that has to be redone by the people who understand the data. Cutting internal and external SMEs out of content creation is the most consistent source of this, though the data analysis side is causing plenty of problems of its own, and the costs land in finances, time, opportunity, and goodwill.

I like the chaotic, ground-up approach in principle; getting hands-on with tools and scaling what holds up is how a lot of good practice starts, but in most organisations right now it’s producing waste and frustration, and licence costs are climbing fast enough to make that waste expensive.

What’s needed is the systems thinking we’d apply to any other change, so before you automate anything, work through this:

• Map the process. Who’s involved, which policies it touches, and which parts of the business own or hold a stake in it.

• Cost it as it stands. If the process is running fine on almost no effort, energy, or resource, leave it alone and find something of greater value to work on.

• Ask whether it needs to be more efficient. Optimisation for its own sake is a trap; efficiency counts only when you’re saving something worth saving.

• Ask whether AI is the right tool for that saving. Starting with the tool and hunting for a job to give it is how we got here.

• Agree what success looks like with your stakeholders, including the ones downstream, so you don’t make one person’s job easier at the expense of quality or somebody else’s workload.

• Run a short pilot over a pre-agreed period, then reconvene and check the results against what you agreed.

If you’ve found real efficiency, roll it out; if you’ve found inaccuracy, work that isn’t getting done, or a process slowed down by the checking tasks you’ve created, roll the pilot back and put your effort somewhere it can do some good.

Rolling something back at pilot stage costs a fraction of a failed full rollout, and it keeps your stakeholders willing to try the next thing; that willingness is the asset worth protecting.

Note: I was prompted to write this short article having commented on an interesting LinkedIn thread on the same topic posted by the fantastic Teresa Rose. I strongly suggest you follow her.

Perhaps unsurprisingly, given this Substack, I’m a big fan and believer in the value of writing, both in terms of developing deeper thought and synthesising ideas. It’s also an interaction that I try to work into initiatives, and I’m always amazed at how surprised organisations are when I tell them their employees will have to write something.

For this article, I thought we’d take a look at the evidence underpinning the effectiveness of writing and then take a look at five practical implementations of this approach.

What the evidence supports

The most substantial recent meta-analysis of writing-to-learn examined 56 experiments involving 6,235 students across science, social studies, and mathematics, and found an overall effect of g = 0.30 (Graham et al., 2020). In plain terms, the average person who wrote about the content outperformed roughly 61% of those who did not, which is a reasonable return on an activity that costs a pen and ten minutes. The effect held across subjects, age groups, and types of writing task, and it appeared on measures of recall, comprehension, and application of content.

An earlier meta-analysis of 48 school-based programmes found a smaller average effect of d = 0.17, alongside two other findings (Bangert-Drowns et al., 2004).

1. Writing tasks that included metacognitive prompts, asking people to examine what they understood and where they were stuck, produced larger effects.

2. Longer writing assignments produced smaller effects. Short and frequent outperforms occasional and lengthy, which is convenient given the constraints most of us are designing within.

Writing forces retrieval, so that when someone closes the deck and writes down what they can remember, they are practising the act of pulling information out of memory, which supports retention more reliably than reviewing the same material again (Karpicke and Blunt, 2011).

Writing also supports self-explanation, where people generate their own account of how something works rather than accepting the account they were handed; a meta-analysis of 69 effect sizes put self-explanation at g = 0.55, a moderate-to-large effect (Bisra et al., 2018).

Writing produces an artefact as well, which makes the thinking visible to the person who did it and, where appropriate, to a facilitator or a line manager who can respond to it.

A note on workplace evenidence

There’s only really one significant piece of workplace evidence to support this; a field experiment at a business process outsourcing company in India, in which employees during a two-week training programme either continued practising for the final fifteen minutes of each day or spent that time writing about what they had learned, with the writing group reported as scoring 22.8% higher on their final assessment (Di Stefano et al., 2016).

The paper has circulated as a working paper since 2014 under several titles and does not appear to have completed peer review in a journal. One of its four co-authors was subsequently found by Harvard Business School to have committed research misconduct in unrelated work, with four of her papers retracted; she denies the findings and is contesting them in court (Science, 2024). The reflection paper is not among the retracted ones, and no specific allegation has been made against it, so this is a reason for caution rather than dismissal.

The case for written tasks does not depend on that study. It rests on decades of classroom research and on well-replicated cognitive mechanisms, which come with far clearer instructions about what makes the difference.

The limitations

The effect sizes above are modest; a figure of g = 0.30 is a helpful nudge rather than a ‘game changer, and the smaller result from Bangert-Drowns and colleagues sits at the point where practical significance becomes debatable. Writing is a cheap intervention with a small-to-moderate payoff, which is a reasonable trade in most contexts.

Almost the entire evidence base comes from schools and universities. Graham and colleagues restricted their review to students in Grades 1 to 12, with academic content and academic tests as the outcome measure, and the same is broadly true of the self-explanation research. Applying those findings to adults learning job-relevant skills is an extrapolation; a defensible one, given how well the underlying cognitive mechanisms generalise, but an extrapolation nonetheless.

Two practical cautions follow.

1. Summarisation, the most obvious form of written task and the one most likely to be reached for, is the weakest option available. Dunlosky and colleagues (2013) rated summarisation as low utility in their review of learning techniques, largely because doing it well requires training that most people have never received; asking a room of adults to “write a summary of what we covered” produces transcription from the majority of them.

2. The physical act of handwriting is not the active ingredient. The widely repeated finding that longhand notes beat typed notes did not replicate cleanly when Morehead and colleagues (2019) reran the study with additional conditions, and the sensible reading is that what matters is whether the person reformulates the material or copies it down, which either tool permits.

There is also an accessibility question we ought to be asking before we design a written task. Written activities can disadvantage people with dyslexia, people working in a second or third language, people with motor impairments, and people who were told at school that they couldn’t write and have believed it ever since. An equivalent route should always be available, whether that’s a voice note or a spoken exchange in a pair, and the quality of the writing should never be assessed or commented on, because the writing is a vehicle for thinking and not the thing being developed.

Note: The continued improvement in the quality of assistive technologies is making the accessibility barrier to written exercises easier and easier to overcome, but is still always worth considering.

What written tasks are useful for

Writing helps where the target is comprehension or planning: consolidating a concept, working out whether someone has grasped a distinction, surfacing confusion, and converting a vague intention into something specific enough to act on. It contributes far less where the target is a physical or procedural skill, where repetition against feedback is what moves performance. It does nothing whatsoever for the environmental barriers that form the source of most workplace performance problems; nobody has ever written their way out of a broken process, an absent job aid, or a manager who penalises the behaviour the training asked for.

Five practical approaches

1. The two-minute recall. At a natural break in a session, ask everyone to close their notes, turn the handout over, and write down everything they can remember about the topic just covered. Two to three minutes is enough, and nobody needs to share what they wrote. Then put the material back on screen so people can see their own gaps, which is where most of the value sits. The same prompt sent by email a week later, with no warning and no opportunity to review first, buys you a spaced retrieval opportunity for the cost of a single message.

2. The confusion log. “Which part of this are you least confident about, and what would you need in order to be confident?” This is the metacognitive prompt that predicted stronger effects in the Bangert-Drowns review, and it does a second job by handing you live diagnostic data about where your design is failing people. Collect the responses anonymously, read them over the break, and adjust the next section accordingly.

3. Explain it to someone who wasn’t here. Ask people to write a short explanation of the concept for a colleague who missed the session, pitched at a level that particular person would understand. An instruction to summarise produces copying, whereas an instruction to explain something to another individual with prior knowledge forces the reformulation that self-explanation research supports.

4. The written if-then plan. Replace “what will you do differently?” with a fixed written format: “If [specific situation], then I will [specific action].” Implementation intentions in this form produced a medium-to-large effect on goal attainment across 94 independent tests (Gollwitzer and Sheeran, 2006). Insist on the format, and make sure it is written down before people leave the room rather than promised for later.

5. Draft an artefact. Rather than having people write about the work, have them write the work: the customer email, the case note, the risk assessment, the piece of developmental feedback, the incident summary. The writing-to-learn literature does not directly support this one; the justification is task alignment, and the fact that it gives you something to give feedback on that resembles the job. I’m including it here because I found it massively effective in my own workshops.

Making the space for it

None of these approaches requires new technology or a budget, which is part of why they are worth trying and part of why they get overlooked. What they require is a facilitator willing to sit in the silence whilst people write, and a designer willing to spend session time on something that may feel a little unusual.

Leave a comment

References

Bangert-Drowns, R. L., Hurley, M. M. and Wilkinson, B. (2004) ‘The effects of school-based writing-to-learn interventions on academic achievement: A meta-analysis’, Review of Educational Research, 74(1), pp. 29-58.

Bisra, K., Liu, Q., Nesbit, J. C., Salimi, F. and Winne, P. H. (2018) ‘Inducing self-explanation: A meta-analysis’, Educational Psychology Review, 30(3), pp. 703-725.

Di Stefano, G., Gino, F., Pisano, G. P. and Staats, B. R. (2016) ‘Making experience count: The role of reflection in individual learning’, HEC Paris Research Paper No. SPE-2016-1181 / Harvard Business School NOM Unit Working Paper No. 14-093. Working paper; not peer reviewed.

Dunlosky, J., Rawson, K. A., Marsh, E. J., Nathan, M. J. and Willingham, D. T. (2013) ‘Improving students’ learning with effective learning techniques: Promising directions from cognitive and educational psychology’, Psychological Science in the Public Interest, 14(1), pp. 4-58.

Gollwitzer, P. M. and Sheeran, P. (2006) ‘Implementation intentions and goal achievement: A meta-analysis of effects and processes’, Advances in Experimental Social Psychology, 38, pp. 69-119.

Graham, S., Kiuhara, S. A. and MacKay, M. (2020) ‘The effects of writing on learning in science, social studies, and mathematics: A meta-analysis’, Review of Educational Research, 90(2), pp. 179-226.

Karpicke, J. D. and Blunt, J. R. (2011) ‘Retrieval practice produces more learning than elaborative studying with concept mapping’, Science, 331(6018), pp. 772-775.

Morehead, K., Dunlosky, J. and Rawson, K. A. (2019) ‘How much mightier is the pen than the keyboard for note-taking? A replication and extension of Mueller and Oppenheimer (2014)’, Educational Psychology Review, 31(3), pp. 753-780.

Science (2024) ‘Honesty researcher committed research misconduct, according to newly unsealed Harvard report’, Science, 14 March.

A recent qualitative study offers an early marker of that shift (Passmore and Daly, 2026). The authors interviewed eleven senior L&D and HR leaders in large European organisations, each of whom had used an AI coaching tool within the previous twelve weeks, and analysed how those leaders evaluated the tools and reasoned about adoption. This is a study of how a small group of senior buyers perceive these tools; it is not a controlled trial of outcomes.

Read on those terms, it’s useful. The leaders still framed the core value in terms of scale and cost, with several describing the tools as effectively ‘free’; the authors sensibly note that this reflects marginal unit cost rather than the total cost of running such a system responsibly, once data protection and ongoing governance are counted. Most favoured a hybrid model, with AI handling structured, lower-stakes reflection and humans reserved for the emotionally complex work. Several were candid about the limits, describing responses that were generic and repetitive, and the absence of the emotional nuance and challenge that good coaching depends on. These are the people who chose to trial the tools in the first place, so their candour about the ceiling carries some weight.

What the study can’t tell us is whether any of this works. It doesn’t measure whether behaviour changes, or whether the reflective habit survives once the novelty fades; nor does it tell us what happens to a person’s own capacity to reflect when a machine does the scaffolding.

So here are the questions I’d want the next wave of research to answer:

• does AI-supported coaching produce durable behavioural outcomes, or pleasant conversations that feel productive without changing much?

• who does it work for, and who does it leave behind?

• what is the real cost once the governance is priced in?

Effectiveness is the harder question, and it’s the one worth asking.

Leave a comment

References

Passmore, J. and Daly, J. (2026) ‘Are AI Coaching Agents Learning Friend or Foe?: A Qualitative Study of Learning and Development Leaders’ Perceptions of AI Coaching Agents Using Unified Theory of Acceptance and Use of Technology 2’, International Journal of Training and Development, pp. 1-10.

We do a version of this already; we share our work with stakeholders inside the business and walk them through the thinking, asking what they make of it. What we do far less often is set aside time for us and our teams to critique each other’s work as peers, and I don’t mean a quick pat on the back or a “well done for hitting the deadline”; I mean an in-depth look at the output you and your colleagues have produced over a month, a quarter, or half a year, analysing it for its quality and its impact, identifying where it could be stronger, and offering suggestions for how to get it there.

These internal retrospectives are worth the time they cost. If you work in the vendor space, or as a freelancer or consultant, then building a peer group around yourself that you can do this with becomes essential, because the structure won’t exist unless you build it yourself. I get a lot from a small group I meet with once a quarter; we critique each other’s work carefully, and we hold one another to a high standard precisely because we want to push and challenge each other to keep improving. For any of this to be useful the critique has to be honest; a peer group that only reassures each other is pleasant company and little else, and the point is to come away from each session with a clearer view of what to fix.

Working out loud tends to get oversimplified to telling people what you’re doing, and there is value in that; the value is amplified, though, when the second half of the conversation is present, when you are on the receiving end of challenging feedback and advice rather than applause.

So if you’re at the start of your career, or well into it, find a way to work out loud that includes the feedback loop: share your work on social media, in online communities like the Learning Network, with your colleagues internally, or with a handful of carefully chosen peers from across the industry, and then ask them to tell you where it falls short.

Leave a comment