The old-timey baseball scouts portrayed in the movie Moneyball are gathered around a conference table assessing free agent prospects for their team. “He’s got a baseball body,” one scout asserts. “Good looking ball player. Good jaw,” adds another. But the General Manager of the team, Billy Beane, looks skeptical. He asks, “Can he hit?” A scout chimes in with confidence: “He’s got a classy swing, it’s a real clean stroke.” Another agrees: “A lot of pop coming off the bat.” Beane, growing frustrated, asks again, “If he’s a good hitter why doesn’t he hit good?”

Conversations about merit in foreign policy are often reminiscent of that scene. The grist used to form our opinions about merit often seems ambiguous and subjective. Overly-confident assertions about who is (and who is not) an expert are laid upon questionable foundations.

The concept of merit has never been more essential in foreign policy. Those who are judged to possess the most merit rise to positions of great power and influence, while those lacking merit are left behind. And ideas about merit drive the design of our institutions of foreign policy. It’s time to get clear on what merit really means, and how we identify it.

Defining Merit

The concept of merit must be anchored in an objective understanding of performance within a specified domain. The more a person is judged to have the capabilities to achieve success, the more merit they have.

A prerequisite for evaluating performance is therefore possessing a clear understanding of the goals at hand. Defining goals in a democracy is a difficult and contested process. Most modern policy environments face competing demands and priorities: Is the goal to prevent war? Defend our borders? Project American values? Fight for human rights? These are complex, highly political decisions. But if the objectives are unclear, there can be no meaningful assessment of merit. At its worst, foreign policy is the art of setting ambiguous goals so one can always claim success. But merit is only a useful concept if it captures the ability of some actor to achieve certain goals.

A healthy organization will design its personnel processes to reward the qualities most likely to lead success. The salesman may be judged on total revenue; the software engineer will be encouraged to ship error-free code, etc. In contrast, an unhealthy organization may cling to ambiguous understandings of merit. This may result in poor decisions about who to hire, who to promote, and what to train. This introduces distorting incentives. Staff may learn, for example, that the best way to get ahead is to kiss up to the most powerful official in the room.

It is worth noting that merit need not be entirely about performance. In addition to the objective qualities that comprise merit, the concept of merit may also include some more subjective facets, such as the qualities of being courageous, ethical, or friendly. For instance, an organization might decide that those who have triumphed over great obstacles to achieve their position possess more merit than those weaned with a silver spoon, even if their performance is exactly the same.

The Validation Challenge

To have a more constructive conversation about merit, it helps to understand the concept of internal validity. There are two categories: construct validity and inter-rater reliability.

Construct validity is the extent to which a metric accurately captures what it claims to measure. For instance, if a coach is trying to identify the best chess players to recruit, measuring shoe sizes will offer poor construct validity. Measuring SAT scores would offer better construct validity. Best yet, looking at applicants’ official chess rankings would offer high construct validity. But notice that the coach needs to be clear on the precise goal in recruitment: Does she care only about demonstrated skill or long-term prospects? Does it matter if the recruit is a supportive teammate?

Inter-rater reliability evaluates the uniformity between scores from different judges or measurements. If you ask five people, are you going to get the same five answers? For instance, imagine a team of magazine editors judging a poetry contest. If the question presented to the reviewers is “Which poem is best?” there will likely be wide divergence between the judges. Low inter-rater reliability doesn’t mean that none of the poems are beautiful, just that “best” isn’t a concept that’s satisfactorily measured in this way. Recognizing this challenge, the judges might agree to use more precise metrics to evaluate their contest. They may, for instance, evaluate “which poem is the most technically complicated?” or “which poem is the most original?” They could even average these two scores.

Ideally, measurements have both high construct validity and high inter-rater reliability. Then we can really trust that the identified metric measures what it claims to measure, and does so reliably.

Evaluating the Foreign Service Entrance Exam

The Department of State sometimes makes strong claims about its ability to measure merit and performance. In 2006, then-Director General of the Foreign Service George Staples was exploring some major changes to the exam. Before his decision was inked, the Department commissioned McKinsey & Company to review the evaluation process, which concluded that the exam was “the best screen, and people who do well there are going to do well in the Foreign Service.” Many at the State Department continue to parrot McKinsey’s claim that the exam represents the “gold standard” for hiring processes.

Unfortunately, the claim lacked any supporting evidence (at least that I’m aware of). It did not appear that McKinsey conducted the sort of evaluation tests that would have validated this claim. But it is useful to imagine what kind of evidence would be convincing.

First, it would help to set up clear definitions: the better someone does on the entrance exam, the more likely they will be to succeed in the Foreign Service (exam score → success). But, a) How exactly do we measure test scores? and, b) How do we measure success in the Foreign Service? For (a), we might just acquire each officer’s original test scores. A discerning observer might raise some concerns here. What do we do with parts of the test that are scored more subjectively (such as the essay, biographical information, and in-person interviews)? And how do we account for changes in the test over time? Regarding question (b), What does it mean to do well in the Foreign Service? I have chosen to use the metric “Years to promotion into the Senior Foreign Service.” Again, there are weaknesses in this measure: The promotion process has been heavily criticized as somewhat arbitrary. We might worry, for instance, that those most quickly promoted aren’t necessarily the “best” employees for a variety of reasons. Good science demands we carefully answer these questions. But let’s ignore these wrinkles for now.

Now that we’ve carefully defined our terms, we can collect some data and run some analysis. Below, I produced two graphs to demonstrate hypothetical relationships between entrance exam scores and promotion speed (to be clear, this data is entirely made up). The first graph shows that officials who scored best on the entrance exam tend to do better in the Foreign Service. According to this data, those who score in the top 10 percent are expected to get promoted into the Senior Foreign Service two years faster than those in the bottom 10 percent. This would be a meaningful finding; the test seems to be capturing merit.

Next, I produce an alternative dataset. This one suggests that there is no correlation between scores on the entrance exam and years to promotion. While scoring 100% on the exam seems better than 60%, the scores demonstrate no relationship with future promotion prospects. This finding would seriously call into question the construct validity of the entrance exam.

Evaluating the Promotion Process

Let’s turn to another example focused on the Foreign Service’s promotion process. I’ll briefly describe the process so this makes sense to outsiders: the State Department dictates that every official gets a performance evaluation every year. When an employee’s time arrives to be considered for promotion, five years’ worth of evaluations will be stapled together into a file for each official. This file will be read and rated by multiple judges. The judge’s scores are averaged and, after some discussion, the highest-rated officials get promoted. (It’s a bit more complicated than this, but you get the idea.)

The State Department’s Bureau of Examinations (BEX) boasted in 2025 that “candidates could demonstrate all dimensions on virtual platforms and assessors could assess candidate performance accurately.” The office also explained that subsequent training of assessors and program assistants had ensured the ability to “administer and score the exercises accurately and without any potential for bias.” In short, BEX claimed that their evaluation process has both high construct validity and high inter-rater reliability. But, again, they’ve given us no data to validate their claim. Let’s visualize the sort of evidence would support their claim.

Below, I visualize some (entirely artificial) data to exhibit what high inter-rater reliability looks like. I imagine an environment in which there were 40 officers being evaluated (listed across the X-axis), and each of their files is read by the same 5 judges. Individual scores for each file are represented by colored dots, each representing a different judge. The average score (Y-axis) for each file is indicated with a black bar. You can see from the distribution of dots and scores that clear preferences emerge about the best and worst employees. The reviewers’ scores clump together tightly; there is almost perfect agreement about the quality of each file. We might still have important questions about the construct validity of this procedure, but inter-rater reliability is pretty good. If this is what BEX’s data looks like, it would lend confidence to their process.

Let’s look at an alternative situation. Below, I produced another dataset that still demonstrates clear preferences for promotions. But you can see that the inter-rater reliability in this environment is much lower. In fact, when we measure the underlying agreement between raters (using, in this case, a statistical procedure called intraclass correlation coefficient), there is virtually no consistency between reviewers. While the process produces the appearance of preferences, promotions given on this basis would be entirely arbitrary. They may as well have been rolling dice.

Which of these situations most accurately resembles the State Department’s recruitment and promotion processes? We don’t know. And that’s a problem.

It is possible that State Department officials have a reasonably accurate sense of merit. Certainly, there is knowledge within the system about what makes a great foreign policy official. And the aggregated judgment of many managers producing many evaluation reports for their staff likely adds up to something better than completely random. Yet, the experience of most observers of the Employee Evaluation Process is that it is rather arbitrary and unpredictable, often dismissed as little more than a “creative writing contest.”

Further, an institution genuinely committed to evidence-based personnel selection would likely describe its work very differently. It would publish its validation studies and acknowledge the limits of its findings. Then it would learn from the results and constantly innovate to improve and refine its methods of detecting merit. But the divergence between the Department’s confident public characterization of its assessment process and the absence of the underlying validation evidence is, in a sense, a form of evidence. The institution in charge of evaluating the quality of candidates’ critical thinking does not appear to be demonstrating sophisticated evaluation itself.

Indeed, the State Department invests few resources in studying merit. It neglects to study success and failure in our foreign policy. And it presents little agreed-upon measurement, data, or clarity about what it looks like for a staffer to be effective. This leaves us with little ability to comment on the construct validity of any performance management systems. Nor are we able to check to see that there is at least inter-rater reliability in the measurement of performance files.

This is the natural result of a culture of foreign policy that treats itself as “more art than science.” As we look to improve our institutions in the coming years, I hope we can do better.

***

In my next article I want to explore what kind of research the State Department should invest in to better understand merit in foreign policy. I would love to hear ideas from all of you. Any ideas for how we ought to study merit in foreign policy?

I received a TON of interesting feedback and ideas from the community following my last article on reforming the Foreign Service Act. This is a reminder of just how much expertise and energy there is within our community to engage on reform agendas. Perhaps I’m still naive about how Washington needs to work, but I’ve always felt it would benefit us to create more forums for discussing these laws with the people most intimately familiar with their effects.

I’ve organized the comments into two categories: comments on the current proposal and ideas for future additions. Most of this is directly quoted (with some light editing), including from this LinkedIn post thread. Credit goes to the original authors!

Comments on Draft Foreign Service Act Reauthorization

1. re: new service requirements after language training. The bill would require three consecutive language-designated tours for any officer receiving 6+ months of “world language” training. “This could work, I suppose, for Spanish, French, and Portuguese speakers. But assigning those officers to three consecutive tours using that language would shrink the availability of officers to be trained for Hungarian, Hebrew, or Vietnamese. So, down sides would be mixed with up sides… To the Director General and the Bureau, that provision may prove a real headache, and it will cramp career broadening by requiring those language officers to stay in one country a very long time.”

2. re: Foreign Service Reserve Corps pilot: “if every professional level person hired by State (FS or GS) were subject to at least 3 tours overseas (Jr, Mid, and Sr) AND needs of the Service, a Reserve Corp would not be necessary! Think about it: The Reserve Corps means inadequate service…” The commentator, an experienced former FSO, worried about retired folks trying to replay their “20th-century fever dreams.” Another former official countered that the “reserve [corps] makes sense, especially given punitive TICs which only increase during budget crash times.”

3. re: New requirement for interagency service for SFS promotion: “Joint duty ‘makes sense’ except the tiny size and no float on the FS make it impracticable and punitive. Another observer notes that “The number of SFS compelled into retirement by TIC before ever being able to fill a Senate-confirmed position tells that marginalization, not capacity, is the point.”

4. re: Promotion Board changes: “All the newly required back and forth between the Department and Congressional staffers will, I am confident, be tedious and slow. The simple solution is for the bill to require the Department’s IG to annually investigate and report whether Public Members are recruited by the book.”

5. re: changes on legal domicile rules: “I was startled to read the section that would allow FSOs to maintain their legal domicile ‘back home’, as legislation already provides for members of the armed forces. Has this proposal been noted in Richmond? There will be squawks! Virginia will not welcome losing so many taxpayers!”

6. re: dualhatting the Director General as the A/S for HR: “...a swipe at the foreign service, so that the position becomes the reverse of what it was intended... [This will] disempower and degrade the FS in favor of ‘blending’ career categories.”

7. re: Tiger Team on Expeditionary Diplomacy: Tiger Team is an Air Force phrase — I hope the team, when it begins its work, won’t think the FS is (or should be) a military organization.

Ideas for Future Amendments

Skills and capacity issues

1. Require training for FSOs on program design, management, and evaluation (to supplement the loss of USAID skills)

2. Commission a longitudinal study on the efficacy of the FSOT, and of promotion processes. Map scores to career success.

3. “A cap on political appointments and levels at which they can be appointed. For example, only career appointees allowed below A/S, and a limit of 20% political ambassadors, etc.”

4. “A dramatic architectural reform that includes domestic service beyond Washington. The reason it was so easy to dismantle a significant portion of the diplomatic effort during the DOGE days is because State, USAID, and others were structurally unable to answer one key question: Why does this matter to the American people? The answer made sense in Washington, but it was abstract or absent in Dayton, St Petersburg, or Omaha.”

5. “Administrative streamlining such as eliminating travel management centers and purchasing flights directly/getting reimbursed.”

6. “A reformed and valuable leadership development training modeled on cohort experience like the military (all infantry officers that get promoted to major spend six weeks in KY).”

7. “Reporting officers should be given a pot of money to plan programming that aligns with objectives.”

8. “Streamlined the annual planning system that puts in for visits.”

9. “An innovation fund for employees to explore ideas about how to make the department operate better. And a small team to coordinate innovation projects, give out grants, run pilots and competitions, and set up an institutional architecture and ecosystem so the department can continuously improve itself -- without expensive consultants.”

Pocketbook and family issues

10. “A Foreign Service Families addition that makes the FS more accessible for families: 1) prevent FS spouses from having to serve in different posts; 2) ensure that families with special needs kids will be supported by local schools; 3) guarantees tour extensions for things like allowing high school seniors to graduate at post; 4) create significant career opportunities for spouses through better paid/more meaningful engagements at post and or via virtual work supporting DC; 5) create and fund more post based positions for spouses and make it possible to transfer from post to post; 6) Allow spouses to earn credit for service and retirement benefits so they aren’t 100% dependent on their spouse for income and retirement; 7) help with access to child care when relocating to DC (where there can be long wait lists for access).”

11. “End the ban on retired FSOs from receiving pay for any civil service position. We all know the Department has, in its bureaus, many retired colonels. They are indeed qualified for many positions, and they can receive both their military retired pay and the civil service salary in the new job. [But] there’s no reason for a retired FSO to work at, say, one of the war colleges on the faculty, because they are barred from receiving Civil Service pay on top of their FS annuity.”

12. “Align with the military and provide housing allowances for domestic assignments so that Washington service, which is usually seen as mandatory for senior promotions, is not reserved to only those who can afford it.”

My overall take is that the bill is constructive. Nothing in this bill aims for deep structural change, and both supporters and critics of the Foreign Service might have desired more creative challenges to the status quo. Nevertheless, it is commendable that HFAC is following through on its responsibilities to regular reauthorizations. The bill will still need to make its way through the full House and Senate approval and signature into law, so there may be substantive additional changes throughout the process. But here’s where the legislation stands for now, along with a bit of my own analysis:

The legislation calls for a Tiger Team to be established at the State Department to develop a plan to improve the capability and willingness for expeditionary diplomacy. The report will be due in two years. I see this as a positive initiative, especially in contrast to the unhelpful politics that surrounded the 2012 Benghazi attacks. It may be politically helpful that this report will be completed during a Republican administration. That said, this issue has been thoroughly studied, and one might have hoped that HFAC could have simply legislated changes now rather than further postponing the legislative work. Nevertheless, this is an important step.

The legislation establishes a Diplomatic Reserve Corps Pilot Program. This program will test a surge capacity mechanism that can recruit retired and former officers back into operations during crises. This idea was drafted six years ago by the powerful former Undersecretary for Management, Patrick Kennedy, and has remained a priority for the American Academy of Diplomacy. I have written previously that it is unclear what lessons the Reserve Corps proposal learned from the defunct Civilian Response Corps. I hope this pilot goes well and provides a useful service to the Department.

Speaking of management issues, the legislation would codify that the Director General of the Foreign Service serves concurrently as Assistant Secretary for Human Resources. The DG position was designed as an institutional advocate for the Foreign Service, so dual-hatting this position seems more likely to disempower the Foreign Service; perhaps that is the goal.

Perhaps the most impactful proposal in the legislation is the addition of a joint-duty assignment requirement before promotion into the Senior Foreign Service. The logic seems to be that Senior Foreign Service officers are frequently called on to collaborate with other agencies, or even exercise Chief of Mission authority over them. While this requirement will be phased in over five years, I wonder about the workforce impact of many senior folks scrambling to exit the already short-staffed Department for external assignments. In principle, however, I like this idea. While time in training and on external assignments is typically seen as “not career-enhancing” for Foreign Service Officers, inserting this into promotion precepts means everyone will be on equal footing.

And, the legislation adds some career protections for members on external training, education, or details, preserving their rank, tenure clock, promotion eligibility, and right to a performance evaluation.

The legislation makes a small but notable tweak to qualifications for the Board of Examiners by striking “or training in the fields of testing” and inserting “academic study, background in diplomacy”. This seems like a good change, but the promotion process remains highly subjective. I have always been interested in a deeper study of how the State Department can and should measure merit.

At the same time, the bill directs the Secretary of State to convene an independent commission to review the structure, relevance, and effectiveness of the Foreign Service’s career tracks (formerly known as cones: Political, Economic, Public Diplomacy, Consular, and Management). It is hard to say where this could go, but it could be interesting.

The legislation directs the Department to author a new cybersecurity/technology governance framework for missions abroad, covering AI governance, hardware supply-chain security, and incident response. This seems smart.

The legislation encourages the appointment of knowledgeable and experienced officials, suggesting that “detailed knowledge and requisite experience formulating and executing United States foreign policy, including a working understanding of Department operations and procedures, is vital for chiefs of mission, Assistant Secretaries of State, and other senior officials at the Department…” It’s a nice thought, but the Senate already has confirmation authority over these senior positions, and inexperienced outsiders have been entering the Department at unprecedented rates, so...

The legislation also does a lot for Veterans. Most prominently, it creates a Foreign Service Pathway for Veterans Program that will pilot a streamlined hiring process to recruit veterans for Diplomatic Security, cybersecurity, and any other position deemed mission-critical. It seems like an easy win if we can find highly qualified applicants from the military. But many are concerned that veterans’ preferences have eroded meritocratic hiring principles: in our current hiring system, a long-serving veteran will typically beat out a civilian with far more relevant expertise. There are some hazards in fast-tracking veterans rather than reforming the hiring process more holistically.

Adds additional training requirements across a range of issues, including: antisemitism/Holocaust distortion, AUKUS partnership coordination, international narcotics/law enforcement, foreign online scam syndicates, defense cooperation authorities for chiefs of mission, cybersecurity/AI, crisis leadership, and consular issues.

Requires timely transmission of tenure/promotion lists to the President for Senate confirmation, with reporting if deadlines are missed.

Adds detailed anti-nepotism and transparency rules for recruiting public members of Foreign Service selection boards, including conflict-of-interest disclosure, documentation of “word-of-mouth” referrals, and recordkeeping.

Adds legislative-branch experience (e.g., congressional fellowships) as a qualifying credential for promotion.

Addresses tax residency treatment for Foreign Service Officers posted abroad.

* * *

As always, I would love to hear from you about your thoughts about the Foreign Service Act reauthorization. Are there other ideas for changes you wish would have appeared in this bill?

A recent report from Harvard’s Belfer Center suggests that AI may soon be capable of performing most of a diplomat’s job, including policy analysis, negotiation, crisis response, writing talking points, news assessment, managing diplomatic protocol, translation, and processing visa applications. If you believe the author, this is a great opportunity for diplomacy. Other commentators even suggest going all-in on AI is a strategic imperative.

But the report also contains a startling admission: “Artificial intelligence will further intensify diplomats’ struggle for relevance and could reduce the workforce of foreign affairs ministries to a bare minimum within the next fifteen to twenty years.” This serves as a warning to diplomats: AI is coming for your job.

It was thus a surprise to me when the Foreign Service Journal – “the voice of the Foreign Service” – decided to publish an edition of its journal dedicated to AI, largely extolling its usefulness (Note: I am a member of the FSJ’s editorial board). I have great respect for the authors, and there is obviously enormous opportunity with these new tools. But the excitement around AI risks causing us to overlook the many ways it can go badly wrong. Such failures could result in catastrophic damage to American diplomacy and national security.

Nobody can say for sure how this will all turn out. Some of the advances in AI are truly extraordinary. But we need to be prudent and examine the assumptions underlying this debate. Proponents of AI are making three big bets that require scrutiny:

1. AI can help make us better and faster.

2. The technology is going to continue to get better.

3. We can make AI both superintelligent and super-safe.

Each of these might be right. But each is also a critical failure point.

Will AI make us better and faster?

Maybe! But, then again, maybe not.

When people speak about AI these days, they’re usually referring to generative AI powered by Large Language Models (LLMs). LLMs that run tools like ChatGPT are “giant statistical prediction machines that repeatedly predict the next word in a sequence,” according to IBM. But such tools offer only “the illusion of thinking.” LLMs are an impressive technology, but they also present serious reliability concerns.

There are some settings in which LLMs seem to thrive. For instance, customer service agents are found to be much more productive when supported by AI. This job works well for LLMs because there are millions of transcripts of human-led support calls used to train the model to be fast and accurate. And the domain is not too complicated; there’s usually a right answer to any support question.

But foreign policy is a hard environment for AI because the domain is complex, contextual, and dynamic. These features create problems for LLMs. A literature review finds that while “LLMs have demonstrated significant potential in parsing and analyzing extensive datasets to identify patterns, predict future events, and detect anomalous behavior across various domains,” it also identifies critical challenges, including that LLMs are weak at applying discovered patterns to novel situations, such as generating policy solutions. Another study concludes that LLMs are “bad at declining to answer questions they couldn’t answer accurately, offering incorrect or speculative answers instead.” When the ability of popular tools to provide accurate citations was evaluated, they failed, on average, 60% of the time. Grok 3 failed an eye-popping 94% of the time. If one lacks the specific expertise and workflow to detect errors, hallucinations can go unnoticed, leading to disastrous results.

The solution to these challenges, many AI proponents suggest, is to place a “human in the loop.” The human partners with AI to monitor the output and catch any mistakes. The promise of the human-in-the-loop model is that each diplomat could do more, better, and faster. Imagine an expert diplomat, assisted by AI, covering a broader portfolio because it’s so much easier to summarize wider swaths of information. Better yet, imagine a single officer replacing an entire country desk. Oh, the efficiency and coordination improvements we could achieve!

But the evidence so far is sobering. A 2024 meta-analysis found that, on average, AI-human collaboration performed “significantly worse” than either the highest performing humans or AI models. That said, the collaborations outperformed the average humans in the studies. The results depended on the task: AI-human teams were very poor at decision tasks, but very good at content creation, for example.

A 2025 randomized trial of experienced software developers found that those partnered with AI believed they were working 20% faster – but testing showed they were actually taking 19% longer to complete tasks. In other words, partnering with AI produced overconfident users who did not know they were being slowed down; they were tricked into thinking their performance improved! This effect seems to be playing out at the organizational level as well: A July 2025 report from MIT NANDA found that “despite $30–40 billion in enterprise investment into GenAI… 95% of organizations are getting zero return.”

Again, what’s so pernicious about AI is that it all feels so convincing. Even when they’re wrong, they seem right. It is because of this that I remain quite wary of AI’s use for tasks on which the user can’t personally validate the quality of the answer the machine produces. Miles Williams, in an article on AI and international affairs, says it well:

“using AI tools well is not really a skill; it’s a second-order implication of having skills. The people I see using AI effectively in their work today are people who developed expertise and skills before AI tools were readily available… they know how to double-check and refine what their AI agents produce because they already know how to do the tasks they outsourced.”

In other words, well-trained diplomats remain a necessary ingredient for the State Department to use AI effectively (even aside from the fact that it might just slow them down). Unfortunately, as I have written about extensively in this Substack, the State Department does a poor job of prioritizing expertise.

This leads us to some tricky labor questions. The threat of AI to workers has led many unions, prominently including the Writers Guild of America, to prioritize AI protections for their labor force. But at the State Department, the experts best positioned to fact-check the AI output may also be the most expendable. Pressure to pursue efficiency gains may result in career officials being sidelined, and their forms of historical knowledge may be perceived as the easiest to replicate with AI. In contrast, the more subjective decision-making authority endowed in political appointees may protect them. Indeed, this is already happening. Why task a costly, slow-thinking human with a job when one can just query ChatGPT?

There is also a dimension of this debate that receives insufficient attention: CSIS’s Futures Lab tested leading AI models against 400 diplomatic crisis scenarios and found evidence of systematic escalation bias. They reported that AI was “hawkish to a fault.” In other words, the more we rely on AI, the more likely it is that we end up in unnecessary conflict. Meanwhile, our adversaries are building AI too. Chinese LLMs like DeepSeek and Qwen exhibit similar escalation bias, particularly toward Western nations. If American diplomats are consulting tools with one set of biases while Chinese counterparts use tools with a different, potentially conflicting set, the diplomatic risks compound dangerously.

We are also discovering that the use of AI appears to cause cognitive decline. A study from Switzerland on nearly 700 people found “a significant negative correlation between frequent AI tool usage and critical thinking abilities.” An MIT study used electroencephalography to demonstrate that people literally power down their brains when using AI, eroding their critical thinking skills. A Microsoft Research and Carnegie Mellon survey of 319 knowledge workers confirmed the pattern: “higher confidence in GenAI is associated with less critical thinking.” AI also appears to damage our intellectual autonomy through “a process of gradual de-skilling, where we lose skills that we currently take for granted.” Other research offers similar warnings: AI usage triggers “metacognitive laziness,” declining motivation in one’s work, and reductions in effort and confidence.

Two scholars issue a stark warning in a new paper, How AI Destroys Institutions:

“AI systems are built to function in ways that degrade and are likely to destroy our crucial civic institutions. The affordances of AI systems have the effect of eroding expertise, short-circuiting decision-making, and isolating people from each other. These systems are anathema to the kind of evolution, transparency, cooperation, and accountability that give vital institutions their purpose and sustainability. In short, current AI systems are a death sentence for civic institutions, and we should treat them as such.”

Will the technology get better?

Definitely. But how much better? If you believe proponents like Elon Musk, AI will become “smarter than any human by the end of this year.” But others are skeptical, suggesting the next breakthrough may be decades away. Other leaders in the field believe that LLMs are a dead end and are pivoting their focus to other technological pathways.

LLMs are not magic; they have physical limitations stemming from energy use, processing power, the limited availability of training data, and the inherent limitations of the underlying technology. Without getting into the technical details of the challenge of scaling LLMs, let’s use an analogy: One might have believed that engineers could make internal combustion engines steadily more efficient over time. But that assumption was wrong. It turns out that the laws of physics get in the way. The same may be true for AI.

Most likely, AI will get better at certain tasks, and not others. It may thrive at balancing our accounting books, but struggle to adjudicate visa interviews accurately. Or, it is possible I am entirely wrong, and the sky is the limit for LLM models. Indeed, some very smart people believe that LLMs will continue to improve, quickly achieving artificial general intelligence (AGI), surpassing humans in virtually all domains.

Will superintelligent AI be safe?

There’s a parable in the AI community about a robot trained to make paperclips. At first, the robot hums along, making piles of paperclips. But it soon runs out of supplies at its factory, and realizes that to achieve its paperclip-making work, it needs to get more materials. So it first dismantles the factory walls, turning its surroundings into paperclips. Then it starts gobbling up everything in sight, turning the whole city into new paperclips. The robot deems the terrified humans trying to shut down production as bad for productivity, so it gobbles them up too. Soon, nothing is left except the robot and a planet’s worth of paperclips.

Science fiction movies feature superintelligent robots who hatch schemes to wipe out the human race. That’s plausible. But many AI safety experts believe the more likely scenario is that an AI is too stupid to realize the implications of its actions. This is called the “alignment problem.”

Despite safety concerns, government agencies and private companies are advancing AI development and integration faster than society can understand the implications of what we are building. Some policymakers responsible for AI governance and regulation have little technical knowledge of the tools they implement. In national security, big questions about safety are being set aside in the rush to ensure technological superiority over our adversaries. The private sector, whose leaders have a substantial financial interest in rapid adoption, is spending heavily to persuade policymakers – and especially the national security establishment – that their AI tools are essential, inevitable, and safe. Maybe they are right. But independent evaluators are giving the private sector poor grades on AI safety. In their rush to capture market share, companies are demonstrating a “striking lack of commitment to many areas of safety.”

The result is that some of the most prominent A.I. experts are terrified by what they built, just like the scientists who built the atomic bomb, calling AI an “existential risk.” They worry about things like a computer starting a nuclear war that will destroy the planet, hatching a bioweapon, or some other terrible thing that we haven’t even imagined. In response, AI optimists suggest it is more likely that AI will deliver us into a utopia in which humans achieve “unparalleled abundance” and an end to scarcity and war.

The Takeaway

AI holds great promise. But the utility of AI in international relations rests on several shaky assumptions. If any one of these assumptions fails, the whole AI edifice may crumble. Meanwhile, the field of diplomacy seem poorly-positioned to shape the rewards and risks to our profession on this critical issue. I hope that begins to change.

As always, I welcome your feedback. If there is research here that I have missed, please share it. And I would love to hear from professionals inside and outside government about your experiences so far with AI.

I am excited to share with you a chapter I contributed to a new book about public diplomacy. The book, Diplomacy’s Public Turn: Prospects for Theory and Practice, is an edited volume compiled by the wonderful scholars Kathy Fitzpatrick and Bruce Gregory. Public diplomacy is in the midst of generational changes, and the opportunity for a group of authors to step back and reflect on its future is immensely useful. I hope folks will grab a copy of the book: https://link.springer.com/book/10.1007/978-3-032-12857-7

My chapter is titled What Is an Expert? Public Diplomacy in a Complex and Uncertain World. The chapter argues that public diplomacy practitioners are shifting toward a new culture of expertise grounded in empirical evidence, rigorous evaluation, and adaptive learning. Expertise requires two essential conditions: clear feedback and deliberate integration of that feedback through effortful practice. Yet public diplomacy often lacks these feedback loops, leading to overconfidence and ineffective programming. Through a case study of a public diplomacy campaign that inadvertently exacerbated political violence, the chapter highlights the risks of relying on anecdotal evidence rather than systematic evaluation. It proposes a new framework for expert public diplomacy emphasizing: (1) clearly articulated, research-informed goals; (2) robust monitoring and evaluation systems; (3) iterative learning loops to refine strategies; and (4) a workforce trained in analytical and evaluative competencies. Ultimately, it calls for institutional reforms to embed evaluative rigor and intellectual humility— namely, individual expertise—across the diplomatic enterprise, ensuring that public diplomacy is effective and accountable in an increasingly complex global environment.

Here’s the full list of chapters with abstracts:

Introduction, Kathy R. Fitzpatrick & Bruce Gregory

This chapter introduces a collection of works from thought leaders and innovators who explore the growing societization of diplomacy at home and abroad and near universal acceptance of public diplomacy’s centrality in contemporary diplomatic practice—what we call diplomacy’s public turn. Contributors share ideas and insights on three important trajectories in a shifting diplomatic landscape. First, the complexity of state-society relations with stunning growth in numbers, categories, and impacts of diplomatic actors and publics. Second, a redefined diplomatic operating environment due to the expanding scope and scale of societal interconnections, disruptive geopolitical conflicts, societal polarization, and declining trust in institutions and mediated information. Third, the transformation of academic study and everyday diplomatic practice resulting from blended digital and analog technologies, rampant disinformation, and the threats and opportunities of artificial intelligence. This chapter provides a preview of the authors’ views on these transformational changes and the impact of diplomacy’s public turn on diplomacy’s future.

The Societization of Diplomacy: Transitioning Operational Landscapes, Shifting Lenses, Jan Melissen & Lavinia Pacifici

The study of diplomacy has long been centered around the architecture of states and their external interactions, yet rapid societal transformations are reshaping its foundations. The increasing politicization of foreign policy, the rise of populism, and the growing weight of public concerns in global governance are among the forces challenging the traditional separation between the domestic and international spheres. We introduce the concept of the societization of diplomacy to describe this shift and its significance, illustrating how diplomatic practice is rooted in evolving state-society dynamics that cross borders. Introducing the social contract as a first reference, diplomacy emerges not so much as a tool of statecraft but as a practice significantly influenced by citizens’ expectations of legitimacy, representation, and participation. Society is no longer peripheral—if it ever was—and now plays a central role in shaping diplomacy, requiring recognition both in theory and practice. This reconceptualization expands the scope of diplomatic studies, incorporating insights from political theory, sociology, and other social sciences to better understand diplomacy as a governing practice. Ultimately, the societization perspective redefines diplomacy as a collaboratively produced effort where the state and society are deeply interconnected, continuously negotiating interests, the meaning and purposes of foreign policy, and global responsibilities.

Societizing Diplomacy or Diplomatizing Society?, Paul Sharp

This chapter presents diplomacy as a distinctive and valuable social practice. It argues that students of diplomacy should shift their attention from applying the insights of other social practices to how diplomats should act in a public environment populated by social actors. Instead of looking at the societization of diplomacy, they should be researching, theorizing, and advocating for the diplomaticization of social actors and the conditions under which this is more and less likely to occur.

Pathways to Humanity-Centric Approaches to Public Diplomacy as an Instrument of Progress, Prosperity, and Peace, R. S. Zaharna

This chapter challenges traditional battle-oriented public diplomacy, arguing for humanity-centered approaches that align with global connectivity and diversity dynamics amid diplomacy’s expanding societal engagement. The research identifies a critical misalignment between competitive “battle for hearts and minds” strategies—rooted in separateness assumptions—and the prevailing dual dynamics of connectivity and diversity that characterize contemporary cross-border challenges. Through analysis of off-stage initiatives, the study delineates features of humanity-centric approaches: holistic perspective, diversity integration, relationality, global consciousness, issue-driven focus, process orientation, and collaboration. Expanded functions are also explored: problem-solving for complex global challenges, identity mediation, emotion mediation during crises, and conflict mediation and peacebuilding. Contemporary case studies are used to demonstrate humanity-centric approaches in practice. The chapter advocates for consensus communication that aligns with humanity-centric features and the global dynamics of connectivity and diversity. As global challenges demand collective action, humanity-centric approaches offer a practical imperative for leveraging connectivity and diversity to achieve human progress, prosperity, and peace.

Regional Diplomatic Culture and Diplomacy’s Public Turn, Geoffrey Wiseman

As concerns mount about the downsides of globalist approaches to world politics and the rise of illiberal nationalist ones, debates continue as to whether and how regionalist perspectives might enhance international cooperation. This chapter considers whether increased scholarly and practitioner focus on the concept of regional diplomatic culture can contribute to thinking about diplomacy and its “public turn.” It identifies three criteria for evaluating the idea of diplomacy’s public turn: more openness (transparency) about diplomatic work, wider and deeper engagement with home publics about diplomacy (societization), and more interactive (relational, or dialogical) public diplomacy. The chapter considers these criteria in relation to five disparate “regions” — the European Union, North Asia, South Asia, Southeast Asia, and the Pacific Islands. While regional conceptions of identity and diplomatic interaction are highly varied worldwide, the chapter concludes that a regional-cultures interpretation of the public turn adds normative weight to longstanding appeals to democratize diplomacy.

The Boundaries of Public Diplomacy: Public Turn while Maintaining the Diplomatic Ground, Kadir Jun Ayhan

The first step in delineating the boundaries of public diplomacy is to ground it firmly as a diplomatic practice. In this chapter, I suggest that, for the sake of analytical coherence, public diplomacy activities must satisfy the following conditions: communication must take place between groups separated by geopolitical boundaries; the political agenda must be for public or collective interests represented by institutionalized actors with intentionality, ultimately to influence or support foreign policies. I explain these conditions to distinguish public diplomacy from other forms of human interactions.

Rethinking the Public in Public Diplomacy: Contemporary Characteristics and Implications, Jian Wang

This chapter reflects on the evolving nature of “the public” in contemporary society. It calls for a renewed focus on the public as the central concern of public diplomacy efforts. The discussion begins with a reconsideration of Walter Lippmann’s seminal insights into how publics form perceptions of the world beyond their immediate experience, which remain strikingly relevant in today’s digital information environment. The essay then outlines six key characteristics that define the contemporary public. It concludes by emphasizing the need for more advanced approaches to audience analysis as essential elements of effective public diplomacy.

Reputational Security on the Edge: Diplomacy’s Public Dimension in a Post-Soft Power World, Nicholas J. Cull

This chapter argues that one of the main ways of understanding the societization of diplomacy—the theory and practice of Soft Power—is now so diminished as to constitute the end of an era. It posits the idea of Reputational Security as a substitute. The chapter explains core elements of Reputational Security, including its openness to collective work, its connection to the core business of the state, and its linkage to reform. It considers future development of the approach using the well-known five-element model of public diplomacy: listening, advocacy, cultural diplomacy, exchange diplomacy, and international broadcasting. It concludes that the reputations of western countries and especially the United States are fragile, and countries that fail to work together to address the shared challenges of our era will be judged critically, eroding both reputation and—by implication—security.

Bridging Gaps in Public Diplomacy Research, Eytan Gilboa

This work presents and analyzes four gaps in public diplomacy (PD) research: between and among disciplines, between theory and practice, between theory and reality, and between theory and technology. The first two are known and have been researched albeit in general connection with foreign policy, not with PD. The other two are just beginning to be researched. PD is the most multidisciplinary field ever developed. Yet, scholars from many disciplines in the humanities, social sciences, and technology haven’t yet found ways to expand collaboration. The world order is moving from a liberal system to an illiberal one. Much of the present literature has been pursued within the liberal order. PD scholars need to explore the roles and functions of PD in the emerging one. AI is relatively new but has already had significant effects on PD. The combination of AI and disinformation could be disastrous to PD. This chapter presents ways to reduce the gaps, but the main conclusion is that only sustained and creative multidisciplinary research can close or reduce these gaps.

Mediatization, Logics of Diplomacy, and the Crisis of Epistemic Security in the Digital Age, James Pamment

This chapter investigates the theoretical and practical intersections of mediatization theory, the “logics” approach to public diplomacy, and the concept of epistemic security. Mediatization theory challenges our understanding of diplomacy by showing how diplomacy is not merely communicated through media but increasingly constituted by media logic. The “logics” approach expands on this by introducing a pluralistic, institutionalist framework that explains the dynamic rationales underpinning different public diplomacy practices. This chapter examines these theories against the backdrop of what might be termed a crisis of epistemic security; that is to say, significant challenges to what constitutes valid knowledge in foreign policy. Using three brief empirical cases, the chapter illustrates how the mediatization of diplomacy and its guiding logics both shape and are shaped by the increasing instability of shared knowledge frameworks in the post-truth era. The essay concludes that promoting and safeguarding epistemic security is now a central task for diplomacy, and that mediatization and logics theories offer valuable theoretical tools to understand and navigate this challenge.

Towards AI-Enhanced Public Diplomacy? A Quantum Reflection, Ilan Manor

Over the past two decades, digital technologies have disrupted the practice of public diplomacy giving rise to new norms, values, and practices among diplomats and ministries of foreign affairs (MFAs). This chapter seeks to examine how artificial intelligence (AI) could further disrupt the practice of public diplomacy. The chapter uses concepts and theories from quantum mechanics to argue that AI will further accelerate the fragmentation of digital publics into networks that subscribe to different definitions of reality. AI-generated content, disseminated and filtered through social media algorithms, may create a plurality of plausible realities to which networked individuals subscribe. The chapter further argues that public diplomacy by nature seeks to create a shared definition of reality between states and publics. As such, AI presents a substantial challenge to the practice of public diplomacy, one that could be overcome through diplomatic collaboration with epistemic communities and the bolstering of epistemic authority. The chapter concludes by outlining how such collaborations may prove more beneficial than existing attempts to regulate AI at the national or international level.

Digital Public Diplomacy and the Illusion of the Online/Offline Divide, Elsa Hedling

The rise of social media platforms in the early 2010s marked a shift in public diplomacy practice, resulting in the emergence of “digital public diplomacy.” This chapter explores the evolution of digital public diplomacy by emphasizing digital culture as a significant source of social and political change in a broader societization of diplomacy. Moving beyond a narrow focus on digitalization, it argues for a broader understanding of digital public diplomacy through the lens of a digital culture that encompasses the norms, behaviors, and political expressions shaped by pervasive digital technologies. Drawing on research on the European Union’s public diplomacy, the chapter critiques the conceptualization of the online/offline divide and advances digital sociability as a more instructive lens for analyzing contemporary public diplomacy practices and their centrality in cross-border collaboration. It concludes by reflecting on the implications of digital culture for the future of public diplomacy research and suggesting new directions for inquiry in an era where digital engagement and persuasive power are both essential and increasingly contested.

Reimagining U.S. Diplomatic Practice: The Case for Public Diplomacy, Vivian S. Walker

At a moment when the U.S. government appears to be systematically devaluing diplomacy’s public dimension, and the underlying values that frame it, this essay offers a realistic, practitioner-focused assessment of the evolving roles, responsibilities, and value of public diplomacy. The networked age has resulted in the definitive and irrevocable broadening of the community of practice. It has also introduced communication technologies that, while disruptive, allow for more effective projection of national legitimacy and power in the global information space. To assure U.S. public diplomacy’s continued relevance, practitioners must engage in ruthless program prioritization and consolidation; more consistent, consolidated, and accessible program data sharing; creation of a flexible mechanism for interagency coordination on message development and delivery; and strategic integration of successive technology innovations in program design, implementation, and assessment. These recommendations for the reimagination of public diplomacy presuppose the centrality of diplomacy’s public dimension and an expanded scope of practice. They assume a continued reliance on persuasion and attraction as viable instruments of statecraft. Ultimately, the reimagination of public diplomacy for this tumultuous age demands a renewed commitment to soft power and the exercise of strategic patience.

Conclusion, Bruce Gregory & Kathy R. Fitzpatrick

Contributors to this collection were invited to consider key issues in diplomacy’s public turn during recent decades and to reflect on what successor generation scholars and practitioners should understand going forward in turbulent geopolitical and technological landscapes. This chapter summarizes areas of consensus on changes from earlier thinking and areas of difference on analytical, normative, and operational issues. The authors share common ground on three developments. First, the societization of diplomacy and diplomacy’s public dimension are established core concepts in academic study and diplomatic practice. Second, digitalized diplomacy and AI will enhance, disrupt, and transform diplomatic institutions and patterns of practice. Third, scholars and practitioners must engage in new ways with knowledge domains across the sciences, social sciences, and humanities. The collection also offers an array of perspectives sure to energize debate and prompt new ideas. Familiar areas of difference on terms, concepts, and definitions are reframed in the context of diplomacy’s societization and changing circumstances. Chapters reflect diverse perspectives on boundaries needed to define analytical concepts, disciplines, and practices of state-centric, sub-national, supra-national, and humanity-centric diplomatic actors. Other differences are revealed in multiple pathways to innovations in diplomacy’s institutions, tools, and methods. The authors’ ideas and insights will be foundational in future research and in advancing diplomatic practice.

Today, my colleague Dr. Thomas Scherer released a report I’m very excited to share: How to Predict the Future in Foreign Policy. This research contains profound implications for the practice of foreign policy. Every policy proposal is a prediction about how the future will transpire; understanding the future is thus a central task for policymakers.

This report offers a first-of-its-kind review of the nascent body of evidence of what works in accurately forecasting the future. It explains what works, and why, and jumpstarts a necessary conversation about how to implement these best practices within the hallways of government. Every foreign policy expert should familiarize themselves with these cutting-edge approaches. (And certainly its lessons travel far beyond the realm of international relations)

There has been an explosion of new methods and research in recent years to help people more accurately grasp the future. It turns out that some of these approaches are backed by great evidence, while some may be little more than snake oil.

The report is organizes common prediction tools into three categories, each informed by a different body of research:

1. Human Judgment Forecasting involves developing highly accurate human predictors by identifying talent, training skills, and refining the prediction process.

2. Quantitative Prediction involves using datasets and statistical methods to model relationships between variables and extrapolate them to make predictions.

3. Foresight (also known as futurism or horizon scanning) involves scanning for signals and trends of possible futures and then working through scenarios based on those futures to improve planning in the present.

The report finds robust evidence to suggest Human Judgment Forecasting and Quantitative Prediction are valuable tools. But it turns up little evidence to demonstrate the efficacy of Foresight methods, despite their growing popularity in the private sector. The lack of evidence for Foresight is a yellow light, not a red one — the field has demonstrated disinterest in evaluating the efficacy of its approaches, and more investments are needed here (and across the other two categories as well) to better understand how these tools work.

Dr. Scherer concludes in this report that “the question we care most about is the extent to which improved forecasting and prediction can improve the effectiveness of a policy’s impact in the real world. Like the Greek myth of Cassandra, knowing the future but being cursed to be ignored is unhelpful.” This is important. Just because a forecasting technique improves accuracy does NOT mean it will necessarily improve policy outcomes. Research demonstrates that predictions often get ignored by policymakers. Thomas thus challenges our government to think hard about how to build these tools into the policy process.

Nevertheless, “the research shows that forecasting and prediction methods have improved to where it would be irresponsible to ignore them. However, it also suggests that care should be taken in their application and that they should be continuously evaluated and adapted. The future is indeed a moving target; our policy processes must move with it.”

As always, I’m eager to learn more from our readers on this juicy topic:

- Have you used forecasting and prediction tools in your work?

- Are there any methods or research we missed?

- Have you discovered ways to effectively integrate these tools into decision-making?

Below is the full text of the report. You can also view it on fp21’s website: https://www.fp21.org/publications/how-to-predict-the-future

How to Predict the Future in Foreign Policy

Success in foreign policy depends on the ability to foresee and prepare for events, such as the U.S. government’s anticipation of Russia’s invasion of Ukraine. Inversely, foreign policy failure is often rooted in being caught off guard by events, such as the Biden administration’s failure to anticipate how quickly the Afghan government would collapse.

Understanding the future is a central task of foreign policy. Policymakers and analysts are regularly confronted with questions such as: What is the adversary’s next move? Will our ally support this action? Which new threats, trends, or technologies must we prepare for? And, perhaps most importantly, how do we expect our policies to shape the future? Every policy and strategy is a claim about how the future will transpire.

A large industry has developed around the field of foreign policy to advance forecasting. The names and methods of this work are varied: forecasting markets, scenario planning, tabletop exercises, risk management, early warning systems, red teams, futurism, geopolitical intelligence, and more. Many of these structures are being experimented with inside and outside of government, some with seeming great success.

Less well known is how well these tools actually work. Fortunately, scholarly research into future studies has grown exponentially in the last twenty years, though the vast majority of forecasting research is on non-foreign policy topics such as weather and the economic demand for goods and services.¹ This report is designed to give a foreign policy audience an overview of common approaches to understanding the future. It describes each approach, reviews the recent research, and identifies critical knowledge gaps for future research. For this report, I reviewed over 100 books and articles with an emphasis on meta-reviews. I used broad reviews of future studies to identify the main methodological categories and then conducted focused searches of recent publications (within the last 20 years) on each category.

Readers will find in this review both good news and bad news. The good news is that there is strong evidence that certain tools are effective at improving one’s ability to understand the future. Even highly trained intelligence analysts have been outdone by new methods of forecasting.² The bad news is that much of the literature remains immature, and even the most optimistic studies counsel humility in the face of the vast uncertainty inherent in international affairs. Nevertheless, in the competitive world of national security, where a single decision can have massive consequences, improving the predictive accuracy of policymakers by small margins could significantly shift geopolitical fortunes.

This report groups research into four categories according to the underlying techniques used to understand the future: Human Judgment Forecasting, Quantitative Prediction, Horizon Scanning, and Foresight. These categories are intended as broad summaries, as many experts and organizations describe their efforts with divergences in terminology and methodology.³

I define my categories as follows:

• Human Judgment Forecasting involves developing highly accurate human predictors by identifying talent, training skills, and refining the prediction process.

• Quantitative Prediction involves using datasets and statistical methods to model relationships between variables and extrapolate them to make predictions.

• Foresight involves scanning for signals and trends of possible futures and then working through scenarios based on those futures to improve planning in the present.

Human Judgment Forecasting

How Does it Work?

Human Judgment Forecasting methods are focused on improving forecasts by identifying the personalities, skills, and techniques that lead to greater accuracy. Research has found that some people are naturally better forecasters, and some are superforecasters, consistently out-predicting other experts. Research has also demonstrated that forecasting is a skill that can be trained and improved. It turns out that reading the crystal ball is more science than art.

Much of our understanding comes from forecasting tournaments, where hundreds of individuals compete to answer questions about the future with the highest accuracy.⁴ A typical tournament might ask forecasters thousands of questions, and then score the accuracy of their predictions in order to rank participants. Researchers can then identify the characteristics associated with the highest-scoring individuals or teams.

Crucial to this effort is the ability to evaluate whether predictions are right. This requires resolvable questions, meaning that they have clear, observable criteria and a deadline. For example, “Will more than 100,000 people be killed in the Russo-Ukrainian War in 2024?” is a resolvable question, whereas “What’s going to happen in the Russo-Ukrainian war?” is not. Typical questions query when something will happen, how many/how much of something will occur, or those that invite a simple yes-or-no answer. Given that the questions are necessarily about the future, a tournament’s results can only be tabulated once reality has provided the answer for each question. For instance, if a question invites one to predict the result of an election in November 2028, it can only be resolved once that date has passed.

Forecasters not only offer their prediction, but also their level of uncertainty. Predicting something will occur with 90% confidence, and ultimately being proven right, will garner a better score than correctly predicting something will occur with 80% confidence. Similarly, registering a prediction with 100% confidence – but being proven wrong – will be very costly to a forecaster’s score. The exact details of how predictions are made and scored vary, but typically, predictions closer to the actual outcome score higher.

Confidence is a useful characteristic in forecasting tournaments because it mirrors the methodology prescribed by the intelligence community. Intelligence Community Directive 203 requires that all analysis “Properly expresses and explains uncertainties associated with major analytic judgments,” using specific expressions of uncertainty (Figure 1).⁵

Figure 1: Required uncertainty terminology in the Intelligence Community.⁶

What Does the Research Say?

Through a series of forecasting competitions and experiments, researchers have generated a growing body of evidence validating the strength of this approach and providing useful insight about how to improve human judgment forecasting. One study tracking forecasts of geopolitical events suggested that while untrained forecasters were 36% more accurate than random guessing, forecasters with just one hour of training were 41% more accurate (Figure 2).⁷

Figure 2: Percent increase in forecaster accuracy over random guesses with training and teams.⁸

Training

Better forecasting can be taught. Training in probabilistic reasoning, especially the use of comparison classes and base rates, improves accuracy.⁹ The impact of this training can be seen in how forecasters use comparison classes and dialectical complexity in their reasoning, as demonstrated by their use of terms like ‘however,’ ‘yet,’ ‘unless,’ and ‘on the other hand.’¹⁰ One experiment found that just even a half-hour lesson on forecasting improved scores.¹¹

Forecasters who were asked to consider analogies in a structured way are more accurate than those who weren’t (46% accuracy vs. 32%). Accuracy improved further when forecasters thought of multiple analogies or analogies that they had direct experience with (60%).¹² If left to their own devices, participants avoided the analogy technique because it was more onerous and cognitively challenging.

Accuracy also improves with practice. One study found that forecasters who scored lower on cognitive tests and had less accurate forecasts showed greater improvement over two years, narrowing the gap between them and the top forecasters.¹³ Feedback that allows forecasters to learn has been repeatedly shown to improve forecasts.¹⁴

Education and Experience

More education and work experience improve forecasting, but only to a point. People who demonstrate greater knowledge of current events have higher forecasting accuracy.¹⁵ Working professionals make more accurate predictions than undergraduate students, but having a PhD offers no additional improvement.

Surprisingly, more years of experience does not further improve predictions among working professionals. Most surprisingly, professionals with specific expertise in a region did not perform better on predictions about that region. People with access to classified information also performed no better than outsiders.¹⁶

Cognitive Style

Research suggests that open-minded forecasters tend to be more accurate. In Expert Political Judgment, Philip Tetlock found that the best forecasters enjoy questions that can be answered in many ways and give serious consideration to each possible answer. They eschew grand theories that promise to explain everything, preferring flexible ‘ad hocery’ that stitches together lessons from a wide range of world views. They succeed because they are more open to dissonant counterfactuals, more willing to update their beliefs, and less susceptible to hindsight bias.¹⁷

Further, forecasters who scored higher on a test for open-mindedness (i.e., those who find value in considering opposing arguments and changing beliefs) made better predictions.¹⁸ More accurate forecasters spend more time on a question, suggestive of greater deliberation, and make more prediction attempts on one question, likely updating their beliefs with new information.¹⁹ Better forecasters also scored higher on cognitive tests that assess fluid intelligence and the ability to resist seemingly obvious but incorrect answers.²⁰ The findings related to open-mindedness have been challenged, however. Another study found that whether a forecaster self-reported open-mindedness was not related to accuracy.²¹

Belief System

Forecasters with moderate beliefs tend to perform better than their ideological counterparts. Across dimensions of right-left (faith in free markets), realist-institutionalist (centrality of power in foreign affairs), and boomster-doomster (optimism about global economic growth), those in the middle of each of these three dimensions were more accurate than those on the extremes in either direction.²²

Teamwork

Groups generally perform better than individuals in forecasting.²³ Even high-performing forecasters show greater accuracy when placed in groups. The best-performing groups are more decentralized, with egalitarian participation; the benefits of teamwork dissipate within a top-down decision-making structure. High-performing groups also gave longer reasons for a forecast with more explanations, and their discussions were more likely to include words associated with good teamwork and analysis.

Groups that discuss forecasts and interact more show greater improvement from working as a group. Groups that receive teamwork training display more of the collaborative behaviors that help unlock the potential gains from working together.²⁴

Teamwork does not always improve forecasts. The benefits of teamwork are vulnerable to overconfidence, as overly confident team members can unduly influence the group’s decision. Teams are most beneficial when each member’s confidence matches their accuracy.²⁵

Optimism can also erode the benefits of teamwork. Teams are as susceptible to optimism bias as individuals.²⁶ Teams tend to make riskier decisions as more cautious individuals fear being seen as the group’s pessimist.²⁷ The gains from teamwork disappear if the group is given an overly optimistic starting point.²⁸

Incentives

Forecasters can be incentivized to be more accurate. Forecasters perform better when they are told they will be evaluated on their accuracy, versus when told they will be evaluated on their process.²⁹

Prediction markets are open platforms that allow bets on future events. The market crowdsources people’s bets on the probability that an event will occur and aggregates the averages into the market price. This is similar to stock trading, except buying low is betting on an event that others believe to be unlikely, and one’s “shares” grow more valuable as more people agree with your bet. Prediction markets with monetary stakes perform slightly better on average than markets that operate with fake money. Research has largely found that predictive markets perform similarly to other human forecasting methods. In an experiment where forecasters were assigned to individual predictions, team predictions, or prediction markets, markets performed better at the start of short-term questions, while team predictions were better at longer-term questions.³⁰ Many of these experiments were completed on private, low-stakes prediction markets; public large-stakes markets raise the prospects of price manipulation and insider trading.³¹

The Delphi Method

The Delphi Method is a structured process to elicit human judgment forecasting from a group while minimizing the pathologies of group decision-making.³² Named for the Delphic oracle of Greek mythology, RAND developed this method in the 1950s to forecast the effects of technology on warfare.³³

The method begins with a group of forecasters making predictions independently. The coordinators collect the predictions and then provide each forecaster with an overview of the results, including the group’s average. The coordinators may also share anonymous justifications that forecasters provided with their predictions.

Next, the forecasters are allowed to update their initial predictions. The results are again shared with the group. This process is repeated until the forecasts stabilize, or until the forecasters reach a consensus.³⁴ Supporters of the Delphi Method explain that the key features are anonymity to free the individuals of any societal influence, feedback to expose the individuals to potentially differing views, and iteration to allow the feedback to have an impact.³⁵

The Delphi method improves forecast accuracy, but its positive effects are similar to other group structures, most of which are faster and take less effort.³⁶ Further, research on the Delphi Method is outdated and weakened by inconsistencies in its execution.³⁷ Differences in who participates, what feedback is given, and how predictions are tabulated affect the outcome and make evaluation difficult.³⁸ The RAND Corporation has published 50 studies using the Delphi method in the last ten years, and recently developed the Delphi Critical Appraisal Tool to assess the quality of Delphi efforts, creating the potential for a reevaluation of the Delphi Method.³⁹

Quantitative Support

An experiment during an Intelligence Community-sponsored forecasting competition found that giving human forecasters statistical model-based predictions only improved the accuracy of some of the forecasters, presumably those who knew when and how to use those predictions.⁴⁰ Another forecasting experiment provided human forecasters with high-quality advice from either humans or an algorithm. Among the forecasters that received advice from an algorithm, those considered experts – i.e., those with national security experience - were more likely to ignore the advice and were ultimately less accurate than the non-experts.⁴¹ The quality of the predictive algorithm obviously matters; the next section dives deep into the details of quantitative predictions.

Hybrid Prediction

Another way to use statistical forecasts with human judgment forecasting is to combine them by taking their average.⁴² This method is sometimes called the Blattberg-Hoch approach, who argue that “any combinations of forecasts prove more accurate than the single inputs.”⁴³

This method can be especially effective in cases when human judgment is likely to stray far from the pattern predicted by statistical forecasts.⁴⁴ Two studies provided experts with quantitative predictions and had them make adjustments. In both cases, the average of the quantitative prediction and the human prediction outperformed each on its own. Weighting the two guesses equally proved to be optimal on average.⁴⁵ It is worth noting that a core precept of good forecasting technique is to begin one’s analysis by evaluating historical patterns predicted by statistical forecasts, called the base rate.⁴⁶

Artificial Intelligence

Forecasters have begun using AI to compete in forecasting competitions with some success. An aggregate of 12 Large Language Models, including ChatGPT 4 and Claude 2, scored similarly to the human crowd in Metaculus’ Fall 2023 competition.⁴⁷ The models are programmed to forecast using the same structured techniques as superforecasters. In Metaculus’ Fall 2025 forecasting competition, the eighth-place finisher was ManticAI, an artificial intelligence.⁴⁸ There were caveats to this success. The human competitors were largely amateurs, and AIs have an advantage just by making predictions early and often.⁴⁹ Indeed, human superforecasters still outperformed top-performing AIs.⁵⁰ There is some evidence that human-forecasting can be improved when paired with an advanced AI agent.⁵¹

There are two major challenges for AI forecasts in geopolitics: data scarcity and human unpredictability.⁵² Where those challenges are overcome, in weather forecasting, for example, AI is starting to match and even outperform some major forecasting models.⁵³

Quantitative Prediction

How Does it Work?

Quantitative prediction uses statistical methods to find patterns in past data and extrapolate them into the future. The human brain may be capable of comparing and contrasting between two similar cases – the wars in Vietnam and Iraq, for instance – but computers are required to compare across many cases. The challenge is that one needs machine-readable information (data) to describe each variable of interest for each case.

The first step in quantitative prediction is to identify the outcome one aims to predict, such as whether a particular country will experience a severe civil war. Next, one considers the factors that might predict this outcome. To continue with the civil war example, one might consider government type, economic development, dependability of public services, whether it has recently experienced conflict, and other factors suggested by previous research. One would then collect historical data on each of these factors. Data in hand, one then conducts a statistical analysis to identify the relationships between those variables. The civil war analysis may show that across many different wars, as the size of opposing armies or the amount of third-party support increases, the severity of the civil war also tends to increase.

The statistical analysis helps one create a model, an equation that best predicts the outcome based on the input variables. One develops a model by choosing the functional form (e.g., linear regression) and the variables to include. These are important choices that can have a significant impact on one’s predictions.⁵⁴ As it is often unclear beforehand which model specification is best, modelers can create multiple models and assess their performance over time.

The last step, once a model has been selected, is to apply the model to the input data to predict the outcome. While one can predict an outcome that will happen in the future, one still needs existing input data. Often, models use inputs from the current time period to predict outcomes for the next period. For example, a model trained on data ending from 2023 may make predictions for 2024.

An IARPA forecasting competition found that poor data quality on some topics resulted in extreme and unhelpful predictions.⁵⁵ High-quality quantitative prediction depends on the availability of data that is complete, accurate, and relevant. These three qualities are important, and I offer brief definitions here:

Completeness refers to how well the data covers the full population or period. Rarely does one have access to every single data point. Instead, datasets are typically representative samples of the concept one is interested in. If data is collected non-randomly and thus exclude some parts of the population (called selection bias), the sample is not representative, and the quality of its forecasts will be harmed. For example, if data on diplomatic meetings is only drawn from publicly available news sources, secretive diplomacy will be excluded from the dataset. Such a dataset would be systematically biased, and any analysis drawn from that data would likely be misleading.

Accurate datasets are able to successfully measure reality. Faulty measurements lead to less accurate data. For instance, a database of deaths in a civil war based on participant reports may be less accurate than data gathered from morgues or reported by trusted IGOs and NGOs.

Relevant data refers to how useful the data is for predicting the outcome, sometimes called its predictive power. There are two dimensions to relevance. First, does the data capture the desired concept? For example, to measure ‘civil war intensity’, battle casualties seems a better measure than threats on social media. Second, is the concept represented by the data useful to predicting the outcome? Relevant concepts are usually identified from theories supported by past research. For instance, theories of civil war duration suggest we are better off relying on data on the number of weapons in a country rather than the number of computers.

It is important to recognize that quantitative models are not perfect reflections of reality. Measurement in the field of foreign policy requires the simplification of infinitely complex social realities that defy clean definitions. Simplification does not make a model useless. Models are intended to be parsimonious. One simply must understand the choices and tradeoffs made to produce the model.

What Does the Research Say?

The online stock trading commercials are right: past performance does not guarantee future results, but quantitative predictions can still be quite helpful.

The recent improvements in computing power, data availability, and quantitative methods have led to a rise in quantitative prediction efforts. In the field of international conflict, efforts such as the Political Instability Task Force (PITF), Integrated Crisis Early Warning Systems (ICEWS), Violence & Impacts Early-Warning System (VIEWS), and the Early Warning Project (EWP) demonstrate the value of quantitative prediction and offer lessons for improvement.

The CIA set up the Political Stability Task Force (PITF) in 1994 to predict state failure (e.g., due to civil war, a coup, or genocide. The task force developed a model to rank countries by risk. The task force considered over forty variables before settling on four in their final model.⁵⁶ They trained the model on data from 1955 to 1994 and then ranked countries by their risk of instability every year from 1995 to 2004. In other words, the model predicted the countries most likely to fail. Setting a threshold that those in the top quintile of risk would experience instability, the model correctly predicted 86% of the instances of instability and 81% of continued stability.⁵⁷

When researchers later reapplied the model to the 2005-2014 period, the model’s accuracy declined over time. It correctly predicted only 35% of instability cases. The immediate lesson is that the drivers of instability change over time – the drivers of instability in the post-9/11 era were different than those in the post-Cold War era. The larger lesson is that predictive models, whether quantitative models or mental models, must evolve over time lest they become outdated and lose accuracy.⁵⁸

The Integrated Crisis Early Warning Systems (ICEWS) was a Department of Defense-funded project designed to forecast global instability events, including rebellion, ethnic/religious violence, and international crises. The project experimented with several different models and aggregated them based on which countries or events they performed best on. The constituent models included those focused on leaders and society based on subject matter expert surveys, models using events from text-parsing 6.5 million global news stories, and a geo-spatial model considering connections and interactions between neighboring countries. This combination of models reportedly reached over 90% accuracy for predicting every type of instability. It is worth noting that for rare events like rebellion and insurgency, where change is unusual, a model that always predicts a continuation of the previous year, known as a naive model, also achieved over 90% accuracy. The tougher, but more important task, is predicting whether instability will persist. A naive model, by definition, predicts 0% of changes; the best ICEWS model was able to predict 56% of changes.⁵⁹

VIEWS is a conflict prediction system that ran a competition to predict monthly changes in the intensity of state-based violence, measured by the number of fatalities, at the state level and at the sub-state level using grid cells (Figure 3).⁶⁰ Compared to PITF and ICEWS, VIEWS challenges forecasters to make predictions at a more granular level: the number of fatalities instead of a binary for instability, for months instead of years, for sub-state grid cells as well as countries. The competition submissions advanced the field with innovative data (e.g., geo-located peacekeeping units, COVID-19 deaths, internet search data) and algorithms (e.g., neural networks, machine learning). The results found that parsimonious models perform at least as well as models with many variables. More importantly, all the submissions were able to beat the benchmark and naive models, demonstrating that investment can lead to predictive gains on even the hardest tasks.⁶¹

Figure 3: VIEWS data dashboard of fatality predictions for January 2025.⁶²

The Early Warning Project (EWP) from the Holocaust Memorial Museum estimates the percentage chance of new mass killings in each country for the next two years.⁶³ They conducted an extensive review of scholarly literature to identify the causes of mass killing. They also collect data on demographics, socioeconomics, governance, human rights, and conflict history from 1960 or earlier. They train many models using different algorithms and variables, and they use the last two years of data to identify the most predictive. Predictive variables include civil liberty inequality, population size, a history of mass killing, infant mortality, coup attempts, political killings, battle deaths, and power monopolized by some social groups.⁶⁴

The EWP advocates that policymakers pay close attention to those countries with the highest risk.⁶⁵ The EWP’s December 2024 report highlights the top 30 at risk in 2024-2025 (Figure 4).

The EWP shows the importance of quickly gathering data. The EWP wants to use the most recent data, but some of the 2023 data is not ready until late 2024. Since a prediction for 2024 published at the end of 2024 is of little use, the EWP predicts a two-year window to provide policymakers with some information for 2025. In terms of accuracy, since 1994, EWP has demonstrated its predictive power by forecasting a higher risk in countries that experienced a mass killing (8%) than in those that did not (3%).

Figure 4: Early Warning Project’s assessment of the 30 countries with the highest risk of mass killing for 2024-2025.⁶⁶

Human Adjustments

Quantitative predictions can improve human forecasts; can human forecasts improve quantitative predictions? The evidence supporting the Blattberg-Hoch approach of combining quantitative and human predictions applies here, suggesting there are gains from human adjustments. A study of human adjustments to 60,000 forecasts by supply chain companies found that human adjustments increased accuracy on average. The study also found evidence of optimism bias, demonstrating that positive adjustments by humans to the algorithmic estimates tended to be less accurate than their negative adjustments.⁶⁷

Foresight

How Does it Work?

Foresight is thinking about possible futures and the opportunities and challenges they could bring. Foresight is not about predicting what will happen, but rather about what may happen.⁶⁸ Foresight considers different plausible futures and the challenges that each entails to help organizations or individuals prepare for any future.⁶⁹

As a recent review put it, “Foresight cannot predict the future nor reduce uncertainty. What foresight can do is to explore a wide range of possible futures and to engage participants in “what if” exercises to help them deal with uncertainty and better understand possible consequences at the system level and inform their decision-making to be more effective at creating the future they want.”⁷⁰

The US Office of Management and Budget characterizes foresight as “insight into how and why the future might be different from the present.” It differentiates this from forecasting, “Which seeks to make statements or assertions about future events based on quantitative and qualitative analysis and modeling.”⁷¹

The UK Office for Science gives a broader definition: “Foresight refers to the application of specific tools/methods for conducting futures work, for example, horizon scanning (gathering intelligence about the future) and scenarios (describing what the future might be like).” Some descriptions of foresight are directly in line with rational choice theory – trying to better understand the possible outcomes of choices to make a sound decision.⁷² Similarly, corporate foresight is when private-sector firms identify and interpret factors that are likely to threaten the status quo, and propose options for preparing appropriately.⁷³

Foresight practice is being intentional and systematic about foresight efforts to increase their effectiveness. Exact foresight processes vary, but their steps usually include some form of Framing, Scanning, Hypothesizing, Workshopping, and Broadcasting.

Framing

Framing defines the bounds of the issue or problem to be studied. This could include (but is not limited to) identifying key questions, selecting topics, setting time horizons, or establishing geographic boundaries.⁷⁴ Narrower framings allow for quicker, cheaper, and/or more detailed analysis, but they also risk missing relevant future challenges that fell outside of the framing.

For example, the US Coast Guard runs a strategic foresight process every four years called Project Evergreen. The framing for Evergreen V, started in 2018, focused on what global and regional changes might mean for the Coast Guard’s operations 15-25 years in the future.⁷⁵

Scanning

Scanning is the search for signals and drivers that portend impactful trends, surprises, and possible futures.⁷⁶ Signals are “early whispers of change,” and drivers are “forces that propel and shape change”.⁷⁷ The National Intelligence Council’s quadrennial Global Trends report is an example of such a scan; the 2021 report reviews trends in demographics, environment, economics, and technology.⁷⁸

Scanning can target a wide range of sources, including expert interviews, group discussions, sector reports, social media, and mainstream news. A scan evaluating how future technology may change war efforts might collect information from industry associations, conference presentations, military projects, and patent filings. As weak signals can come from unexpected issues, some recommend scanning broadly beyond the immediate subject, for example, by following the PESTLE taxonomy: political, economic, social, technological, legal, and environmental.⁷⁹ This taxonomy can also be used for Driver Mapping to see what areas are absent from a scan.⁸⁰

Scans may include tools such as the previously discussed Delphi Method as well as others like System Mapping, Seven Questions, and Three Horizons. System Mapping is creating a visualization of key elements in a system (nodes) and the relationships between them (links) to identify areas of possible change worth scanning.⁸¹ Seven Questions is a guided framework for interviews.⁸² As part of framing and scanning, one should also identify the assumptions held about the topic to be interrogated later.⁸³ Three Horizons is a technique to explore assumptions over three different time horizons.⁸⁴

Horizon Scanning is a systematic process to detect early signs of important future developments. These efforts are similar enough to Foresight to include here, but adopt some different steps and frameworks. What Foresight calls ‘scanning’, Horizon Scanning may be further subdivided into gathering information, filtering, prioritizing, analysis, and dissemination.⁸⁵

In 2009, a horizon scan to identify future trends relevant for EU policymaking identified six relevant factors: demography/migration, economy/trade, environment/energy, research/innovation, governance/social cohesion, and defense/security. For each area, experts analyzed 20-25 reports on each area and identified 370 specific issues, such as a global decline in defense spending or radical groups deploying more sophisticated violence. Each of these specific issues was then rated on relevance, novelty, and probability using a survey of 270 external experts. For example, NATO’s increased openness to outside partnerships would be highly relevant. The widespread diffusion of surveillance sensors globally by 2020 was judged to be relevant, likely, and highly novel. A major war by 2020 was judged highly relevant but unlikely.⁸⁶

Scenarios

The next step is to make sense of and gain insights from the signals and trends. This is typically achieved by creating and discussing scenarios, often in a workshop setting, to identify possible futures and explore what each future would mean for the participants, as well as how they would prepare for it. By imagining themselves in these scenarios, participants identify how the current policies and plans are insufficient for possible futures.⁸⁷ Scenario analysis has also been used in academic settings to spark overlooked research agendas that may be highly relevant in the future.⁸⁸

Scenarios can be built around Change Drivers – statements that explain a group of signals – or by taking two axes of uncertainty and using them to create a 2x2 matrix with four possible scenarios.⁸⁹ A SWOT analysis (Strengths, Weaknesses, Opportunities, Threats) can also inform the scenarios. Visioning, Backcasting, and Roadmapping are approaches that start by envisioning an ideal future scenario and then identifying the steps needed to connect it to the present. Future Wheel is a tool to structure brainstorming around future events and casualties.⁹⁰ Scenario Stress-Testing is injecting specific situations into a scenario to compare responses.⁹¹

Informing

The final step is ensuring that the ideas and insights from the scenarios inform future decisions, policies, and plans. This could include written products that summarize weak signals and drivers of change, possible future scenarios, and patterns of challenges and opportunities.⁹² Firms and organizations may have existing change management processes to connect the results to, or may set up new steps towards transformation.⁹³

Foresight proponents also argue that participants are informed just by participating. During the scenarios, participants clarify and test their assumptions, develop mental models of the system they operate in and how it can change, and identify future challenges, opportunities, and uncertainties. Importantly, they have done this as a group, thus creating a shared understanding and sense of purpose as they move forward together.⁹⁴

Decision Making Under Deep Uncertainty

Decision Making Under Deep Uncertainty, also called Robust Decision Making, was developed by scientists at RAND as a decision-making tool (whereas foresight and other prediction methods are analysis tools). Deep Uncertainty is defined as when parties to a decision cannot agree on the likelihood of future events or the consequences of policy choices.⁹⁵

The goal of RDM is to choose a strategy that is robust, meaning that it “performs well, compared to the alternatives, over a wide range of possible futures.” The procedure leads analysts to frame their problem and goals, and then extrapolate a wide range of plausible futures. The likely effects of any proposed policy responses are then predicted for each of the diverse conditions presented by the many plausible futures. This exploratory modeling generates a map of outcomes that reveals where and why particular strategies might fail. Analysts then identify the vulnerabilities of each strategy through scenario discovery and refine the options by incorporating adaptive elements that can respond to changing circumstances.

For example, an early RDM study on reducing greenhouse gas emissions while supporting economic growth started with two policy choices: whether to have carbon taxes and whether to subsidize low-carbon technologies. It then identified important inputs such as the future impacts of climate change and the cost and performance of solar technology, each of which had several potential values. It ultimately identified thirty total inputs, which it combined to create 1,611 plausible futures.⁹⁶

In this study, the models showed that it was always better to have a carbon tax. It then looked at which futures supported having a subsidy and found that subsidies were worthwhile except in futures where climate change impacts were small and technology easily diffused. Assigning probabilities to those two inputs, the study concludes that taxes and subsidies are more likely to achieve the policy goals.⁹⁷

What Does the Research Say?

Evaluation of foresight focuses heavily on outputs such as the number of publications and participant satisfaction surveys, and looks for evidence of impact such as endorsements from decision-makers and references in policy documents. This is also true for horizon scans; an early assessment of the EU exercise reported metrics participant satisfaction, a claim that the recommendations were useful in discussions in the European Commission, and the citation of the final report in the 2010 Communication on the Innovation Union.⁹⁸

Unlike human judgment and quantitative prediction, the foresight community has shown little interest in evaluating the accuracy of its forays into the future. As one evaluation explains, “foresight cannot predict the future nor reduce uncertainty.”⁹⁹ The word ‘prediction’ itself is viewed so negatively that even theorizing the effects of foresight practice is discouraged.¹⁰⁰ As such, relative to human judgment and quantitative prediction, the research on foresight is underwhelming and often ad hoc.

One of the more interesting studies looked at the foresight needs and practices of 70 firms in 2008 and compared them to their profitability in 2015. Firms with foresight practice levels equal to their needs (termed vigilant) had an average profitability of 16%, compared to 10% in firms with foresight practices one level lower (vulnerable) or higher (neurotic) than their needs (Figure 5). This research suggests that foresight is useful, but only up to a certain point! Too much foresight may be just as costly as too little foresight.¹⁰¹

Figure 5: Average profitability of firms by future preparedness levels.¹⁰²

When large firms were asked about the benefits of their strategic foresight practices, approximately one-third confirmed that foresight led to insights into changes in the environment (35%), helped understand the market (31%), and aided in understanding customer needs (30%). Fewer affirmed that their foresight practices identified opportunities and threats to their portfolio (26%), reduced uncertainty (25%), created the ability to adopt alternative perspectives (20%), facilitated organizational learning (17%), and reduced uncertainty in R&D (11%). Top performing firms report more value from their strategic foresight than low performing firms.¹⁰³ All of this data is self-reported and thus must be taken with a grain of salt.

A case study of a scotch whiskey manufacturer found that a series of scenario workshops helped the management team face operational realities and let go of tightly-held assumptions that had served them poorly.¹⁰⁴

Some research in this space is suspect. One team of researchers surveyed 66 major firms and concluded that the use of foresight practices is positively associated with the use of foresight in a firm’s strategy process.¹⁰⁵ If that sounds circular, that’s because it is, like claiming “the patients who received the penicillin treatment were overwhelmingly found to have penicillin in their bloodstream.” Research design should always be scrutinized, especially in foresight studies where some researchers work as corporate foresight consultants and have a strong incentive to find foresight beneficial.

Foresight is already common in the US government. In 2018, 18 federal agencies ranging from CIA to FEMA to Office of Personnel Management reported using at least one foresight method, and more have risen since.¹⁰⁶ In 2022, the US State Department set up a Policy Risk and Opportunity Planning Group to conduct strategic foresight and proactive policy planning.¹⁰⁷ Its creation was spurred by State’s after-action review of the Afghanistan withdrawal, which recommended better planning for worst-case scenarios.¹⁰⁸ The Office of Personnel Management has a guide for how all Federal agencies can implement foresight.¹⁰⁹

The US Coast Guard began cyclical scenario planning exercises starting with Project Long View in 1998 and continuing with Project Evergreen since 2003. A 2021 review found that over the effort’s history, over 1,000 personnel have participated.¹¹⁰ Participants reported bringing the strategic foresight perspective back to their station and their colleagues. Some said the exercise allowed them to think more about the future and the bigger picture in their daily activities. Some of the planning from the 1998 session allowed the Coast Guard to adapt rapidly to a post-9/11 reality.¹¹¹

One might be especially skeptical when foresight programs are a one-off and not incorporated into existing planning processes. In case studies of foresight exercises by Dutch provinces and municipalities, researchers struggled to find a connection between the exercises and subsequent policy analysis and strategy. Researchers found that inexperience, poor timing, organizational culture, and lack of leadership hindered any impact. Interviews with national ministers similarly found room for improvement in connecting foresight studies to decision-making.¹¹²

When participants in the 1997 New Zealand Health Sector Scenarios were asked twenty years later about impact, they said that the scenarios were discussed for a short time but quickly forgotten. One participant suggested scenario thinking would take years to properly embed.¹¹³

There is evidence that foresight programs affect the opinions of the participants. In scenario workshops on infrastructure investments, participants changed their opinions on certain investments depending on how useful the investments were in their scenario.¹¹⁴ It just isn’t clear whether those changes have a greater impact.

A scenario workshop in Mali centered on what local agriculture should look like in 15 years. A year after the workshop, 92% of participants agreed that it helped them build relationships, 65% said it enabled them to learn from different perspectives, 62% credited it with bringing about novel changes in their behavior, and 15% developed a deeper understanding of systems. However, the researchers concede that the behavior may not have been novel or attributable to the workshop. They conclude, “there was only debatable evidence of transformational change amongst individuals … and no obvious alterations to policies or policy-making norms.”¹¹⁵

Of the 12 studies on foresight outcomes included in a recent review by the Association of Professional Futurists, two examined the effects of foresight training at the individual level.¹¹⁶ One found that university students who had taken a futures course were more likely to agree to statements related to openness to alternatives than their peers, suggesting the class may have positively affected their ability to make good decisions. But the same study found that the students showed no difference on statements related to long-term thinking and were less likely to agree on concern for others.¹¹⁷ The other only examined whether students engaged with the material and liked the course.¹¹⁸ Such results make clear conclusions difficult.

Foresight’s Resistance to Evaluation

The Association of Professional Futurists reported, “Despite 40+ years of scholarship on the benefits of evaluation and its application in actual foresight initiatives in the peer review literature, the field has been very slow to assess the quality and impact of its work.”¹¹⁹

One review summarizes some of the key challenges. “Foresight evaluation needs to address specific challenges such as complexity, long-time horizons, the qualitative nature of the work, and documenting largely intangible and multiple foresight outcomes. However, these challenges are not unique to foresight work.”¹²⁰

Other reviews find that mainstream foresight research lacks theoretical frameworks, weakening its contribution to scientific knowledge.¹²¹ The scenario literature is dominated by case studies written by advocates trumpeting their positive outcomes, which is useful for advocacy but less so for knowledge building.¹²² The field’s resistance to scientific theory is exacerbated by apparent misunderstandings of what theory and scientific assessment are and outright rejection of mainstream scientific research.¹²³

Another hindrance is the “cult of personalities” around foresight practitioners who, in the words of one scholar, resist scientific theory and method as potential threats to their professional identity. Based on their experience in the foresight community, they conclude, “Efforts to persuade practitioners to adapt their processes and practices on the basis of scientifically sound theory and research, without first addressing more directly the inherently cognitive-affective, social, cultural, and political dimensions of their work, will ultimately fail to achieve the desired outcomes, to the mutual detriment of science and practice.”¹²⁴ Another scholar believes that to advance foresight as a science, scientists must challenge and beat out non-scientists in the marketplace of ideas.¹²⁵

Conclusion

This report reviewed a large tranche of research related to several prominent methods for understanding and predicting the future. Recent developments in human forecasting evaluation and quantitative methods have led to an uptick in research on how these methods can best predict future events.

The accuracy of a given method varies greatly with how well it is implemented. Human forecasting and quantitative predictions have had significant advances in identifying when each will perform better and worse, thanks to their focus on evaluation. Before other methods are discarded for poor performance or lack of evidence, it should be clear why they performed poorly and whether there are any simple remedies.

This review identified several significant gaps in our understanding of forecasting and prediction methods. Studies have repeatedly shown how the details of when and how a method is applied is crucial to understanding its usefulness. While this review highlighted studies used in a foreign policy setting, the vast majority of forecasting applications and the resulting lessons are from other domains, mostly in the private sector. Lessons drawn from those domains should be re-evaluated as they are applied to foreign policy to assess their validity.

Foresight has not had a similar boost and currently has little evidence beyond the anecdotal to support its added value beyond commonplace planning practices. This is not to say that foresight is not worth its cost in time and effort. The meager evidence that does exist suggests that foresight may be more effective with leadership buy-in, participant familiarity, and embedding it in the policy process. This is partly because accuracy isn’t necessarily the goal for foresight. But without clear measure of effects and outcomes, we are left with no approach to assessing the benefits of these methods. Additional work to develop and test this class of methods is welcome.

Ultimately, the biggest question is whether better predictions lead to better policy, and to better outcomes in general. This is the classic problem of focusing on outputs – accuracy – when we really care about better outcomes. The question we care most about is the extent to which improved forecasting and prediction can improve the effectiveness of a policy’s impact in the real world. It is generally assumed that a more accurate prediction will cause better outcomes. This is a strong assumption that may not be true. For example, in a follow-up review of the forecasting tournament sponsored by the Intelligence Community, scholars found virtually no evidence that even the most accurate forecasts were used in intelligence products, let alone by policymakers.¹²⁶

Like the Greek myth of Cassandra, knowing the future but being cursed to be ignored is unhelpful. An undertheorized question that emerges from this research is: How does one integrate futurism into the decision process? fp21 has previously described four possible models for integrating forecasting into policymaking (Figure 6).¹²⁷ Future publications will need to examine how these methods can support policymakers in the policymaking process.

Figure 6: Four Models for Integrating Forecasting into Policymaking.¹²⁸

The tools and techniques reviewed in this report carry important implications for every stage of the decision-making process. However, further research is needed to understand exactly how these tools can be deployed. One study found that decision-makers are more likely to avoid worst-case outcomes when analysis is presented with an emphasis on worst case scenarios versus an emphasis on most-likely scenarios.¹²⁹ Others propose combining scenarios with clusters of human judgment forecasts that give a sense of how likely each possible future is.¹³⁰ This sort of research into blending-techniques to inform decision-making, especially in the foreign policy space, remains limited but presents exciting opportunities.

Finally, there is almost no discussion about the costs associated with each method. While evidence suggests that predictive forecasting is more effective than human judgment in many settings, it may still be impractical for an organization to implement if it requires training or costly data collection. Foresight can also take significant resources.¹³¹ According to the Canadian government’s foresight center, a foresight study on a complex issue can take 2 to 12 months.¹³²

The research shows that forecasting and prediction methods have improved to where it would be irresponsible to ignore them. However, it also suggests that care should be taken in their application and that they should be continuously evaluated and adapted. The future is indeed a moving target; our policy processes must move with it.

Acknowledgements: The author wishes to thank Dan Spokojny for significant contributions in conceptualization, writing and editing. Additional thanks to Gary Gomez and Darrow Godeski Merton for their thoughtful comments.

End Notes:

[1]: Zellner, M., Abbas, A. E., Budescu, D. V., & Galstyan, A. (2021). A survey of human judgement and quantitative forecasting methods. Royal Society Open Science, 8(2), 201187. https://doi.org/10.1098/rsos.201187

[2]: Tetlock, P. E. (2005). Expert Political Judgment: How Good Is It? How Can We Know? (REV-Revised). Princeton University Press. https://www.jstor.org/stable/j.ctt1pk86s8

[3]: Ægisdóttir, S., White, M. J., Spengler, P. M., Maugherman, A. S., Anderson, L. A., Cook, R. S., Nichols, C. N., Lampropoulos, G. K., Walker, B. S., Cohen, G., & Rush, J. D. (2006). The Meta-Analysis of Clinical Judgment Project: Fifty-Six Years of Accumulated Research on Clinical Versus Statistical Prediction. The Counseling Psychologist, 34(3), 341–382. https://doi.org/10.1177/0011000005285875; Grove, W. M., Zald, D. H., Lebow, B. S., Snitz, B. E., & Nelson, C. (2000). Clinical versus mechanical prediction: A meta-analysis. Psychological Assessment, 12(1), 19–30. https://doi.org/10.1037/1040-3590.12.1.19; Kuncel, N. R., Klieger, D. M., Connelly, B. S., & Ones, D. S. (2013). Mechanical versus clinical data combination in selection and admissions decisions: A meta-analysis. Journal of Applied Psychology, 98(6), 1060–1072. https://doi.org/10.1037/a0034156; Zellner et al., A survey of human judgement

[4]: Tetlock, P. E., Mellers, B. A., & Scoblic, J. P. (2017). Bringing probability judgments into policy debates via forecasting tournaments. Science, 355(6324), 481–483.

[5]: Director of National Intelligence. (2015, January 2). https://www.dni.gov/files/documents/ICD/ICD-203.pdf

[6]: Director of National Intelligence. (2015, January 2). https://www.dni.gov/files/documents/ICD/ICD-203.pdf

[7]: Schoemaker, P. J. H., & Tetlock, P. E. (2016, May 1). Superforecasting: How to Upgrade Your Company’s Judgment. Harvard Business Review. https://hbr.org/2016/05/superforecasting-how-to-upgrade-your-companys-judgment

[8]: Schoemaker, P. J. H., & Tetlock, P. E. (2016, May 1). Superforecasting: How to Upgrade Your Company’s Judgment. Harvard Business Review. https://hbr.org/2016/05/superforecasting-how-to-upgrade-your-companys-judgment

[9]: Chang, W., Chen, E., Mellers, B., & Tetlock, P. (2016). Developing expert political judgment: The impact of training and practice on judgmental accuracy in geopolitical forecasting tournaments. Judgment and Decision Making, 11(5), 509–526. https://doi.org/10.1017/S1930297500004599; Kahneman, D., & Tversky, A. (1973). On the psychology of prediction. Psychological Review, 80(4), 237; Mellers, B., Stone, E., Murray, T., Minster, A., Rohrbaugh, N., Bishop, M., Chen, E., Baker, J., Hou, Y., Horowitz, M., Ungar, L., & Tetlock, P. (2015). Identifying and Cultivating Superforecasters as a Method of Improving Probabilistic Predictions. Perspectives on Psychological Science, 10(3), 267–281. https://doi.org/10.1177/1745691615577794; Tversky, A., & Kahneman, D. (1981). Evidential impact of base rates. https://apps.dtic.mil/sti/html/tr/ADA099501/

[10]: Karvetski, C. W., Meinel, C., Maxwell, D. T., Lu, Y., Mellers, B. A., & Tetlock, P. E. (2022). What do forecasting rationales reveal about thinking patterns of top geopolitical forecasters? International Journal of Forecasting, 38(2), 688–704.

[11]: Benjamin, D. M., Morstatter, F., Abbas, A. E., Abeliuk, A., Atanasov, P., Bennett, S., Beger, A., Birari, S., Budescu, D. V., Catasta, M., Ferrara, E., Haravitch, L., Himmelstein, M., Hossain, K. T., Huang, Y., Jin, W., Joseph, R., Leskovec, J., Matsui, A., … Galstyan, A. (2023). Hybrid forecasting of geopolitical events. AI Magazine, 44(1), 112–128. https://doi.org/10.1002/aaai.12085

[12]: Green, K. C., & Armstrong, J. S. (2007). Structured analogies for forecasting. International Journal of Forecasting, 23(3), 365–376. https://doi.org/10.1016/j.ijforecast.2007.05.005

[13]: Mellers et al., Identifying and Cultivating Superforecasters

[14]: Lawrence, M., Goodwin, P., O’Connor, M., & Önkal, D. (2006). Judgmental forecasting: A review of progress over the last 25 years. International Journal of Forecasting, 22(3), 493–518. https://doi.org/10.1016/j.ijforecast.2006.03.007

[15]: Mellers et al., Identifying and Cultivating Superforecasters

[16]: Tetlock, Expert Political Judgment

[17]: Tetlock, Expert Political Judgment

[18]: Mellers et al., Identifying and Cultivating Superforecasters

[19]: Mellers et al., Identifying and Cultivating Superforecasters

[20]: Mellers et al., Identifying and Cultivating Superforecasters. Fluid intelligence was measured by the Ravens Advanced Progressive, and the Cogntive Refection Test measured their ability to reevaluate incorrect responses that look right at first glance.

[21]: Mellers et al., Identifying and Cultivating Superforecasters

[22]: Tetlock, Expert Political Judgment

[23]: Horowitz, M., Stewart, B. M., Tingley, D., Bishop, M., Resnick Samotin, L., Roberts, M., Chang, W., Mellers, B., & Tetlock, P. (2019). What Makes Foreign Policy Teams Tick: Explaining Variation in Group Performance at Geopolitical Forecasting. The Journal of Politics, 81(4), 1388–1404. https://doi.org/10.1086/704437; Mellers et al., Identifying and Cultivating Superforecasters; Mellers, B. A., McCoy, J. P., Lu, L., & Tetlock, P. E. (2024). Human and Algorithmic Predictions in Geopolitical Forecasting: Quantifying Uncertainty in Hard-to-Quantify Domains. Perspectives on Psychological Science, 19(5), 711–721. https://doi.org/10.1177/17456916231185339; Önkal, D., Lawrence, M., & Zeynep Sayım, K. (2011). Influence of differentiated roles on group forecasting accuracy. International Journal of Forecasting, 27(1), 50–68. https://doi.org/10.1016/j.ijforecast.2010.03.001

[24]: Horowitz et al., What Makes Foreign Policy Teams Tick

[25]: Silver, I., Mellers, B. A., & Tetlock, P. E. (2021). Wise teamwork: Collective confidence calibration predicts the effectiveness of group discussion. Journal of Experimental Social Psychology, 96, 104157. https://doi.org/10.1016/j.jesp.2021.104157

[26]: Buehler, R., Messervey, D., & Griffin, D. (2005). Collaborative planning and prediction: Does group discussion affect optimistic biases in time estimation? Organizational Behavior and Human Decision Processes, 97(1), 47–63

[27]: Önkal et al., Influence of differentiated roles on group forecasting accuracy

[28]: Önkal et al., Influence of differentiated roles on group forecasting accuracy

[29]: Chang, W., Atanasov, P., Patil, S., Mellers, B. A., & Tetlock, P. E. (2017). Accountability and adaptive performance under uncertainty: A long-term view. Judgment and Decision Making, 12(6), 610–626. https://doi.org/10.1017/S1930297500006732

[30]: Atanasov, P., Rescober, P., Stone, E., Swift, S. A., Servan-Schreiber, E., Tetlock, P., Ungar, L., & Mellers, B. (2017). Distilling the Wisdom of Crowds: Prediction Markets vs. Prediction Polls. Management Science, 63(3), 691–706. https://doi.org/10.1287/mnsc.2015.2374

[31]: McHugh, C. (2025, October 24). Pollsters Have a New Kind of Competitor. They Should Be Worried. POLITICO. https://www.politico.com/news/magazine/2025/10/24/political-betting-markets-political-predictions-accuracy-00620431

[32]: Gordon, T. J., & Helmer-Hirschberg, O. (1964). Report on a Long-Range Forecasting Study. RAND Corporation. https://www.rand.org/pubs/papers/P2982.html

[33]: Khodyakov, D., Grant, S., Kroger, J., & Bauman, M. (2023). RAND Methodological Guidance for Conducting and Critically Appraising Delphi Panels. RAND Corporation. https://www.rand.org/pubs/tools/TLA3082-1.html; Rescher, N. (1997). Predicting the future: An introduction to the theory of forecasting. State University of New York Press.

[34]: de Loë, R. C., Melnychuk, N., Murray, D., & Plummer, R. (2016). Advancing the State of Policy Delphi Practice: A Systematic Review Evaluating Methodological Evolution, Innovation, and Opportunities. Technological Forecasting and Social Change, 104, 78–88. https://doi.org/10.1016/j.techfore.2015.12.009; Linstone, H. A., & Turoff, M. (2011). Delphi: A brief look backward and forward. Technological Forecasting and Social Change, 78(9), 1712–1719. https://doi.org/10.1016/j.techfore.2010.09.011; Nasa, P., Jain, R., & Juneja, D. (2021). Delphi methodology in healthcare research: How to decide its appropriateness. World Journal of Methodology, 11(4), 116–129. https://doi.org/10.5662/wjm.v11.i4.116; Shang, Z. (2023). Use of Delphi in health sciences research: A narrative review. Medicine, 102(7), e32829. https://doi.org/10.1097/MD.0000000000032829

[35]: Khodyakov et al., RAND Methodological Guidance for Conducting and Critically Appraising Delphi Panels.

[36]: Woudenberg, F. (1991). An evaluation of Delphi. Technological Forecasting and Social Change, 40(2), 131–150. https://doi.org/10.1016/0040-1625(91)90002-W

[37]: Nasa et al., Delphi methodology in healthcare research; Shang, Use of Delphi in health sciences research

[38]: Dalkey, N., & Helmer, O. (1963). An Experimental Application of the Delphi Method to the Use of Experts. Management Science, 9(3), 458–467; de Loë, R. C., Melnychuk, N., Murray, D., & Plummer, R. (2016). Advancing the State of Policy Delphi Practice: A Systematic Review Evaluating Methodological Evolution, Innovation, and Opportunities. Technological Forecasting and Social Change, 104, 78–88. https://doi.org/10.1016/j.techfore.2015.12.009; Drumm, S., Bradley, C., & Moriarty, F. (2022). ‘More of an art than a science’? The development, design and mechanics of the Delphi Technique. Research in Social and Administrative Pharmacy, 18(1), 2230–2236. https://doi.org/10.1016/j.sapharm.2021.06.027

[39]: Khodyakov et al., RAND Methodological Guidance for Conducting and Critically Appraising Delphi Panels. The studies count is based on search results from the RAND website: https://www.rand.org/topics/delphi-method.html

[40]: Benjamin et al., Hybrid forecasting of geopolitical events

[41]: Logg, J. M., Minson, J. A., & Moore, D. A. (2019). Algorithm appreciation: People prefer algorithmic to human judgment. Organizational Behavior and Human Decision Processes, 151, 90–103. https://doi.org/10.1016/j.obhdp.2018.12.005

[42]: Lawrence et al., Judgmental forecasting

[43]: Blattberg, R. C., & Hoch, S. J. (1990). Database Models and Managerial Intuition: 50% Model + 50% Manager. Management Science, 36(8), 887–899. https://doi.org/10.1287/mnsc.36.8.887, as cited in Petropoulos, F., Fildes, R., & Goodwin, P. (2016). Do ‘big losses’ in judgmental adjustments to statistical forecasts affect experts’ behaviour? European Journal of Operational Research, 249(3), 842–852.

[44]: Petropoulos et al., Do ‘big losses’ in judgmental adjustments to statistical forecasts affect experts’ behaviour?

[45]: Wang, X., & Petropoulos, F. (2016). To select or to combine? The inventory performance of model and expert forecasts. International Journal of Production Research, 54(17), 5271–5282. https://doi.org/10.1080/00207543.2016.1167983; Franses, P. H., & Legerstee, R. (2011). Combining SKU-level sales forecasts from models and experts. Expert Systems with Applications, 38(3), 2365–2370.

[46]: Chang, W., Chen, E., Mellers, B., & Tetlock, P. (2016). Developing expert political judgment: The impact of training and practice on judgmental accuracy in geopolitical forecasting tournaments. Judgment and Decision Making, 11(5), 509–526. https://doi.org/10.1017/S1930297500004599; Kahneman & Tversky, On the psychology of prediction; Mellers et al., Identifying and Cultivating Superforecasters; Tversky & Kahneman, Evidential impact of base rates.

[47]: Schoenegger, P., Tuminauskaite, I., Park, P. S., Bastos, R. V. S., & Tetlock, P. E. (2024). Wisdom of the silicon crowd: LLM ensemble prediction capabilities rival human crowd accuracy. Science Advances, 10(45), eadp1528. https://doi.org/10.1126/sciadv.adp1528

[48]: Metaculus Cup Summer 2025. Retrieved October 15, 2025, from https://www.metaculus.com/tournament/metaculus-cup-summer-2025/?page=1

[49]: Ostrovsky, N. (2025, September 18). AI Is Learning to Predict the Future—And Beating Humans at It. TIME. https://time.com/7318577/ai-model-forecasting-predict-future-metaculus/

[50]: Karger, E., Bastani, H., Yueh-Han, C., Jacobs, Z., Halawi, D., Zhang, F., & Tetlock, P. E. (2025). ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities (No. arXiv:2409.19839). arXiv. https://doi.org/10.48550/arXiv.2409.19839

[51]: Schoenegger, P., Park, P. S., Karger, E., Trott, S., & Tetlock, P. E. (2025). AI-Augmented Predictions: LLM Assistants Improve Human Forecasting Accuracy. ACM Trans. Interact. Intell. Syst., 15(1), 4:1-4:25. https://doi.org/10.1145/3707649

[52]: Knack, A., Balakrishnan, N., & Clancy, T. (2025). Applying AI to Strategic Warning. https://cetas.turing.ac.uk/publications/applying-ai-strategic-warning

[53]: Allen, A., Markou, S., Tebbutt, W., Requeima, J., Bruinsma, W. P., Andersson, T. R., Herzog, M., Lane, N. D., Chantry, M., Hosking, J. S., & Turner, R. E. (2025). End-to-end data-driven weather prediction. Nature, 641(8065), 1172–1179. https://doi.org/10.1038/s41586-025-08897-0

[54]: Petropoulos, F., Kourentzes, N., Nikolopoulos, K., & Siemsen, E. (2018). Judgmental selection of forecasting models. Journal of Operations Management, 60, 34–46. https://doi.org/10.1016/j.jom.2018.05.005

[55]: Benjamin et al., Hybrid forecasting of geopolitical events

[56]: Those four variables were regime type, with five levels from full autocracy to full democracy, infant mortality, armed conflict in 4 or more neighboring states, and state-led discrimination.

[57]: Goldstone, J. A., Bates, R. H., Epstein, D. L., Gurr, T. R., Lustik, M. B., Marshall, M. G., Ulfelder, J., & Woodward, M. (2010). A Global Model for Forecasting Political Instability. American Journal of Political Science, 54(1), 190–208.

[58]: Bowlsby, D., Chenoweth, E., Hendrix, C., & Moyer, J. D. (2020). The Future is a Moving Target: Predicting Political Instability. British Journal of Political Science, 50(4), 1405–1417. https://doi.org/10.1017/S0007123418000443

[59]: O’Brien, S. P. (2010). Crisis Early Warning and Decision Support: Contemporary Approaches and Thoughts on Future Research. International Studies Review, 12(1), 87–104. https://doi.org/10.1111/j.1468-2486.2009.00914.x

[60]: The Violence & Impacts Early-Warning System. (n.d.). VIEWS. Retrieved October 11, 2025, from

https://viewsforecasting.org

. The grid squares are based on longitude and latitude. They are 0.5 x 0.5 decimal degrees, about 55 x 55 km at the equator.

[61]: Vesco, P., Hegre, H., Colaresi, M., Jansen, R. B., Lo, A., Reisch, G., & Weidmann, N. B. (2022). United they stand: Findings from an escalation prediction competition. International Interactions, 48(4), 860–896. https://doi.org/10.1080/03050629.2022.2029856

[62]: The Violence & Impacts Early-Warning System. (n.d.). VIEWS. Interactive data visualisation tool. Retrieved November 19, 2025, from

https://data.viewsforecasting.org/

.

[63]: Early Warning Project. Retrieved October 11, 2025, from

https://earlywarningproject.ushmm.org

[64]: Early Warning Project. Statistical Model. Retrieved August 17, 2024, from https://earlywarningproject.ushmm.org/methodology-statistical-model

[65]: Early Warning Project. (2024). Countries at Risk for Intrastate Mass Killing 2024–25: Statistical Risk Assessment Results. https://earlywarningproject.ushmm.org/storage/resources/3186/Early%20Warning%20Project%20Statistical%20Risk%20Assessment%202024-25.pdf

[66]: Early Warning Project. (2024). Summary Handout | Countries at Risk for Intrastate Mass Killing 2024–25: Early Warning Project Statistical Risk Assessment Results. https://earlywarningproject.ushmm.org/reports/summary-handout-countries-at-risk-for-intrastate-mass-killing-2024-25-early-warning-project-statistical-risk-assessment-results

[67]: Fildes, R., Goodwin, P., Lawrence, M., & Nikolopoulos, K. (2009). Effective forecasting and judgmental adjustments: An empirical evaluation and strategies for improvement in supply-chain planning. International Journal of Forecasting, 25(1), 3–23.

[68]: del Pino, J. S. (2024). Why foresight? European Public Mosaic (EPuM). Open Journal on Public Service, p. 27. Others give foresight a broader definition. For example, “The term ‘Foresight’ is often used to refer to the application of specific tools or methods for conducting Futures work.” Government Office for Science. (2021). A brief guide to futures thinking and foresight.

[69]: Government of Canada. (2024). Module 1: Introduction to Foresight. https://horizons.service.canada.ca/en/our-work/learning-materials/foresight-training-manual-module-1-introduction-to-foresight/index.shtml

[70]: Gardner, A. L., Bontoux, L., & Barela, E. (2024). Applying evaluation thinking and practice to foresight evaluation. Association of Professional Futurists.

[71]: Office of Management and Budget. (2024). Circular No. A-11: Preparation, Submission, and Execution of the Budget. https://www.whitehouse.gov/wp-content/uploads/2018/06/a11.pdf

[72]: del Pino, J. S. (2024). Why foresight? European Public Mosaic (EPuM). Open Journal on Public Service, p. 27.

[73]: Rohrbeck, R., Battistella, C., & Huizingh, E. (2015). Corporate foresight: An emerging field with a rich tradition. Technological Forecasting and Social Change, 101, 1–9. https://doi.org/10.1016/j.techfore.2015.11.002; Rohrbeck, R., Kum, M. E., Jissink, T., & Gordon, A. V. (2018). How leading firms build a superior position in markets of the future Corporate Foresight Benchmarking Report. Strategic Foresight Research Network. https://pure.au.dk/ws/files/131917199/Corporate_Foresight_Benchmarking_Report_2018_electronic_version.pdf

[74]: FEMA. (2024). Strategic Foresight 2050 Final Report. https://www.fema.gov/sites/default/files/documents/fema_strategic-foresight-2050-final-report.pdf, p. 5; Government Office for Science, A brief guide to futures thinking and foresight.

[75]: Davenport, A. C., Ziegler, M. D., Resetar, S. A., Savitz, S., Anania, K., Bauman, M., & McDonald, K. (2022). USCG Project Evergreen V: Compilation of Activities and Summary of Results. https://www.rand.org/pubs/research_reports/RRA872-2.html

[76]: Schwarz, J. O. (2023). Strategic foresight: An introductory guide to practice. Routledge; Government of Canada, Module 1

[77]: FEMA, Strategic Foresight 2050 Final Report.

[78]: National Intelligence Council. (2021). Global Trends 2040: A More Contested World. https://www.dni.gov/files/images/globalTrends/GT2040/GlobalTrends_2040_for_web1.pdf

[79]: Government of Canada, Module 1; Government Office for Science, A brief guide to futures thinking and foresight; FEMA, Strategic Foresight 2050 Final Report.

[80]: Government Office for Science. (2024). The Futures Toolkit. https://assets.publishing.service.gov.uk/media/66c4493f057d859c0e8fa778/futures-toolkit-edition-2.pdf

[81]: Government Office for Science, A brief guide to futures thinking and foresight.

[82]: Government Office for Science, The Futures Toolkit

[83]: Government of Canada, Module 1

[84]: Government Office for Science, The Futures Toolkit

[85]: Cuhls, K. E. (2020). Horizon Scanning in Foresight – Why Horizon Scanning is only a part of the game. FUTURES & FORESIGHT SCIENCE, 2(1), e23. https://doi.org/10.1002/ffo2.23; Hines, P., Yu, L. H., Guy, R. H., Brand, A., & Papaluca-Amati, M. (2019). Scanning the horizon: A systematic literature review of methodologies. BMJ Open, 9(5), e026764. https://doi.org/10.1136/bmjopen-2018-026764

[86]: Boden, J. M., Cagnin, C., Carabias, B. V., Haegeman, K., & Könnölä, T.-T. T. (2011). Facing the Future: Time for the EU to meet Global Challenges. https://doi.org/10.2791/4223; Könnölä, T., Salo, A., Cagnin, C., Carabias, V., & Vilkkumaa, E. (2012). Facing the future: Scanning, synthesizing and sense-making in horizon scanning. Science and Public Policy, 39(2), 222–231. https://doi.org/10.1093/scipol/scs021

[87]: FEMA, Strategic Foresight 2050 Final Report; Government of Canada. (2024). Module 6: Scenarios and Results. https://horizons.service.canada.ca/en/our-work/learning-materials/foresight-training-manual-module-6-scenarios-and-results/index.shtml; Government Office for Science, A brief guide to futures thinking and foresight; Schwarz, Strategic foresight

[88]: Barma, N. H., Durbin, B., Lorber, E., & Whitlark, R. E. (2016). “Imagine a World in Which”: Using Scenarios in Political Science 1. International Studies Perspectives, 17(2), 117–135. https://doi.org/10.1093/isp/ekv005

[89]: Government of Canada, Module 6; Government Office for Science, The Futures Toolkit

[90]: FEMA, Strategic Foresight 2050 Final Report; Government Office for Science, The Futures Toolkit

[91]: FEMA, Strategic Foresight 2050 Final Report

[92]: FEMA, Strategic Foresight 2050 Final Report

[93]: Schwarz, Strategic foresight

[94]: Government Office for Science, A brief guide to futures thinking and foresight.

[95]: Lempert, R. J., Popper, S. W., Groves, D. G., Kalra, N., Fischbach, J. R., Bankes, S. C., Bryant, B. P., Collins, M. T., Keller, K., Hackbarth, A., Dixon, L., LaTourrette, T., Reville, R. T., Hall, J. W., Mijere, C., & McInerney, D. J. (2013). Making Good Decisions Without Predictions: Robust Decision Making for Planning Under Deep Uncertainty. https://www.rand.org/pubs/research_briefs/RB9701.html

[96]: Lempert, R. J. (2019). Robust Decision Making (RDM). In V. A. W. J. Marchau, W. E. Walker, P. J. T. M. Bloemen, & S. W. Popper (Eds.), Decision Making under Deep Uncertainty: From Theory to Practice (pp. 23–51). Springer International Publishing. https://doi.org/10.1007/978-3-030-05252-2_2

[97]: Lempert, Robust Decision Making

[98]: SEC. (2010). 1161 Final. Commission Staff Working Document A Rationale for Action, Accompanying the Europe 2020 Flagship Initiative Innovation Union COM(2010) (pp. 9 and 89). http://ec.europa.eu/research/innovation-union/pdf/rationale_en.pdf, as cited in Könnölä et al., Facing the future

[99]: Gardner et al., Applying evaluation thinking and practice to foresight evaluation

[100]: Fergnani, A., & Chermack, T. J. (2021). The resistance to scientific theory in futures and foresight, and what to do about it. Futures & Foresight Science, 3(3–4), e61. https://doi.org/10.1002/ffo2.61

[101]: Rohrbeck, R., & Kum, M. E. (2018). Corporate foresight and its impact on firm performance: A longitudinal analysis. Technological Forecasting and Social Change, 129, 105–116. https://doi.org/10.1016/j.techfore.2017.12.013

[102]: Rohrbeck, R., & Kum, M. E. (2018). Corporate foresight and its impact on firm performance: A longitudinal analysis. Technological Forecasting and Social Change, 129, 105–116. https://doi.org/10.1016/j.techfore.2017.12.013

[103]: Rohrbeck, R., & Schwarz, J. O. (2013). The value contribution of strategic foresight: Insights from an empirical study of large European companies. Technological Forecasting and Social Change, 80(8), 1593–1606. https://doi.org/10.1016/j.techfore.2013.01.004. Percentages are how many responded “yes” vs. “partly”, “no”, or “don’t know”. Top performers are the 12 firms with the highest growth rate in the last three years in the sample; low performers are the 12 with the lowest growth rate.

[104]: Burt, G., & Nair, A. K. (2020). Rigidities of imagination in scenario planning: Strategic foresight through ‘Unlearning.’ Technological Forecasting and Social Change, 153, 119927.

[105]: Schwarz, J. O., Rohrbeck, R., & Wach, B. (2020). Corporate foresight as a microfoundation of dynamic capabilities. Futures & Foresight Science, 2(2), e28. https://doi.org/10.1002/ffo2.28

[106]: Greenblott, J. M., O’Farrell, T., Olson, R., & Burchard, B. (2019). Strategic Foresight in the Federal Government: A Survey of Methods, Resources, and Institutional Arrangements. World Futures Review, 11(3), 245–266. https://doi.org/10.1177/1946756718814908; Scoblic, J. P. (2021). Strategic Foresight in U.S. Agencies. New America. http://newamerica.org/future-security/reports/strategic-foresight-in-us-agencies/

[107]: Foreign Affairs Manual § 090 Vol. 2. (2024). https://fam.state.gov/FAM/02FAM/02FAM0090.html

[108]: After Action Review on Afghanistan. (2022). U.S. Department of State. https://www.state.gov/wp-content/uploads/2023/06/State-AAR-AFG.pdf; Blinken, A. (2024, October 30). American Diplomacy for a New Era. https://geneva.usmission.gov/2024/10/30/american-diplomacy-for-a-new-era/

[109]: Office of Personnel Management (OPM). (2022). Developing & Applying Strategic Foresight for Better Human Capital Management. Government Printing Office. https://www.opm.gov/policy-data-oversight/human-capital-management/strategic-foresight/

[110]: Scoblic, Strategic Foresight in U.S. Agencies

[111]: Scoblic, Strategic Foresight in U.S. Agencies

[112]: Rijkens-Klomp, N., & Van Der Duin, P. (2014). Evaluating local and national public foresight studies from a user perspective. Futures, 59, 18–26. https://doi.org/10.1016/j.futures.2014.01.010

[113]: Menzies, M. B., & Middleton, L. (2019). Evaluating Health Futures in Aotearoa. World Futures Review, 11(4), 379–395. https://doi.org/10.1177/1946756719862114

[114]: Phadnis, S., Caplice, C., Sheffi, Y., & Singh, M. (2015). Effect of scenario planning on field experts’ judgment of long-range investment decisions: Effect of Scenario Planning. Strategic Management Journal, 36(9), 1401–1411. https://doi.org/10.1002/smj.2293

[115]: Totin, E., Butler, J. R., Sidibé, A., Partey, S., Thornton, P. K., & Tabo, R. (2018). Can scenario planning catalyse transformational change? Evaluating a climate change policy case study in Mali. Futures, 96, 44–56. https://doi.org/10.1016/j.futures.2017.11.005

[116]: Association of Professional Futurists. (2022). APF Foresight Evaluation Task Force Report: Building field and foresight practitioner evaluation capacity. https://www.apf.org/product-page/apf-foresight-evaluation-task-force-report

[117]: Chen, K.-H., & Hsu, L.-P. (2020). Visioning the future: Evaluating learning outcomes and impacts of futures-oriented education. Journal of Futures Studies, 24(4), 103–116. Results are based on an analysis ofbased on 20-questions divided into 5 factors. Differences for individual questions are unreported.

[118]: Mengel, T. (2019). Learning Portfolios as Means of Evaluating Futures Learning: A Case Study at Renaissance College. World Futures Review, 11(4), 360–378. https://doi.org/10.1177/1946756719851522

[119]: Association of Professional Futurists. (2022). APF Foresight Evaluation Task Force Report: Building field and foresight practitioner evaluation capacity. https://www.apf.org/product-page/apf-foresight-evaluation-task-force-report

[120]: Gardner et al., Applying evaluation thinking and practice to foresight evaluation

[121]: Münch, C., & von der Gracht, H. A. (2021). A bibliometric review of scientific theory in futures and foresight: A commentary on Fergnani and Chermack 2021. Futures & Foresight Science, 3(3–4), e88; Piirainen, K. A., & Gonzalez, R. A. (2015). Theory of and within foresight—“What does a theory of foresight even mean?” Technological Forecasting and Social Change, 96, 191–201.

[122]: Hodgkinson, G. P., & Healey, M. P. (2008). Toward a (Pragmatic) Science of Strategic Intervention: Design Propositions for Scenario Planning. Organization Studies, 29(3), 435–457. https://doi.org/10.1177/0170840607088022

[123]: Fergnani, A., & Chermack, The resistance to scientific theory in futures and foresight

[124]: Hodgkinson, G. P. (2021). Why appealing to the virtues of scientific theory (and method) is necessary but insufficient for effecting systemic change: Commentary on Fergnani & Chermack, 2021. Futures & Foresight Science, 3(3–4), e79. https://doi.org/10.1002/ffo2.79

[125]: Mandel, D. R. (2021). A positive future for futures and foresight science needs fierce competition in the marketplace of ideas: A commentary on Fergnani and Chermack 2021. FUTURES & FORESIGHT SCIENCE, 3(3–4), e67. https://doi.org/10.1002/ffo2.67

[126]: Samotin, L. R., Friedman, J. A., & Horowitz, M. C. (2023). Obstacles to harnessing analytic innovations in foreign policy analysis: A case study of crowdsourcing in the U.S. intelligence community. Intelligence and National Security, 38(4), 558–575. https://doi.org/10.1080/02684527.2022.2142352

[127]: Spokojny, D. (2022, January 26). Forecasting in Policymaking: Beyond Cassandra. fp21. https://www.fp21.org/publications/forecasting-in-policymaking-beyond-cassandra

[128]: Spokojny, D. (2022, January 26). Forecasting in Policymaking: Beyond Cassandra. fp21. https://www.fp21.org/publications/forecasting-in-policymaking-beyond-cassandra

[129]: Gong, M., Lempert, R., Parker, A., Mayer, L. A., Fischbach, J., Sisco, M., Mao, Z., Krantz, D. H., & Kunreuther, H. (2017). Testing the scenario hypothesis: An experimental comparison of scenarios and forecasts for decision support in a complex decision environment. Environmental Modelling & Software, 91, 135–155. https://doi.org/10.1016/j.envsoft.2017.02.002

[130]: Scoblic, J. P., & Tetlock, P. E. (2020, October 13). A Better Crystal Ball. Foreign Affairs, 99(6). https://www.foreignaffairs.com/articles/united-states/2020-10-13/better-crystal-ball

[131]: Havas, A., Schartinger, D., & Weber, M. (2010). The impact of foresight on innovation policy-making: Recent experiences and future perspectives. Research Evaluation, 19(2), 91–104.

[132]: Government of Canada, Module 1

Dear readers,

Sanctions have become one of the go-to instruments of U.S. foreign policy, but the government doesn’t really know if they work. They might even be backfiring. This new paper from my colleague Amanda Kadlec exposes how the sanctions evaluation processes are fragmented and weak. Policymakers should equip themselves with rigorous systems of feedback that evaluate the impact of sanctions. Anything less is like practicing archery in the dark. Expertise cannot form under such conditions.

Enjoy,

Dan Spokojny

Will the Russian Sanctions Work?

By: Amanda B. Kadlec

The latest round of U.S. sanctions on Russia – this time targeting the country’s largest oil producers, Rosneft and Lukoil– was unexpected. According to the Treasury Department, this move is intended to punish President Vladimir Putin for refusing to agree to a ceasefire in Ukraine.[1]

But will this added measure have any effect on Putin’s decision to end the war? His government has faced a staggering sanctions regime thus far, but the sanctions have not only failed to achieve their intended effect but may be creating unintended consequences as Putin has sought deeper alliances with U.S. adversaries like North Korea and China.

In an ideal policy environment, the administration would have rigorously tested the likely impact of the policy instrument before implementing the measures. Best and worst-case scenarios would have been mapped and evaluated. Clear lines of responsibility would have been established if the actual sanctions failed to produce the intended result. Unfortunately, neither Trump nor other recent administrations have developed this ideal policy environment, resulting in the appearance of sanctions being deployed in haphazard, ad hoc, and perhaps counterproductive ways.

These measures from the Trump administration are part of a fourfold increase in the use of sanctions over the past decade, contributing to a sharp rise since the fall of the Soviet Union in the 1990s. ^[2] This has prompted a raft of quantitative and qualitative academic studies exploring their effectiveness and downstream effects.^[3] Many scholars now argue that the tool has been used in excess and with limited success.^[4]

Despite this, the government has only the roughest idea of how this tool works. Government-led evaluations of sanctions remain siloed, narrow in scope, and infrequent. From one administration to the next, few attempts have been made to assess sanctions’ effectiveness in any comprehensive way – either in acute impact or broad policy terms – despite 32 active sanctions programs and over 20,000 listed individuals.^[5]

Moreover, today’s multipolar environment threatens the primacy of the dollar as the cornerstone of the international system for the first time since the post-World War II era. Powerful players and shadow economies offer competing financial structures to challenge the existing U.S.-led international system. Are policymakers prepared for the possibility that sanctions may no longer act as the formidable threat and alternative to war that they once did, and adjust processes accordingly? Might sanctions be precipitating the decline of the U.S.-led system as opposing governments?

This paper reviews the literature on the effectiveness and impacts of sanctions. It also evaluates the process by which sanctions policy is created. Might the many arms involved in the development of sanctions and assessment —the National Security Council, Intelligence Community, Departments of State, Treasury, Commerce, and Defense, and Congress —be empowered with a more rigorous process to determine when and how sanctions might be used to their greatest effect?

An Alternative to War

Economic sanctions emerged as an alternative to military power during the First World War when France and Britain waged “economic war” on Germany through blockade and dominance of global trade.^[6] Following WWII, the United States emerged as the dominant economic, political, and military hegemon, able to wield unprecedented power to inflict financial and economic pain on adversaries.

Modern sanctions may be levied to coerce or counter specific behaviors, ranging from terrorism or organized crime to nuclear weapons development or military aggression. Among these, human rights and democracy standards are the most frequently invoked justifications. While this use category had earlier precedent, such as against apartheid South Africa and Zimbabwe, the congressionally led Magnitsky Act in 2012, which initially targeted Russian individuals implicated in the torture and murder of Kremlin whistleblower Sergei Magnitsky, expanded in 2016 to include a global mandate for individualized sanctions. Other recurring objectives include curbs to terrorism and transnational organized crime or strategic threats on the part of certain players or states like Iran or North Korea.^[7]

Sanctions also range widely in type, function, target, time horizon, and purpose. Economic, financial, and trade sanctions are most frequently employed, and can include asset freezes, import-export blocks and banking restrictions, and sector-specific embargos (or analogous measures like the oil price cap), like those applied to energy markets. They may target specific individuals, companies, sectors, and states. Broadly defined, the purpose of such measures may be defined as “the deliberate, government-inspired withdrawal, or threat of withdrawal, of customary trade or financial relations” in response to conduct deemed sanctionable by the issuing authority.^[8]

With over half of international banking still conducted in U.S. dollars,^[9] the mere threat of being excluded from the U.S. financial system has often proved more coercive than the sanctions themselves.^[10] Yet in today’s multipolar world, where alternative financial networks and avenues for evasion abound, this leverage is weakening. Despite this, sanctions measures are being implemented more frequently, not less. Within government, they are perceived as a low-cost, low-stakes policy instrument relative to costlier and riskier tools like espionage or military action. Their principal function seems to be to signal disapproval or resolve, to disrupt a pattern of behavior rather than compel a full-scale behavior change, or to fulfill a broader policy goal. These are limitations that policymakers may even acknowledge.^[11] Precisely because they are seen as minimally risky yet symbolically powerful, sanctions have attracted far less institutional oversight and review across government compared to other foreign policy tools.

The development and implementation of sanctions necessitate coordination of multiple agencies – any one of which might be the starting point for targeting and strategy. Legal authority for sanctions predominantly derives from the International Emergency Economic Powers Act (IEEPA), which empowers the President to impose sanctions unilaterally and with broad discretion.^[12] Consideration of punitive measures is therefore almost always initiated by the executive branch at the direction of the President upon NSC recommendation.^[13] These recommendations may be based upon assessments provided by the State Department and intelligence communities – typically upon request by the NSC that outline the target, objective, and expected impact, and which most often focus on specific countries like Russia or Iran. ^[14] This interagency process is facilitated through interagency working groups directed by the Office of Foreign Assets Control (OFAC) at Treasury, which serves as the legal implementer. Other agencies may house their own bodies tasked with improving cross-government communication, such as the State Department’s Office of Sanctions Coordination.^[15] At other times, however, these two agencies might initiate targeting measures in collaboration with the NSC. While the sanctions development process is relatively established, it is chaotic and still lacks comprehensive pre-implementation analysis of potential effects. A structured evaluation process after they are in place remains absent.

Evaluating Sanctions Measures

There is no regular, comprehensive, or uniform assessment of sanctions effectiveness in government. As administrations transition, the relevance of existing listings can simply fade with shifting priorities, and responsibility for reviewing or reforming prior sanctions diminishes. At the same time, the capacity of subsequent policymakers to alter or “de-list” entities as targets is legally complex and arduous. There is no established review process.^[16] The lack of institutional memory and political will to revisit entrenched sanctions regimes means that negative economic and humanitarian impacts are often underappreciated or ignored.^[17] The collapse of Venezuela’s oil economy and the resulting migration pressures on neighboring countries, including the U.S., may be an example of failing to fully consider the second-order effects of crippling economic sanctions.

The sanctions power granted to presidents is weakly constrained: they wield a unilateral power to impose sanctions swiftly and flexibly, yet they are under no duty to ensure they align with broader U.S. interests. Moreover, the institutional mechanism to encourage the evaluation of their effectiveness or the potential consequences once policies are enacted is lacking. The IEEPA does not require mandatory Congressional notification or approval, and oversight mechanisms within Congress remain both limited and unacknowledged as a necessity.^[18]

Executive orders delineate sanctions objectives, but often in vague or shifting terms influenced by evolving political messaging from multiple officials. The Biden Administration’s 2021 E.O. 14024^[19] ordering sanctions on the Russian Federation is one example. It names “harmful foreign activities” – ranging from cyber operations and corruption to territorial aggression and efforts to undermine U.S. elections – placing disparate behaviors under one authority. This lack of clarity around policy goals complicates evaluation efforts as there is no baseline against which to measure success or failure. Consequently, any assessment would struggle to determine whether sanctions meaningfully disrupt or alter target behavior or advance U.S. foreign policy aims, especially when outcomes are diffuse or indirect. Modern sanctions may be levied to coerce or counter specific behaviors, ranging from terrorism or organized crime to nuclear weapons development or military aggression. Among these, human rights and democracy standards are the most frequently invoked bases. Strategic threats emanating from players or states like Iran or North Korea are also targeted by sanctions.^[20]

While the President, the NSC, and Congress can call to impose sanctions, the Department of State typically holds the prerogative to assess their effect on foreign policy and national security objectives.^[21] This assessment can be conducted in consultation with Treasury, Commerce, and the Intelligence Community. Yet while such assessments are available, they are not standard practice, partly because other agencies will move to push through a measure without much consultation, but also because State lacks the dedicated offices to do so. A 2019 Government Accountability Office (GAO) study found that independent agencies “cited difficulties in assessing sanctions’ effectiveness in meeting broader U.S. policy goals… their agencies have not conducted such assessments on their own.”^[22] According to staff involved in sanctions processes at Treasury, State, and within the IC, post-implementation evaluations are typically made only upon request from within the respective agency and are focused on the immediate effect of the sanction (e.g., impact on Russia’s economy) rather than the effect on behavior (deterrence of military aggression).^[23]

The institutional landscape dedicated to sanctions evaluation is fractured and lacking capacity. Different agencies have contributed different support to reviewing sanctions. The State Department’s Bureau of Intelligence and Research (INR) and Treasury’s Office of Intelligence Analysis on identifying targets and detecting evasion^[24], while the CIA and the Office of the Director of National Intelligence (ODNI) supply additional operational intelligence.^[25] Treasury and Commerce each maintain units that investigate sanctions using a mix of all-source intelligence, projections, and simulations, with some interagency coordination, but the evaluations tend to be limited to direct financial impacts rather than political outcomes. The Department of Commerce lacks sufficient in-house analytical capacity and often contracts external consultants. Sanctions evaluation efforts at State are fragmented between offices and dependent on exceptional political will, such as with the Russia sanctions surge under the Biden administration.^[26]

Within Treasury, the Office of Foreign Assets Control (OFAC) leads most targeting and designations, but also enforcement and compliance. But its mission has historically centered on implementation and narrow assessments rather than strategic evaluation. The Biden administration stood up the Sanctions Economic Analysis Division office within OFAC to assess microeconomic impacts and facilitate mitigation measures.^[27] While in principle its presence was a first step toward an improved process, its focus on economic analysis and discreet impact often misaligned with the State Department’s focus on broader political and strategic outcomes.^[28] Moreover, due to its placement as an evaluating body subordinate to implementing offices – and constrained by limited financial and resources - the independence and oversight potential of the new division may have been curtailed.

The most recent Treasury Department sanctions review in 2021– the first publicly released, comprehensive call for improved evaluations within that agency–outlined several key questions for policymakers to consider pre- and post-implementation^[29]:

1) Do the listings impart the desired first-order intended effect (e.g., expected negative impact on the Russian economy)? If so, for how long (e.g., do those effects weaken over time)?

2) Do those measures have unintended, negative localized consequences (e.g., price inflation in Europe)?

3) Do they produce the preferred and clearly identified policy outcome (e.g., Russian withdrawal from Ukraine, improved U.S. negotiating position)?

4) Do those sanctions end up having unintended geopolitical consequences (e.g., Russia’s closer alliance with China, expansion of BRICS to circumvent effects)?

In practice, agency offices that conduct post-implementation assessments focus almost exclusively on the first two questions – if such an evaluation takes place at all. Missing is the comprehensive strategic consideration reflected in questions three and four, which concern the primary purpose of the sanctions measure. While some attention is given to how overlapping U.S. and multilateral sanctions regimes interact, conclusions are not necessarily data- or consensus-driven.^[30] Critically, despite the Treasury report and other findings, there is no overarching mechanism within any branch or agency to evaluate these themes comprehensively.^[31] The biggest hurdle to this effort may be that agencies involved in the development process are not required to determine the effectiveness, leaving an accountability gap.

The Current Environment

Despite widespread recognition of the increasingly central role sanctions are playing in US foreign policy, reforms across administrations have yet to yield more consistent or rigorous analysis of sanctions effectiveness. Writing in the midst of a government shutdown and on the heels of widespread staffing cuts across federal agencies, any recommendations for new offices, analysts, and interagency processes will likely fall on deaf ears. Still, advancements can be made even with limited resources by prioritizing sanctions as a policy area worthy of systemic and analytical attention.

Following Treasury’s 2021 review, the Biden administration induced several interagency changes to attempt to address these gaps. Secretary of State Anthony Blinken reinstated the Office of Sanctions Coordination at the Department of State, which had been dismantled during the first Trump administration. Although the subsequent reorganizations under Secretary of State Marco Rubio and the Department of Government Efficiency (DOGE) have downsized and restructured most of the government, the Office of Sanctions Coordination at State remains in place and could implement an annual review process. The State Department’s regional bureaus, because of their deep country-specific expertise, have had a prominent role in policy planning, implementation, and evaluation. These offices have taken increased prominence under the current administration, managing expansive portfolios previously handled by thematic offices. Despite this, regional bureaus have nevertheless been reduced in their power to effect sanctions policy.^[32] All the while, policymakers and implementers across agencies still face the challenge of having to respond to fast-changing global events and shifting administration goals with limited evaluative tools or processes.

Feasible options are available to address the challenges introduced by these changes while still improving evaluation processes. Experts suggest that despite workforce reductions, Treasury can strengthen its ability to evaluate sanctions by placing an evaluations division under the Office of the Chief Economist, or by establishing a distinct unit separate from OFAC’s enforcement function to preserve analytical rigor and autonomy.^[33] At State, regional bureaus could have an annual, committed sanctions review process that provides assessments to the Coordination Office.^[34] Integrating sanctions evaluation into a regular policy review cycle would help to normalize it as a standard, ongoing practice rather than an ad hoc response. Practical techniques such as scenario planning – otherwise known as ‘war gaming’ or ‘red-teaming’ – are also available to support this effort. These tools are widely used in defense to test assumptions and anticipate responses in domains ranging from cyber-attacks to natural disasters to military confrontation, yet are rarely applied to sanctions policy. Emerging applications of artificial intelligence now enable rapid modeling of possible trajectories that human expertise can assess and refine, offering a cost-effective option to strengthen sanctions analysis and foresight without adding cost and bureaucratic layers.

Normalizing Evaluations Processes

The current U.S. sanctions system privileges sanctions implementation and enforcement over ensuring effectiveness and alignment with broader foreign policy strategy. Analytical review remains an ad hoc, outsourced function, leaving interagency staff without a clear understanding of when or why sanctions succeed in their stated goal. Indeed, the goals of sanctions are often unclear, rendering any attempt to evaluate them aimless. Without a concerted effort to normalize regular evaluation, sanctions will continue to fall short in achieving their targeted objectives while potentially producing additional obstacles to our broader foreign policy.^[35]

This weak evaluative process has created a structural disconnect between the use of sanctions and their actual impact. Instead, sanctions are often deployed for expediency rather than to achieve a clearly defined, measurable outcome. In a world of accelerating multipolar competition and proliferating alternative financial systems, such inefficiency carries policy risk.

To remain a useful instrument of statecraft, sanctions policy must evolve from a narrow and reactive posture to a more reflective and comprehensive one. Moreover, sanctions evaluation should be perceived not as a policy afterthought, but as the foundational policy instrument that it has become, warranting deeper scrutiny. Establishing this mindset would not require vast new spending or offices, but rather a modest shift in priorities – to embed systemic review within existing structures, foster analytical independence, and align sanctions policy more closely with the broader U.S. interests they aim to serve.

[1] Treasury Department. “Treasury Sanctions Major Russian Oil Companies, Calls on Moscow to Immediately Agree to Ceasefire. October 22, 2025. https://home.treasury.gov/news/press-releases/sb0290

^[2] Felbermayr, Gabriel, T. Clifton Morgan, Constantinos Syropoulos, and Yoto V. Yotov. Economic Sanctions: Stylized Facts and Quantitative Evidence. Working Paper 2024001. Center for Global Policy Analysis, LeBow College of Business, Drexel University, December 2024.
https://www.lebow.drexel.edu/sites/default/files/2024-12/wp2024001-economic-sanctions.pdf.

^[3] Felbermayr, Gabriel, T. Clifton Morgan, Constantinos Syropoulos, and Yoto V. Yotov. Economic Sanctions: Stylized Facts and Quantitative Evidence. Working Paper 2024001. Center for Global Policy Analysis, LeBow College of Business, Drexel University, December 2024.
https://www.lebow.drexel.edu/sites/default/files/2024-12/wp2024001-economic-sanctions.pdf.

^[4] Drezner, Daniel W. “Sanctions Sometimes Smart: Targeted Sanctions in Theory and Practice.” International Studies Review 13, no. 1 (2011): 96–108.

^[5] As of 2020, OFAC reported over 17,000 listed individuals. CNAS reported another 2,502 in 2023 and 3,135 in 2024, taking this total to well over 20,000. https://ofac.treasury.gov/sanctions-programs-and-country-information/where-is-ofacs-country-list-what-countries-do-i-need-to-worry-about-in-terms-of-us-sanctions

https://www.cnas.org/publications/reports/sanctions-by-the-numbers-2024-year-in-review

^[6] Mulder, Nicholas. The Economic Weapon: The Rise of Sanctions as a Modern Tool of War. New Haven, CT: Yale University Press, 2022. Chapter 1, “The Origin of the Economic Weapon,” 1–24.

^[7] Felbermayr, Gabriel, et al. “Economic Sanctions: Stylized Facts and Quantitative Evidence.” CGPA Working Paper, January 2024.
https://www.lebow.drexel.edu/sites/default/files/2024-12/wp2024001-economic-sanctions.pdf

^[8] (Tariff measures may also involve disruptions to these customary relations, they are typically applied to address trade imbalances or domestic economic concerns, whereas sanctions are meant to be punitive). Hufbauer, Gary Clyde, Jeffrey J. Schott, Kimberly Ann Elliott, and Barbara Oegg. Economic Sanctions Reconsidered. 3rd ed. Washington, DC: Peterson Institute for International Economics, 2009.

^[9] Boocker, Sam and David Wessel. “The Changing Role of the U.S. Dollar.” Brookings Institution, August 23, 2024. https://www.brookings.edu/articles/the-changing-role-of-the-us-dollar/

^[10] Weber, Patrick M., and Gerald Schneider. “How Many Hands to Make Sanctions Work? Comparing EU and US Sanctioning Efforts.” European Economic Review 130 (2020): Article 103595. https://doi.org/10.1016/j.euroecorev.2020.103595.

^[11] Brooks-Rubin, Brad. “The Misunderstood Use of Economic Sanctions.” August 11, 2025. https://newlinesmag.com/argument/the-misunderstood-purpose-of-economic-sanctions/

^[12] The International Emergency Economic Powers Act: Origins, Evolution, and Use. September 1, 2025 (Accessed October 8, 2025). https://www.congress.gov/crs-product/R45618

^[13] Congress can also make its own inquiries and pass legislation, as with the Magnitsky Act for example, but it typically plays a deferential role.

^[14] U.S. Government Accountability Office (GAO). “Economic Sanctions: Agencies Assess Impacts on Targets, and Studies Suggest Several Factors Contribute to Sanctions’ Effectiveness.” https://www.gao.gov/assets/gao-20-145.pdf.

^[15] U.S. Congress. House. House Financial Services Committee on Monetary Policy and Trade, Hearing on “Evaluating the Effectiveness of U.S. Sanctions Programs,” 115th Cong., November 30, 2017.
https://www.congress.gov/event/115th-congress/house-event/106687/text.

^[16] Interview with foreign service officer, by video call, February 21, 2025.

International Crisis Group. “Sanctions, Peacemaking, and Reform: Recommendations for U.S. Policymakers.” Crisis Group Report, April 2024.
https://www.crisisgroup.org/united-states/8-sanctions-peacemaking-and-reform-recommendations-us-policymakers.

^[17] Interview with former senior Treasury official, 5 March, 2025.

^[18] Congressional Research Service. “The International Emergency Economic Powers Act (IEEPA).” CRS Report R45618, updated March 27, 2024.
https://www.congress.gov/crs-product/R45618

^[19] Executive Order 14024 of Aprill 15, 2021. Federal Register, Vol 86, No. 73, Monday, April 19, 2021. (Accessed October 8, 2025). https://ofac.treasury.gov/media/57936/download?inline

^[20] Felbermayr, Gabriel, et al. “Economic Sanctions: Stylized Facts and Quantitative Evidence.” CGPA Working Paper, January 2024.
https://www.lebow.drexel.edu/sites/default/files/2024-12/wp2024001-economic-sanctions.pdf

^[21] U.S. Government Accountability Office (GAO). “Economic Sanctions: Agencies Assess Impacts on Targets, and Studies Suggest Several Factors Contribute to Sanctions’ Effectiveness.” October 2, 2019.
https://www.gao.gov/products/gao-20-145.

^[22] Ibid.

^[23] Interview with Foreign Service Officer, by video call, February 21, 2025.

^[24] U.S. Congress. House. House Financial Services Committee on Monetary Policy and Trade, Hearing on “Evaluating the Effectiveness of U.S. Sanctions Programs,” 115th Cong., November 30, 2017.
https://www.congress.gov/event/115th-congress/house-event/106687/text.

^[25] Interview with former IC officer. 18 March 2025 by video call.

^[26] Interview with foreign service officer, by video call, February 21, 2025.

^[27] Interview with former senior Treasury official, 5 March, 2025.

^[28] Interview with former senior State Department official, 7 October 2025.

^[29] U.S. Department of the Treasury. The Treasury 2021 Sanctions Review. Washington, DC: October 2021.
https://home.treasury.gov/system/files/136/Treasury-2021-sanctions-review.pdf.

^[30] Interview with former senior Treasury official, March 5, 2025.

^[31] Ibid.

^[32] Interview with former State Department Official, 26 September 2025.

^[33] Ibid.

^[34] Ibid.

^[35] International Crisis Group. “Sanctions, Peacemaking, and Reform: Recommendations for U.S. Policymakers.” Crisis Group Report, April 2024. https://www.crisisgroup.org/united-states/8-sanctions-peacemaking-and-reform-recommendations-us-policymakers

I’ve been writing for years about the science of expertise in foreign policy. This began with my organization’s cornerstone report from 2022, entitled “Less Art, More Science,” which has become an unofficial slogan for fp21. But our intention has never been to eliminate art from foreign policy. Instead, our mission is to demonstrate that foreign policy requires both art and science — and to find the proper balance between the two.

I spend most of my energy developing ideas to advance the science of foreign policy, but I thought it would be a useful exercise to flesh out and explain what many senior leaders call “the art of foreign policy.”

International affairs is an intricate tapestry woven from history, culture, and power politics. Navigating this environment requires cold, hard calculations. But it also requires a strong moral compass, creativity, a keen sense of politics, and a knack for building trusted relationships across cultures.

Just as artists interpret the world through idiosyncratic perspectives, skilled policymakers must translate their nuanced view of reality into actionable strategies.

Let's delve deeper into four key areas where the art of foreign policy must shine bright:

1. Considering Morals: A masterpiece reflects not just the artist's skill, but their vision and values. Foreign policy fundamentally requires moral considerations. Policy practitioners must balance light and shadow, carefully navigating the ethical complexities of international relations. This may involve recognizing the human cost of decisions, weighing the distributive consequences across different populations, and striving for solutions that uphold core values even in the face of trade-offs. This requires moral clarity, empathy for the diverse stakeholders involved, and the courage to make difficult choices. 

2. Defining National Interests: Every masterpiece starts with a vision. In foreign policy, this vision begins with defining one’s national interests. This requires a deep understanding of our country’s culture, ethics, and politics. National interests emerge from a complex mosaic of security, economic, and normative priorities. These priorities constantly shift as our society evolves. Being a good leader in a democracy means listening to the public as much as to one’s own intuition. Just as there’s no objectively “correct” morals, there’s no way to define national interests objectively. Policymakers must be able to identify these priorities, understand their interrelationships, and prioritize them strategically. They must, like artists, make informed decisions about where to apply fresh paint, and which brushstrokes to use. 

3. Setting Goals: The world is not a coloring book with pre-defined lines. Just as an artist explores an infinite variety of colors, mediums, and forms, foreign policy demands creative solutions. Studying the masters helps improve one’s craft, but rarely does the world invite photocopied solutions. In a constantly evolving global landscape, one must constantly think outside of the box. This requires intellectual agility, the courage to challenge orthodoxy, and the willingness to embrace the unexpected. “Scenario analysis,” explains the scholar Naazneen Barma, “constitutes the art of juxtaposing current trends in unexpected combinations in order to articulate surprising and yet plausible futures, often referred to as ‘alternative worlds.” This creative, artistic act lies at the heart of policymaking.

4. Interpersonal Skills: At its core, foreign policy is about people. There is an art to building relationships, forging alliances, and establishing trust and credibility. The foreign policy practitioner must be a skilled communicator, and their words must be carefully chosen to bridge cultural divides — whether foreign or domestic. They must possess a deep empathy allies and opponents alike, and the ability to appreciate the nuances of human emotions and motivations. This requires emotional intelligence, cultural sensitivity, and the ability to build genuine connections across diverse cultures and perspectives. It demands the adaptability of a chameleon, seamlessly shifting communication styles to resonate with different audiences. One must also be a good manager, which draws upon a similar set of interpersonal skills.

With these four brushstrokes, practitioners can master the art of foreign policy, shaping the international landscape with creativity, empathy, and a deep understanding of human relationships. Art is the foundation of foreign policy.

Art and Science Must Work Together

Yet there’s great risk when artistry is allowed to crowd out the science of foreign policy. Artistry devoid of scientific technique leads to policies based on wishful thinking and personal biases, lacking the foundation of rigorous analysis and realistic planning. This results in wasted resources and ineffective strategies.

For instance, it was widely agreed that Saddam Hussein was a despotic leader and a scourge on his people. Many believed that it was in the US national interest to remove him from power. The implications of beliefs, however, are more empirical questions: How large an invading army would it take to overthrow him? What resources would it require to rebuild Iraq?

Whereas the art of foreign policy is creative, scientific approaches discipline policymakers to evaluate the implications of their worldview objectively, design implementation plans likely to achieve their goals, and accumulate skill over time by continually learning from their successes and failures. Brilliant ideas are meaningless without realistic plans to achieve results. It is for these reasons that we so passionately advocate that foreign policy must be evidence-based.

Imagine an architect designing a masterpiece. They possess technical expertise in engineering and structure and wield a deep understanding of the properties of the materials they work with (science). But they also infuse their creation with creativity, emotion, and a unique vision for how their space will be experienced (art). Similarly, effective foreign policy requires both the rigor of science and the human touch of art.

By recognizing the complementary roles of art and science and harnessing them in tandem, policymakers can create effective strategies that aspire to our highest ideals and are at the same time grounded in hard reality. This is the key to navigating the complex and ever-changing landscape of international relations and building a more just, secure, and prosperous future for all.

Ask anyone working at the U.S. State Department to name the most innovative, forward-looking officials in Foggy Bottom, and you’ll probably hear the name Paul Kruchoski. As the Director of Policy, Planning and Resources in the Undersecretariat of Public Diplomacy and Public Affairs (R/PPR), Paul initiated and led many exciting efforts to upgrade the practice of public diplomacy, many of which carry lessons for the broader conduct of U.S. foreign policy. He left government in June of this year, at which point I was excited to chat with him about his work “on the record.” This conversation has been lightly edited for clarity:

Dan: Can you orient us to the “old model” of public diplomacy, before PD Foundations, before R/PPR. What’s working? How is the work done? What are the weaknesses of the old model?

Paul: I generally peg “new PD” as starting in about 2011/2012. PD was oriented around programs or press engagement, and success was measured by outputs: number of people reached through media, number of program participants. This had a range of weaknesses, including that programs were generally thematically aligned but didn’t address specific policy problems. That weakened the argument for PD as a relevant tool; it also led many in Congress to question whether PD was a good value for the money.

“New PD” was an effort in the Bureau of International Information Programs and R/PPR to reorient people toward “campaigns”: simultaneous deployment of different tools to reach specific audiences and achieve near-term policy outcomes. This approach had worked for some EUR posts, and it was time to go global.

None of our structures, processes, or tools really supported this. Starting in about 2014, R/PPR’s orientation changed from resource management to organizational transformation.

Can you explain what you mean by “none of the structures, processes, or tools really supported this new campaign model?” Why couldn’t leadership just snap their fingers and say “this is how we’re going to do things now?”

This is a bit of an exaggeration, but it helps you see the point: State was largely built in a Taylor-esque organizational chart. Teams had a scope of duties, and they were expected to be largely self-sufficient. Team members each had an assigned role, with the ability to direct what was in their scope, subject to direction from their boss vertically. Communication and feedback generally go up, then come back down via silos to staff. If you have been in the position of running a program for 15 years on your own, you probably have a set of ways you solicit input and your own preferences for pacing and program rollouts. Maybe you got some input from your boss, but you could shape it in predictable ways.

In a campaign model, you have to collaborate more horizontally. Now, all of a sudden, you’re being asked to adapt your program so that you primarily align it with a set of shared goals (that you may set together), and you plan your work so that the timing aligns well with others. That is a huge change – and a lot of it requires talking with other people: to agree on goals, to agree on audiences, to agree on timing and sequencing of programs.

While we believe the campaign model will have more impact – and even enable you to do fewer activities to get the same or better effects – on the surface, it looks like a lot of new work.

Collaboration is also a skill. It takes work to figure out how to negotiate and navigate shared ownership of work with others. It’s even harder in a cross-cultural context where different team members may have more or less conflict aversion, explicit/implicit communication styles, and much more.

Can you help me understand how you went about addressing these shortcomings and opportunities? What do you see as the biggest changes that were seeded during your tenure?

By 2019, we had a PD Modernization effort underway, which focused on reorienting toward a model of policy-driven, audience-focused initiatives that deliver measurable results. It had four big paradigm shifts embedded in it:

1. PD should deliver on specific policy goals in definable ways.

2. PD should focus on audiences that matter to the policy goal and can have their attitudes or behaviors changed.

3. PD should use appropriate PD tools in tandem (e.g., press and programs together).

4. PD results should be defined, measured, and used to improve over time.

PD Modernization was really about delivering what practitioners needed to make these changes: structures, skills, and tools.

PD Foundations was the tangible expression of that new model. It addressed two other big problems I saw. PD had dozens of conceptual models it used to teach and relay information. R/PPR alone had promulgated five different strategic planning frameworks over the previous decade, never officially decommissioning any of them. PD also had a lot of folk wisdom – things that people believed or shared that may or may not be true. We drafted PD Foundations to consolidate the state of the art and to give us a single primary working model across the community. We drafted it laboriously over about two years, with multiple rounds of input from field practitioners, some scholars, and our staff in R/PPR.

My favorite definition of “doctrine” comes from the US Army: “the approved body of knowledge that is taught.” Is that an apt description for what you undertook? Can you say more about what comprised PD Foundations?

Yes, it is actually one of the definitions we use, and a major reason why we worked so closely with the Foreign Service Institute to ensure that their courses align with it!

We envisioned PD Foundations in three major parts, two of which are published.

The first publication (Principles of PD) focuses on defining what public diplomacy is conceptually, how it fits together with defense, military, and economic power (in the DIME model), and what its general goals are. It breaks down the two general mechanisms by which PD operates: shaping information environments and how they operate; and changing specific attitudes and behaviors. It also describes the programmatic mechanisms (exchanges, press engagement, information campaigns, international broadcasting, etc.) and the causal logic behind them.

The second publication (PD in Practice) focuses on how modern public diplomacy is done–the arc from identifying a policy goal to designing and executing an initiative: articulating a realistic policy objective, then identifying the audiences that matter most for achieving that objective, and then creating activities that are likely to affect attitudes and behaviors necessary to achieve that goal. This is where we introduce ideas like logic models and key metrics that you can use to test whether an intervention works.

There are some other publications in the series, including a good monitoring and evaluation toolkit.

The hard part came next: making sure that all our IT systems, training, and other work aligned with this model and were interoperable.

To what extent are the ideas and practices contained within the PD Foundations documents institutionalized within the PD world today?

We actually started tracking this with data in 2019/2020 with our State of Public Diplomacy survey, which asks questions about attitudes and behaviors across the field. And what you see from 2019 to the present is a gradual increase in both beliefs and behaviors around those core ideas above.

The biggest attitudinal shift is in the appropriate policy orientation and contribution of PD programs. Today, there is almost universal belief that PD programs must and do contribute to specific goals in specific ways, rather than being broadly aligned with them.

The belief in outcomes, not outputs, as a measure of success, has also seen a massive change. But it is something that people struggle with doing well, thus the supplementary resources there.

Audience selection has advanced over the last decade. When I started in R/PPR in 2016, the most common audience segments identified in PD plans were “General Public,” and “Youth.” Most posts now give detailed breakdowns of who matters, how we believe our work reaches them, and why we think it will lead to specific changes.

Overall, Locally Engaged Staff (LES – the name for the mostly non-Americans who staff our embassies and consulates) have led the way on understanding these changes and supporting them. Their attitudes and behaviors consistently outscore American Officers. There’s a reason I call our LE Staff the “heart and soul of public diplomacy.”

Do you have any insight into why LES seem to be faster to modernize than Americans?

A number of them have told me that these changes address many of their own frustrations with their work over the previous decades. Intrinsic motivation matters a lot.

Structural motivation matters, too. This new model led to updated position descriptions that LE Staff were given during the modernization. This matters for things like Merit-based Compensation. American officers didn’t have the same big change in their position descriptions, and position descriptions don’t matter meaningfully for Foreign Service promotions. That isn’t the whole reason, obviously–but it does make a difference.

Does anyone criticize PD Modernization and PD Foundations? What do they say, and how do you respond to their concerns?
Absolutely. I’ve heard two primary criticisms: First, this takes the art and human dimension out of diplomacy. Yes, this makes a number of steps routine. But rather than taking away your creativity, it allows you to focus creative energy and your thinking where they matter most: strategy and actually building human relationships.

The second criticism is that doing extensive planning and evaluation take too long and are unnecessary. This is partially right – extensive planning takes longer, and it costs money to run an evaluation. All of that is true. Further, the reality is that when you actually assess programs, they often have high participant satisfaction but result in low attitude or behavior change. That isn’t success! But the desire to achieve high levels of outputs (audience members reached; participants traveling, etc) is a driver of burnout because it focuses on quantity over quality. The new model reverses that equation.

As I said before, our experience is that officials become fans of these new approaches as they familiarize themselves. Success will lead to more success.

Built into these ideas is the concept of learning. Can you name a lesson you learned the hard way through this process?

Absolutely. The first big impact evaluation I ran cost us about $600,000. The program we were evaluating was about $750,000. It was a big investment, but other embassies were considering replicating the model, so we wanted to see if it worked. The investment in a randomized control trial felt sound.

When we got the results back, I was pretty stunned. The program wasn’t effective – it had pretty close to zero effect on its desired attitudes and behaviors. The reason why was also obvious when we reviewed the data: all the program participants already held the desired attitudes and behaviors. Pre-program screening could’ve told us that at a much lower cost.

This changed how we taught PD and did monitoring and evaluation. In PD Foundations, we talk a lot about making sure your actual audience matches your desired audience before you start a program. We also started screening our evaluation projects carefully.

While I was initially disappointed, we more than made back the cost of the evaluation. There are so many projects we have helped redirect or stop thanks to this insight, which turned out to have applicability far broader than we initially thought. That made the whole thing worthwhile.

Paul, you’ve built a reputation as a change agent at the State Department. Do you accept this label? What did it take to develop this reputation?

I’ll take it, in part because I’m not sure I could shake it if I tried!

Change is about leaving things in a better state than you found them. That can be at any scale: Document a hard-to-manage process. Put your important historical files in one place. Build a new partnership.

But the real secret to long-term change? People. Be generous in helping your peers solve problems. Invest in mentoring people. Train people on the secret superpowers you’ve mastered. If you invest in people, you will leave an impact that will last a lifetime.

Efficiency is defined as “achieving maximum productivity with minimum wasted effort or expense.” Thus, if an organization’s understanding of its own productivity is ambiguous, then efficiency is impossible to measure. This is the situation in which US foreign policy finds itself.

An essential task for advocates of efficiency and effectiveness in foreign policy is to build a robust system to measure and evaluate success.

Foreign policy can learn a lot from other fields. Key performance indicators (KPIs) is a common approach in the business world. All companies evaluate themselves (and are evaluated by others) on topline metrics like profits and stock price, but also more subtle measures such as capacity utilization rate, lead conversion rate, customer satisfaction, or R&D performance. Monitoring and Evaluation (M&E) is the public sector’s equivalent approach to KPIs. The M&E toolkit is widely used for foreign assistance expenditures, but virtually never for high-level policy and strategy questions. Certainly, the public sector is different from the private sector, and foreign assistance is only one component of foreign policy, but the State Department (and foreign policy in general) has a lot to learn about measuring productivity and efficiency.

With USAID being swallowed into the State Department, foreign assistance will be managed by policy-focused regional bureaus for the first time. If this is to work, policymakers will need to master new skills of program design and management, especially measuring effectiveness and efficiency. This is no easy task, but it provides an opportunity to unify the work of foreign policy and foreign assistance and improve the quality of both fields.

That the effectiveness and efficiency of US foreign policy should be measured is obvious to some. But the dominant paradigm within the State Department rejects this idea. Prominent foreign policy leaders suggest there is no use for evaluation of foreign policy strategies — that they should be exempt, judging the field as too complex and unpredictable for such scrutiny. Others claim that “evaluation” is already baked into the day-to-day work of foreign policy—that it is inseparable from decision-making. The result is that the State Department and the broader foreign policy establishment does not invest in systematically evaluating their work.

Avoiding discussions of evaluation (and thus efficiency and effectiveness) makes sense if one subscribes to the common belief that “foreign policy is an art, not a science.” But I believe this sentiment has undermined the quality of US foreign policy and sown distrust of foreign policy decision-makers among the American public.

It’s time for foreign policy to mature as a field of practice and overcome its resistance to evaluation. M&E for strategy offers a path forward.

Beyond Compliance

Congress has pushed the State Department to do more to evaluate its strategy processes, even before the merger of State and USAID.

The Government Performance and Results Act (GPRA) laid the foundation for performance-based accountability across federal agencies when it passed in 1993. The law requires agencies to establish performance goals, measure outcomes, and report on their progress. In 2010, the GPRA Modernization Act strengthened existing law by introducing requirements for cross-agency priority goals and emphasizing the use of performance information in decision-making. “Performance evaluation” is a technical term referring to whether a program did the things it said it was going to do, not whether those actions caused any particular external impact.

The Foundations for Evidence-Based Policymaking Act, passed in 2018, expanded the framework for federal M&E by mandating that agencies take a systematic approach to building and using evidence. It directed agencies to integrate evidence-building activities, such as data analysis, program evaluations, and performance assessments, into their policy and management decisions. The legislation induced the State Department to author its first Learning Agenda in 2022. While the questions posed in the Learning Agenda were excellent – “What is the impact of high-level diplomatic engagement?” – the Agenda languished without resources or expertise to advance its work.

The State Department’s Managing for Results framework (18 FAM 300) serves as the Department’s tailored approach to planning and learning. Guidelines for program design, monitoring, and evaluation are spelled out, but these are rarely practiced for strategic or policymaking settings (as opposed to foreign assistance, which tends to be more rigorously evaluated).

The State Department hosts a hierarchical strategy process to set the goals of U.S. foreign policy. These strategy processes provide a natural starting point for implementing high-level M&E. Following the White House-led National Security Strategy, the State Department articulates quadrennial high-level priorities in the Joint Strategic Plan. The JSP helps the State Department develop subordinate Joint Regional Strategies for geographic bureaus and Functional Bureau Strategies for cross-cutting bureaus. Embassies and missions overseas develop Integrated Country Strategies to support the relevant regional and functional strategies. Ad hoc strategies are also sometimes developed, such as by the Bureau of Policy and Planning.

But it is bewildering to discover how little investment is made to formally evaluate the efficacy of these hundreds of strategies. The JSP, Regional, and Functional strategies each receive some basic performance management feedback, but this is not the same thing as M&E. Progress towards the JSP goals, for instance, is measured in the State and USAID Annual Performance Report, but the goals are abstract and not falsifiable. Performance management tends to be useful for tracking whether money was spent and tasks were completed, but unhelpful for understanding impact. It amounts to a compliance exercise.

Here's an example. An objective articulated in the 2022-2026 JSP is Promoting a Stable Cyberspace: “By September 30, 2026, sustain and enhance international cooperation to promote the U.S. vision of an open, interoperable, reliable, and secure Internet and a stable cyberspace.” The goal, typical of the dozens of other objectives in the JSP, is non-falsifiable; phrased in such a way that makes it impossible to know what failure might look like. Virtually any activity can qualify as “sustaining and enhancing.” Indeed, the Department dutifully reports having “met” this objective in 2023 by conducting 175 new or sustained diplomatic engagements on cyber issues, an increase of 171% over the previous year. This figure is almost meaningless; it says nothing about whether Cyberspace is more open or secure.

While the Department boasts that it met or exceeded most of its JSP goals (67 out of 99!), its performance measures say little. It is unclear whether any of the work achieved US interests, or what was learned along the way.

Some suggest – incorrectly – that the intelligence community (IC) might be well-positioned to evaluate the quality of US foreign policy. This misunderstands the IC’s role. A “firewall” has been constructed between policymakers and analysts to prevent the politicization of intelligence analysis. The result is that the IC is excellent at assessing international challenges but unable to analyze the quality of US policies.

Four Essential Principles

As I said previously, M&E is easiest in highly controlled, static environments, but the world of international affairs rarely provides laboratory-like conditions: policy challenges are often “wicked,” featuring a high degree of complexity and uncertainty; strategies often deploy many concurrent activities intended to be mutually reinforcing, and; precise identification of cause and effect is impossible. Some skeptics therefore argue that M&E is a waste of time. This conclusion is wrong.

The goal of M&E is not to seek perfect knowledge in an infinitely complex world. Instead, the goal of M&E is to help policymakers. It achieves this by improving our collective understanding of how policy choices affect the world, mitigating uncertainties, and increasing the likelihood of policy success. Feedback is, after all, the essential ingredient in the development of expertise.

M&E is not effective when it simply suctions up a ton of useless information like a vacuum cleaner. This is a giant waste of time and money. The right approach focuses on collecting knowledge that will improve the policy. A few principles that M&E practitioners working in strategic environments must consider:

1. The challenges we face are highly complex. Most challenges in foreign policy will feature a dizzying array of factors that can affect the outcome. We must remember that policies are dynamic and interrelated. The world is not linear, nor are our policy responses. We must avoid conceptualizing problems or evaluating solutions in overly simplistic or stove-piped ways. M&E practitioners must seek to understand the multivariate nature of the world and apprehend the connections between issues.

2. At the same time, one must not become overwhelmed by complexity. Developing an M&E plan requires parsimony, a prioritization of the issues that matter most. Policy environments are like maps: one can draw a map that shows all the roads, the topography, or the sewer system, but not all three at the same time. One must make choices about what to include and exclude. Policymakers can practice parsimony by prioritizing the most important policy actions in their strategy that require M&E.

3. Systems inherently involve uncertainty. Policymakers are always operating in a state of partial blindness. But uncertainty is not ignorance. Policymaking is like playing a game of cards: one wins by identifying the strategy most likely to succeed, recognizing that failure is a possibility even with the best strategy. The most successful policymakers are experts in uncertainty, taking calculated risks with incomplete information to maximize long-term success. M&E plans should help policymakers mitigate the most meaningful sources of uncertainty throughout the policy cycle.

4. Evidence and feedback from M&E processes are most helpful when directly policy-relevant. Gathering endless data is counterproductive and wasteful. Feedback should seek to deepen our knowledge of both the causes of the problems we face, and the efficacy of the tools we use to address them. When developing an M&E plan, one should always ask: How would confirming or disconfirming evidence change our strategy or behavior? How exactly will this evidence be used to support decision-makers? This helps ensure M&E efforts are useful.

M&E systems that take these four considerations into account – complexity, parsimony, uncertainty, and relevance – will likely be most successful in improving the efficacy of the policy process.

Getting the Details Right: An M&E Primer

M&E is a technical skill that benefits from training and practice. There is a healthy and growing literature from which M&E practitioners and reformers should benefit. At the same time, the particularities should be tailored to the needs of policymakers for each strategic setting. That said, most M&E work follows a similar framework:

The first step – develop the policy or strategy goals – is the foundation upon which the M&E plan will rest. I have written about this elsewhere and will not go into extensive detail here except to say that the purpose of this step is to eliminate ambiguity around what the policy or strategy is attempting to accomplish. Goals should be SMART – specific, measurable, achievable, relevant, and time-bound. Tools like the theory of change, logic maps, and other strategy frameworks help map out all the small steps necessary to achieve one’s goals.

Developing the monitoring plan is the second step. There are few better ways to sharpen a strategy than by interrogating exactly how success will be measured. In complex settings, it can be helpful to begin by identifying the largest uncertainties in the policy environment, and what kind of information would be helpful to mitigate those uncertainties. The information gathered should be useful for improving the policy, not merely scorekeeping purposes.

Selecting the right metrics to track is tricky, but remember that perfection is not the goal. Develop metrics that are both confirming (evidence that would improve confidence that the policy is working) and disconfirming (evidence that would suggest the policy is failing). Track both outputs (the direct results of one’s activities, e.g., how many schools were built) and outcomes (the longer-term impacts, e.g., are the graduates getting good jobs). Consider metrics for performance, context, and resources. Performance monitoring informs you whether implementation is on track (whether your team is doing the tasks they committed to doing), context helps understand external factors that will affect the policy, and resources center on the necessary people and money to get the job done. Quantitative metrics tend to be more objective, though mixing in qualitative metrics can be helpful as well.

The State Department already has standards for data quality, which are useful:

(1) Validity: the data accurately represent the intended measure;

(2) Integrity: safeguards are in place that minimize risk of data manipulation or error to ensure data are accurate and consistent throughout their lifecycle;

(3) Precision: data have a sufficient level of detail for decision making;

(4) Reliability: data collection processes and data analyses are consistent over time; and

(5) Timeliness: data are as current as possible and available at a useful frequency for decision making

The third step is to develop a plan to collect all the data painstakingly identified as necessary in the previous step. Consider investing in surveys, requesting specific field reporting or intelligence collection, or deploying more technical tools such as satellite photos or geo-located data. A reasonable guideline is to invest 5 percent of one’s budget into M&E and requisite staff time.

The fourth step is policy adaptation. Schedule regular evidence review check-ins to evaluate the data collected so far and evaluate the progress of policy. Notice the ways in which the policy is seems to be working, and where unexpected obstacles are arising. Use this feedback to make intentional policy adaptations, including refining the M&E plan itself, based upon what is learned.

The final step – learning – typically occurs after a policy effort or strategy is completed. The goal of this stage is to reflect on the entire policy process and capture lessons about success and failure. These lessons will be useful for future policymaking efforts that make take on similar challenges or attempt to use similar strategies.

Conclusion

I admit to being a zealot when it comes to M&E of foreign policy. I believe it is negligent that the State Department – and the field of foreign policy more generally – does not prioritize M&E for policies and strategies. While the precise discernment of success and failure of diplomatic interventions is often impossible, robbing oneself of feedback altogether is self-imposed ignorance. It is hard to think of another profession with a more tenuous relationship with feedback than foreign policy.

The result is that conversations about efficiency (and productivity, and effectiveness) are grounded almost entirely in subjective perceptions rather than any sort of real evidence. This creates fertile ground for the politicization of foreign policy that is antithetical to expertise.

Meanwhile, the Department of State is struggling to wield influence within the national security establishment. The National Security Council continues to grow in size and influence, not to mention the Department of Defense and Intelligence Community, often relegating the State Department to little more than messengers and tour guides.

Diplomacy needs to up its game. Implementing a new M&E system for policy and strategy is perhaps the most powerful and necessary challenge facing the State Department. The integration of USAID into the Stae Department makes this process more urgent. Nevertheless, it may take years to develop an effective M&E system. It will require investment, trial and error, and dedicated leadership. Ultimately, the quality of the United States’ foreign policy decision-making process may be a more important source of geopolitical power than all of the missiles and fighter jets money can buy. It will be worth the investment of time and money to get M&E for diplomacy right.

The Intelligence Policy Disconnect

By Gary Gomez

The Intelligence–Policy Discourse Disconnect

Thirty years ago, the renowned intelligence scholar Mark Lowenthal captured the essence of the communication dynamic between the producers of intelligence (analysts) and the consumers of intelligence (policymakers):

“A major problem is that the producer-consumer relationship resembles that of two closely related tribes that believe, mistakenly, that they speak the same language and work in the same manner for agreed outcomes. Reality, when viewed from either perspective, suggest something wholly different. Indeed, one is often reminded of George Bernard Shaw’s quip about Britons and Americans being divided by a common tongue.”[1]

It is still true today.

This communication disconnect can degrade the utility of intelligence analysis in the eyes of policy staff, leading to an adverse effect on foreign policy. Indeed, supposed instances of intelligence being ignored might instead be occasions when intelligence offered insightful analysis in a voice or discourse unfamiliar or confusing to policymakers. This dynamic is not an anomaly to be tolerated but a dangerous disconnect to be mitigated. This article will explore the characteristics of the tribal tongue phenomenon and suggest a strategy to alleviate its negative effects on the intelligence-policy relationship.

The Discourse Disconnect

One path toward understanding how organizations perceive, communicate, and use information is through the science of linguistic discourse. Discourse refers to the verbal or written exchange of ideas and the expression of thoughts on a subject. It also organizes knowledge, ideas, or experiences grounded in language and its historical or institutional contexts. Additionally, it influences both organizational and individual perspectives. Discourse is not limited to individual words and sentences; it also represents a form of interaction.[2]

One example of the intelligence-policy discourse disconnect is the 1990 National Intelligence Estimate (NIE) about the breakup of the post-Warsaw Pact Yugoslavia. In a review of the impact of the U.S. intelligence estimate about the impending breakup of Yugoslavia in 1990, Gregory Treverton and Rennah Miles concluded that even though the estimates were remarkably accurate in assessing the internal chaos that would result from the breakup of Yugoslavia in 1990.^[3] The estimate had no apparent effect on policy, and policymakers were seemingly caught off guard when the ethnic killing began.^[4] The NIE assessed that Yugoslavia would break up violently, there was little anyone could do to stop it,^[5] and the Serbs would try to incite uprisings by Serb minorities in Croatia and Bosnia-Herzegovina with the goal of absorbing disputed territory.^[6] Treverton and Miles note that, although intelligence accurately assessed the situation, policymakers did not feel that the assessment provided options for mitigating or addressing the potential chaos. The NIE did not explore any policy options and offered a pessimistic outlook, concluding that the U.S. would have little capacity to preserve Yugoslav unity.^[7]

The different discoursal perspectives were apparent in how the analysts and policymakers perceived the situation. Intelligence analysts assessed the situation and wrote the estimate from the perspective of Slovenian and Croatian self-determination, which had been simmering since the end of World War I. Intelligence discourse was grounded in a deep historical perspective of the region and an ethic of avoiding policy options. However, policymakers interpreted the NIE within the framework of popular concepts, viewing Yugoslavia’s collapse through the lens of Cold War-era Soviet politics and global strategy.^[8] Policymaker discourse expected solutions-based advice relative to Soviet and Eastern Bloc Cold War behavior.

Similarly, a review by Richard A. Best Jr. of the U.S. intelligence leading up to the 2004 invasion of Iraq suggests that policy did not take into account warnings about post-invasion unrest.[9] One assessment, Regional Consequences of Regime Change in Iraq, concluded that the viability of establishing a democracy in Iraq would rest on the stability and success of a new Iraqi government and would require that Iraqi democracy be perceived as developing from within rather than imposed by an outside power.[10] And yet, the intelligence suggested that a U.S.-led war against Iraq would precipitate anti-American riots,[11] and that Iraq would be “a deeply divided society with a significant chance that domestic groups would engage in violent conflict with each other unless an occupying force prevented them from doing so.”[12]

Despite the clear implications of the intelligence community’s analysis, Best concludes, “it cannot be demonstrated, nor is it probably the case, that senior policymakers read the assessments and made a conscious choice to build policy around them.”[13] The intelligence discourse was based on generational ethnic divisions, historical political structures, and a recent brutal regime. The policymaker discourse focused on the cumulative threat of a continued Saddam Hussein regime and assumptions of natural gravitation towards liberal democracy.

Discourse Communities

The tribal tongues phenomenon associated with the intelligence-policy relationship is not unique. Other specialized groups that routinely interact can have divergent terms and outlooks. The concept of discourse community offers a clinical approach to viewing organizational relationships with definable features and attributes and how they might interact more constructively.

Discourse communities are groups that share common goals or purposes and employ established communication practices to achieve these objectives.[14] They can use highly technical terminology and community-specific abbreviations, shorthand, codes, and acronyms to facilitate efficient communication between experts. Unique discourse ultimately influences how each community perceives the world and information from outside sources.

For example, the military and diplomatic communities have common expressions that convey how they conduct business and reflect a broader perspective on how they perceive the world. Each has its distinct mission, business language, and processes. Diplomacy employs terms such as "engage in constructive dialogue" and "we are committed to finding a mutually agreeable solution." At the same time, the military uses terms like "engage the target" and “commit additional forces.” Engage and commit are commonly used terms in both diplomatic and military communities. However, they mean something different to each.

Likewise, the lexicon of intelligence analysts can frustrate policymakers. Whereas policymakers seek quick, straightforward answers with high levels of certainty, intelligence analysis is often inconclusive, ambiguous, or contradictory.[15] To convey uncertainty and the possibility of various outcomes, intelligence reports frequently use phrases and words like ‘if/then,’ ‘on the one hand/on the other hand,’ ‘perhaps,’ ‘although,’ ‘however,’ and ‘maybe.’ Policymakers, in contrast, sometimes derisively refer to these as “weasel words” – intentionally vague or even cowardly. Lowenthal suggests that these constructions can create “octopuses” of sentences in which policymakers interpret expressions of uncertainty as pusillanimity on the part of intelligence analysts. Policymakers may also conclude that intelligence cannot clearly communicate the causes and nature of the uncertainty and ambiguity they are so fond of citing, ultimately alienating the policymaker audience.[16]

Additionally, intelligence analysis is often communicated in a detached, almost mathematical structure to convey objectivity, which can conflict with the concepts, metaphors, and rules of logic that policymakers can easily understand and apply in policy deliberations.^[17] This can contribute to the devaluation of intelligence analysis within the policy process.

Intelligence and policy are distinct discourse communities, each with its own daily goals, perspectives, and processes that reflect ordinary business interactions. They have different organizational worldviews and established vernacular. When presented with identical words, different perspectives and information may be interpreted. The intelligence discourse community communicates in terms of verifiable facts, knowledge transfer, and uncertainty. Meanwhile, a policy discourse community articulates in terms of goals, absolutes, and advocacy.

Narrowing the Discourse Gap

Intelligence and policy must improve their ability to communicate across the divide. Linguistics research suggests that if the communities spend more time in a common linguistic space at a macro level, they will learn to understand one another’s language better.[18]

An example of a macro-level venue is an intelligence-policy working group. The most prominent intelligence-policy working group is the National Security Council. This is a standing, high-level group that includes representatives from across the policy-intelligence spectrum. Other common models exist, such as temporary inter-agency working groups established to address a specific issue, country, or crisis. Examples include regional conflicts, hostages held by terrorists, and treaty or trade negotiations. Such working groups create what is known as a shared discourse community that enhances communication between different groups.[19] These highly-integrated forms of interaction allow for mutual understanding through common experiences, including developing shared mental models that support clear communication.[20] Policy staff can trust that intelligence clearly understands their policy goals and daily information needs in these environments. Intelligence assessments are seamlessly integrated into the mental models of policy staff and are constant components of an iterative policy formulation process. These venues facilitate ongoing discussions about intelligence sources and policy concerns, fostering a shared understanding of questions and answers that lead to the co-construction of knowledge.[21]

Such groups, however, provide only an incomplete solution to this discourse disconnect. Few formal working groups exist, and most are temporary, whereas the vast majority of interactions between intelligence and policy occur at the working level. As the interaction between intelligence and policy representatives proceeds deeper into each bureaucracy and away from venues like working groups, intelligence and policy's unique social and communication aspects become more pronounced and impactful.[22]

Thus, a new paradigm of sustained interaction between the producers of knowledge (intelligence analysts) and consumers of knowledge (policy staff) would improve the discoursal relationship between the two.[23]

The intelligence scholars Josh Kerbel and Anthony Olcott propose a concept that would establish a new framework for a sustained, shared discourse community governing routine interactions between intelligence and policy. They suggest that the intelligence community shift from analyzing for policymakers to ‘synthesizing’ with them, moving from a product-based relationship to a process-based one.

This offers an important contrast to the existing model. The current intelligence-policy paradigm is one in which intelligence analysts are intentionally kept at a distance from policymakers to maintain their analytical objectivity and prevent politicization. The result of this model is a product-focused process where intelligence is only an intermittent and distant contributor to the policy process – intelligence memos are transmitted across the transom with little expectation of feedback. However, this decades-old model has brought into question the relevance of intelligence analysis that is developed intellectually separated from the policymaker.[24]

Kerbel and Olcott envision a new relationship whereby intelligence and policy reimagined themselves in a “client-synthesist” relationship. This synthesis approach would create a series of iterative loops in which both sides would get smarter by making knowledge connections.[25] This framework establishes a virtual shared discourse community where intelligence becomes an integral component of the policy decision-making loop rather than an external, part-time participant. The relationship becomes knowledge-based and focused on inquiry. The synthesist approach is characterized by sustained engagement and information exchange, environments notably different from receiving detached context-free analysis through periodic engagement. Through this approach, both parties continue to converse internally and externally in their own tongues.

This client-synthesis structure would support a discourse in which policymakers engage with intelligence, allowing a free expression of policy goals while intelligence adheres to its rigorous ethic of objective analysis. For example, through ongoing conversation, policy staff can become better informed about the historical context of particular analytic judgments. If a policymaker is uncertain about how to respond to an emerging threat identified by analysts, the intelligence analysts might quickly provide solutions support to policymakers with opportunity analysis that assesses how U.S. interests are affected by evolving circumstances and more precisely reflects the complex array of interests that policymakers are trying to juggle[26], and identifies opportunities or vulnerabilities that the U.S. can exploit to advance a policy. This sort of interaction is much harder under the existing model.

A new client-service synthesis paradigm could also enhance policymakers' trust in the intelligence analysis.[27] For policymakers, the issue is often not whether information is “objectively true” but more about how information is of value. In this client-service relationship, the policy perspective shifts from trusting data to trusting the service provider – the analyst. Analysts earn policymakers' trust when information and analysis are provided in context to understand how actions, events, and actors might intersect and interact to affect outcomes.

Conclusion

The tribal tongues phenomenon has been a well-known and impactful characteristic of the intelligence-policy relationship for decades. It has been accepted as a feature of the interaction between the two that has lessened the value of intelligence analysis to foreign policy. As Lowenthal suggests, certain aspects of the communication gap between intelligence and policy will never be bridged.[28] However, leveraging research in the social sciences can provide a clear understanding of the nature of this dynamic and provide methods to improve interaction. Approaching the intelligence-policy interaction through a discourse lens helps demystify the relationship and provides a framework for discovering modalities that enhance communication and mutual understanding.

Gary Gomez has over 20 years of experience in the U.S. intelligence community, in government, and as a consultant, working with DIA, NRO, NGA, DARPA, and the Office of Naval Intelligence. He also served as a Special Agent with the NCIS, working in counterintelligence and executive protection. Gary is a published author on intelligence, airpower, and national security technology. He has taught intelligence studies for five years, and his research focuses on processes associated with intelligence support to foreign policy.

[1] Mark Lowenthal, “Tribal Tongues: Intelligence Consumers, Intelligence Producers.” in The Domestic Sources of American Foreign Policy, 2^nd ed. Eugene R. Wittkopf (New York: St Martin’s Press, 1994), 265.

[2] Teun A. van Dijk, News as Discourse, Hillsdale: Lawrence Erlbaum Associates, 1998, 29.

[3] Gregory Treverton and Renanah Miles, “Unheeded Warning of War: Why Policymakers Ignored the 1990 Yugoslavia Estimate,” CIA Center for the Study of Intelligence, October 2015, 13.

[4] Treverton and Miles 2015, “Unheeded Warning,”1.

[5] Treverton and Miles, 2015, “Unheeded Warning,”13.

[6] Treverton and Miles, 2015, “Unheeded Warning,”13.

[7] Treverton and Miles, 2015, “Unheeded Warning,”14.

[8] Treverton and Miles 2015, “Unheeded Warning,”4.

[9] Richard A. Best Jr, “What the Intelligence Community Got Right About Iraq,” Intelligence and National Security, vol. 23, no. 3 (2008) 289-302.

[10] Best, “What the Intelligence Community Got Right,” 291.

[11] Best, “What the Intelligence Community Got Right,” 290.

[12] Best, “What the Intelligence Community Got Right,” 292-293.

[13] Best, “What the Intelligence Community Got Right,” 298-299.

[14] Erik Borg, “Discourse community,” ELT Journal vol. 57, no. 4 (2003), 398.

[15] Harry Jones, “Policymaking as discourse: a review of recent knowledge-to-policy literature,” Joint IKM Emergent – ODI Working Paper No. 5 (August 2009), 24.

[16] Lowenthal, “Tribal Tongues,” 269.

[17] Jones, “Policy making as discourse”, 14.

[18] van Dijk, News as Discourse, 26.

[19] J. M. Swales, “Reflections on the concept of discourse community”, ASp [Online], 69 (2016), accessed at http://asp.revues.org/4774; DOI:10.4000/asp.4774, and Jones, “Policymaking as discourse,” 24.

[20] van Dijk, News as Discourse, 107.

[21] Jones, “Policymaking as discourse,” 25-26.

[22] van Dijk, News as Discourse, 27.

[23] Gunilla Eriksson, “A theoretical reframing of the intelligence-policy relation,” Intelligence and National Security, vol. 33, no. 4 (2018), 554.

[24] Gary Gomez, “Intelligence and Policymaking: The Opportunity for a More Collaborative Approach,”

September 18, 2023, fp21, https://www.fp21.org/publications/intelligence-and-policymaking.

[25] Josh Kerbel and Anthony Olcott, “Synthesizing with Clients, Not Analyzing for Customers,” Studies in Intelligence vol. 54, no. 4 (2010), 22.

[26] Fulton T. Armstrong, “Ways to Make Analysis Relevant but Not Prescriptive,” Studies in Intelligence, vol. 46, no. 3 (2002).

[27] Kerbel and Olcott, “Synthesizing with Clients,” 18.

[28] Lowenthal, “Tribal Tongues,” 277.

Why Philanthropy Should Help Build a Better State Department

By: Dan Spokojny | June 3, 2025

Decades of investments demonstrate that philanthropy’s foreign assistance playbook is, at best, incomplete. The size and scope of the international challenges are far too great, and grant makers have yet to identify cost-effective solutions that can be widely replicated and scaled.

The situation is complicated by the closure of the United States Agency for International Development and the evaporation of government foreign aid. Some have called for private philanthropy to rush to fill the space vacated by the federal government. Others argue that philanthropy’s most valuable role is to invest in pilot programs, which could be broadly adopted through government funding under a more receptive future presidential administration.

But these strategies need to be reconsidered. While direct charitable support for international peace and development will always be critical, working around the government is an ineffective — and perhaps even counterproductive — approach. Instead, philanthropy should use its limited resources to bolster the capacity of the U.S. government itself. I’m not talking about funding yet another study of China policy. Philanthropy should help improve how the State Department and other civilian functions of national security work. The ongoing reorganization of the State Department and dismantling of other institutions of civilian foreign policy, including the U.S. Institute of Peace, Radio Free Europe, and the Millenium Challenge Corporation, makes this work more urgent than ever.

Not All About Trump

It’s easy to blame President Trump for the current situation, but doing so ignores a broader trend. Three-quarters of Americans expressed trust in the government 60 year ago. Today only one-fifth say the same. Morale inside the State Department — ostensibly the government’s lead peacemaking institution — has followed a similar downward arc. Even before Secretary of State Marco Rubio announced deep cuts to the organization’s staff and structure, agency employees were increasingly expressing dissatisfaction with working there. A survey taken during the Biden administration found that a third of career diplomats planned to quit.

This dissatisfaction crosses political lines. Republicans believe the State Department is bloated and ineffective and needs to be gutted. When Democrats are in charge, government institutions like the State Department “face a different kind of danger,” suggests political commentator Ezra Klein. “Democrats are too quick to defend and to fortify them, even when they don’t work. The Democrats are often committed to the institution in name, to the process as it exists, as opposed to the outcomes the institution is supposed to deliver.”

A common assumption of philanthropic international assistance is that the government is either not doing enough or is doing the wrong thing. Certainly, this is always at least partly true, and investments in NGO-led foreign assistance, academic research, and peace advocacy have all paid dividends. But limited resources and short time horizons are often tragically misaligned with challenges: To end a war within one year, a funder proposes conducting a 20-person peacebuilding workshop.

While examples of extraordinary successes in peacebuilding have been achieved by nongovernment organizations, they are uncommon. Public health interventions may save lives, for example, but national or international instability is itself a cause of poor health.

An additional challenge is that the assistance provided by funders is often directly contradicted by government policies. For instance, an organization might invest in grassroots peacebuilding at the same time the government bolsters a country’s corrupt elites. Or peacebuilders may work to generate support for reconciliation, while Congress funnels military support to only one side of the conflict.

Meanwhile, the civilian instruments of power — especially the State Department — are growing increasingly outdated. A 2020 report from the Council on Foreign Relations, authored by two high-level Biden administration appointees, cited the State Department’s “lousy reputation of being effective policy players.” Former CIA Director William Burns, one of America’s most respected living diplomats, called diplomacy a “lost art.” The result is an increasingly unhealthy U.S. foreign policy that favors military options over diplomacy and development.

And yet, absent a concerted effort to invest in improving the quality of our national security institutions, the status quo approach churns on year after year.

A New Playbook

Philanthropy may represent the best hope to change this sorry state of affairs. Grant makers will largely need to start from scratch, investing in think tanks, academics, and experts to study how to bolster civilian institutions, especially the State Department. These research projects might explore how to build organizational cultures that can innovate and learn new methods of preventing and resolving conflict, improving methods of decision making, and strengthening the responsiveness of bureaucracies.

Only a handful of small grants currently support such work. For example, the Bridging the Gap project, funded by the Carnegie Corporation of New York and a few others, works to better integrate scholarly research into foreign policy decision making. Other efforts have attempted to popularize the idea of making foreign policy more relevant for the middle class — and increasing domestic political support for the institutions that guide it.

Grant makers can also amplify calls to upgrade the outdated culture that permeates the foreign policy process. Fortunately, a vehicle already exists to help them do that. The bipartisan Congressional Commission on Reform and Modernization of the Department of State, launched during the Biden administration, was born from widespread agreement that the State Department is not performing up to expectations — yet almost no external research or advocacy is available to help Congress strengthen it. The commission was largely ignored by the Biden team, and its future under Trump may suffer a similar fate.

Ensuring this commission is a success could be one of the most effective investments funders could make toward international stability. One model for this work is the role former Google CEO Eric Schmidt played in chairing and helping Congress design the National Security Commission on Artificial Intelligence. When the commission’s work finished, Schmidt committed his own money to build a new think tank dedicated to implementing its recommendations.

Philanthropists are rightfully motivated by acute international crises, such as protecting Ukraine’s sovereignty from a belligerent Russia or stopping the war in Sudan. Investing in improving the government’s capacity to address conflict may not feel as urgent but in the long run is likely to more meaningfully improve the likelihood of peace.

There is sound research to back this up. When governments rely on flawed information, misperception, or poor institutional decision-making, they inadvertently fuel conflicts. Overconfidence seems to have been a damning factor in nearly every war the United States has fought since WWII: Vietnam, Korea, Iraq, Afghanistan. Research demonstrates that cognitive biases — unchecked within our intuition-based foreign policy process — steer decision-making toward hawkishness and war.

Whether one believes that U.S. foreign policy should be more dovish, or that peace is best achieved from a position of strength, history demonstrates that when American diplomacy fails, unnecessary conflict often results. Philanthropists can help avoid these disasters.

It isn’t a viable strategy to simply defend the status quo and lament the loss of USAID. Philanthropies are positioned to drive a more productive conversation about systemic change in the government’s approach to conflict prevention. International peace depends on getting this right.

But I have been critical of our leaders, especially career diplomats, for resisting real reform at the State Department under previous administrations. Between the marginalization of career diplomats, a culture that fails to learn, and an outdated policy process, the necessity for reform has been loud and clear.

I don’t know what the future holds, and it is hard to write about reform while so many people risk losing their jobs. Nevertheless, I want to explore the argument that Secretary of State Rubio’s reorganization plan may lead American diplomacy to a better place. This is a pivotal moment for American foreign policy, and advocates for diplomacy need to join the conversation about reform and help lead this institution to a stronger footing.

Rubio’s Theory of Success?

Rubio’s explanation for cutting bureaus and offices at the State Department centers on efficiency: leaner is better. Eliminating redundancy and cutting the fat, according to this theory, will lead to better policy.

The cynical analysis of Rubio’s plan is that top officials always believe their own policy ideas are brilliant; the bureaucracy just needs to get out of their way. Everyone else’s office is too large and powerful, but one’s own office never has enough resources or authority. If this cynical perspective is right, Rubio’s plan is less about efficiency and more about the consolidation of power at the expense of good foreign policy.

And yet, one can find support for the “streamline” theory espoused by Democrats and career diplomats. Uzra Zeya and Jon Finer, two prominent Obama and Biden administration officials, offered a similar analysis in a 2020 report:

Multiple undersecretaries or their staffs should not be involved on the same policy issues. Reducing the number of undersecretaries, with bureaus allocated underneath them at the discretion of the incoming secretary of state, would help eliminate overlapping responsibilities, empower assistant secretaries and equivalents (such as the counselor and director of policy planning), and reduce layering while leaving sufficient senior management capacity in place.

Three luminary career ambassadors working at Harvard’s American Diplomacy Project similarly called for:

Systematic efforts … to devolve responsibility to the lowest possible levels, including a dramatic streamlining of clearance processes and a flattened organizational structure with fewer bureaus and enforceable limits on the number of Deputy Assistant Secretaries.

I thus think there is plenty of reason to evaluate Rubio’s plan in an optimistic light.

Strengthening Regional Bureaus

The most compelling theory of success for Rubio’s reorg is that a transfer of power to the regional bureaus will not simply streamline, but also encourage increased cooperation within the Department. Let me explain.

One of the State Department’s core organizational design features is the division between regional and functional bureaus (diagramed below). The regional bureaus, organized under the powerful Under Secretary for Political Affairs, own the bilateral relationships with foreign countries. They control policy through their management of the embassies and other diplomatic missions. The functional bureaus, organized under a variety of other undersecretaries, are responsible for cross-cutting, transnational issues. Functional bureaus are typically dependent on the consent of regional bureaus to implement policies in any particular country.

If one compares this new org chart with the old one, one finds that it closes or consolidates many of functional bureaus (including the largest functional undersecretariat formerly known as J), resulting in a transfer of authority to the regional bureaus. This change may have a meaningful impact on the design of foreign policy.

The logic of the functional bureaus is that some topics in foreign policy, like human rights, anti-narcotics, or nuclear proliferation, are so important that they require independent offices with specialized expertise. Under secretary and assistant secretary-level leadership of these issues helps ensure that the issues will always have a loud voice, a powerful advocate, and a bevy of technical expertise. For example, creating a powerful office of human rights office will ensure human rights are never ignored. Right?

Not so fast. The establishment of issue-specific offices does not necessarily lead to better policy. It may only introduce new turf battles. I witnessed this firsthand when I worked in the Conflict and Stabilization Operations Bureau (CSO). The bureau had a great deal of knowledge, but it was constantly fighting for relevance with regional bureaus. It is not as if one becomes the Ambassador to Syria, or somewhere similar, by being ignorant of regional conflict dynamics. As a result, CSO’s impact on conflict policy was limited, at best.

When one imagines a corporation creating a distinct operational unit, one imagines that those uniques produce a distinct output: product design, manufacturing, sales, human resources, etc. The State Department's regional versus functional design seems to violate this organizational design principle. Whereas a regional bureau creates policy for the countries in its purview, the human rights bureau also tries to make policy for those same countries (and, in fact, has an internal office structure that replicates the regional bureau model on a smaller scale, dividing up the world according to region). It would be wrong to suggest the Assistant Secretary for Near Eastern Affairs is ignorant about human rights issues – quite the opposite. Their job is to balance human rights, conflict management, economic, security, and a million other concerns.

The amount of overlap between regional and functional offices varies depending on the issue areas. It makes a lot of sense to exclude from the purview of regional bureaus offices with specialized roles like Diplomatic Security, Legal Affairs, Consular Affairs, and Legislative Affairs. The arms control bureau certainly overlaps regional bureau territory, but it employs nuclear engineers with technical expertise that is inaccessible to typical foreign affairs officials — the case for an independent arms control bureau thus seems reasonable. Can one argue that human rights, conflict stabilization, refugees, or counter terrorism require specialized, technical expertise? The arguments here are less strong.

In sum, Rubio’s plan to pare back the functional bureau model is logical. The State Department has a bad habit of setting up a new office for every new slice of foreign policy. The result is a great deal of competition and turfsmanship, leading to a bureaucracy that often seems duplicative, ineffective, and wasteful. Arduous consensus procedures lead to watered-down and risk-averse conclusions.

Rubio’s reorganization plan will challenge this logic by doubling down on the centrality of the regional bureaus. Will it work? Who knows. But fulfilling the promise of this theory won’t happen automatically. It will require some additional steps.

The Path Towards Effective Reform

If Rubio’s reorganization is to be a success, it will need to help facilitate a more cooperative and collaborative policy environment. There are a variety of steps his team should consider:

Rubio’s team should hack the incentive structures that lead to turf battles by designing new mechanisms that reward coordination between bureaus. This might entail designing a more evidence-based decision-making process to replace the outdated clearance process. Further, investments in enterprise knowledge management can combat information siloing, a telltale sign of turf competition.

Strengthening the strategy and budget process may be the best opportunity for Rubio to create a more effective and efficient Department. Procedures for central planning at the State Department are currently weak and ineffective, and regional and functional families have separate strategy processes. This creates barriers for cooperation, resulting in a muddled strategic landscape that attempts to make every issue a priority, even when goals may work at cross purposes. Introducing a new doctrine for policy engineering would support more effective action to achieve US national security goals.

The promotion process for Foreign Service Officers also deserves attention. The current process encourages a “kissing up and kicking down” mindset that encourages loyalty to one’s direct manager. There is no structural reward for collaboration outside of one’s stovepipe. Because of this, an officer serving overseas will typically be more responsive to the regional bureau that owns their turf while deprioritizing requests from functional bureaus who have no input into their promotion review. A more robust merit evaluation system would hold officials accountable for the achievement of both regional and functional bureau goals.

With regional bureaus being asked to take more responsibility and management of foreign assistance projects in the aftermath of the USAID closure, many foreign assistance professionals worry that these offices lack the skills to manage and support programs. They are right to be concerned. Program management entails a variety of technical skills, including budgeting, design, implementation, monitoring and evaluation, grant and contract management. Regional bureaus, primarily staffed by Foreign Service Officers, tend to lack these skills. Rubio might require that all Foreign Service officers receive program management training. These skills are quite useful for policy design and strategy as well, which would be a boon for the Department.

Updated Organization Requires Updated Culture

The shuttering of dozens of functional bureaus and offices means that regional bureaus will have to shoulder more responsibility for integrating diverse expertise into the policy process. To make this change stick, Rubio must challenge the top-down decision-making culture at the State Department. As the world grows ever more complex, our leaders must relinquish the expectation that they can know everything.

Consider US strategy toward Russia. An effective strategy will require expertise not only on Russian politics, but also the war in Ukraine, fiscal and trade policy, military capabilities, tech competition, energy issues, human rights and democratization, regional alliances like North Korea and China, cybersecurity, and so much more. No one top decision-maker can understand the complexities of each of these issues. Instead, leaders in a streamlined Department will need to demonstrate deep respect and curiosity for their working-level experts. They need to learn to draw together teams with divergent specializations to design carefully calibrated policies.

Ultimately, the State Department boasts expertise on a dizzying array of issues. A weak Department allows those many different teams to compete for attention and resources, expending precious time and energy on bureaucratic battles, resulting in risk-averse policies that impress nobody. A stronger Department will create synergy and cohesion among its many actors. I hope Rubio has the vision and skill to build a stronger, more effective, and more efficient Department.

Diplomats have been waiting with bated breath to hear the Administration’s plans to reorganize the State Department. Last week, the New York Times published a report on a “draft” of the memo proposing a “drastic overhaul” of the Department’s structure. The memo turned out to be largely incorrect. This morning, Secretary of State Marco Rubio published the actual proposal for reorganizing the State Department. Many diplomats are breathing a sigh of relief, or admitting “this could be worse.” (The NYT, it’s worth noting, is nevertheless still naming this a “drastic overhaul,” perhaps embarrassed by their previous miss.)

Rubio is characterizing this plan “a broad reorganization.” A handful of bureaus are being cut, and more are being reorganized and recombined. I’ve seen reports that 132 offices will be closed, which might be right, but it appears that only a handful of bureaus will be closed, most of which were already somewhat marginal.

While strong advocates of diplomacy will argue that any cuts to the State Department right now are a bad idea, this proposal avoids cuts to any core diplomatic functions. The offices proposed for removal strike me as “add-on” functions, such as conflict or energy diplomacy, that can otherwise be handled by other offices or departments. Every new administration adds or subtracts some of these functions depending on its political views. That seems like fair game.

Rubio’s stated theory is that simplifying the org chart will simplify the Department’s clearance process (the consensus-based process used to craft policy), which will result in more responsive, efficient, and effective foreign policy. He writes that, “The Department has long struggled to perform basic diplomatic functions [because…] bureaus and offices fight to be included on the approval chains for the most mundane of memos, only then to reach agreement on drafts that are bloated in length while stripped of all meaning.” This is not a new analysis of the problem; it has been made by prominent Democratic officials, too.

And yet, I doubt Rubio’s medicine will cure the sickness. Using the same outdated decision-making recipe will not likely achieve different results simply because there are fewer cooks in the kitchen. A new culture will require procedural changes, not simply fewer actors. Rubio suggests a need to “confront the underlying bureaucratic culture that prevents the State Department from carrying out an effective foreign policy,” but this memo alone will not achieve that objective.

It will, of course, be important to see how this is all handled. Ideally, this reorganization will be accompanied by meaningful procedural changes to modernize the practice of diplomacy. The worst case scenario is that draconian staffing and funding cuts, and failure to appoint competent leaders, will cripple the Department’s operations. But I see that as a separate conversation.

Proposed Changes

Here is a full list of the proposed changes, according to my quick comparison of the draft org chart (pasted above) with an old org chart.

The biggest change is that the Undersecretariat for Civilian Security, Democracy, and Human Rights (J) has been axed, but some of its functions have been relocated:

• The Bureau of Conflict and Stabilization Operations (CSO) has been cut

• Office of Global Criminal Justice (GCJ) and Trafficking in Persons (TIP) have been cut

• Democracy, Rights, and Labor (DRL) and Populations, Refugees, and Migration (PRM) have been moved to the Office of Foreign Assistance (F). International Religious Freedom (IRF) seems to have been folded into DRL.

• Immigration, Narcotics, and Law Enforcement (INL) has been moved to T

The Undersecretary for Political Affairs (P) will gain some meaningful authority:

• Loses its Counter Terrorism office (CT) to T

• Rubio announces that “All non-security foreign assistance will be consolidated in regional bureaus”

The Undersecretariat for Arms Control and International Security (T) also seems to be gaining some functions:

• INL and CT relocated here

• ISN (International security and non pro) and AVC (arms control, verification, and compliance) combined into a new Arms Control, Nonproliferation, and Stability (ANS)

• New office of emerging threats. Maybe some STAS functions?

And so does the Undersecretariat for Economic Growth, Energy, and the Environment (E)

• The Science and Tech Advisor (STAS), Energy Resources (ENR), and Global Partnerships (GP) have been cut

• Global health security and diplomacy (GHSD) and Cyberspace and Digital Policy (CDP) were given a permanent home here (both previously reported to the Secretary)

The Office of Public Diplomacy (R) seems to survive largely intact:

• The Global Engagement Center, which had previously drawn the ire of Republicans, is gone.

• It appears that the Office of Spokesperson has been removed from R and will now get its own office (S/SPOX) reporting directly to the Secretary

Also noting that the Foreign Service Institute (M/FSI) seems to be missing from the organizational chart. This would, indeed, be a drastic move, but I’m going to assume this is merely a typo until I hear otherwise.

The new(ish) Office of the Coordinator for Foreign Assistance and Humanitarian Affairs (F):

• This office appears to have been moved from reporting directly to S to being on par with an undersecretariat, which is both a demotion in the hierarchy, but also an expansion of the breadth of its authority.

• F will reportedly be taking on some of USAID’s old humanitarian assistance functions, while other forms of development assistance will be handled by the regional bureaus in P

• F will now oversee the bureaus of DRL and PRM

The Undersecretary for Management (M)

• There appears to be a new Office of Global Acquisition (M/GA), which used to be conducted by Administration (A) bureau, if I’m not mistaken. I don’t know the story here.

• The Foreign Service Institute (FSI) is missing from this updated organizational chart, but I suspect (hope) this is merely a typo. If all training and language education is being eliminated from diplomacy, that would indeed but a radical change.

The Secretary’s office tends to collect authority over a handful of specialty offices, and there are some changes here, too:

• Dropping Global Women’s Issues (S/GWI), Diversity and Inclusion office (S/ODI)

• S/OCR (Office of Civil Rights) has swallowed up the former Ombudsman’s office (S/O)

• New S/SPOX office

• It appears from the lines on the org chart that the Executive Secretariat (S/ES) will now report through the Deputy Secretary’s of State (D and D-MR) rather than directly to the Secretary, but it’s not clear.

There was widespread agreement with our diagnosis of the problem: the Department struggles to prioritize effectively, and high-level ideas are often disconnected from working-level implementation. Many agreed that addressing this problem will require something like the policy engineering approach outlined in our article: a more disciplined way of setting goals, taking action, and evaluating progress.

Whereas our previous article focused on the principles of policy engineering, this article digs deeper into how the method might be applied to the work of foreign policy. This article is merely a starting point, a draft. I hope that policy engineering (or something like it) will be robustly developed and adopted as official State Department doctrine. In the meantime, I hope these ideas can be helpful to officials at the State Department today who are trying to develop more effective foreign policy.

Download a PDF of the checklist.

Stage One: Set High-Level Goals

The first stage of policy engineering is to identify high-level goals. This is typically completed by the White House, Congress, or political leadership at the State Department. Pre-existing high-level goals may be found in the National Security Strategy, Congressional budget and appropriations documents, and the Joint Strategic Plan, among other places.

If goals aren’t clearly dictated by existing guidance, strategy engineers must create them, ideally by channeling the Administration and Congress’s priorities. The first step, therefore, is to discern the goals and values of the political leadership. In addition to existing strategy documents, policy engineers might read campaign speeches, executive orders, or articles from key officials.

Make no mistake: setting high-level goals is a political act. But this must not disqualify nonpartisan career staff from leading and participating in this process (in my humble opinion). A bureaucrat’s job is to advance the president’s agenda. Career officials may disagree with aspects of a political agenda, but bureaucratic resistance can result in costly consequences for the agency’s reputation and erode its authority over time.

Regardless of who leads this work, setting high-level goals is the starting point of the policy engineering method. Goals provide the framework that guides the development of specific policies and objectives in later stages. These goals should communicate WHAT needs to be accomplished, but not HOW the work will be done. Goal-setting should consider whole-of-government responses and how different instruments of US national security can work together. Goals should ultimately be formulated as clear statements of success (achieve democratic elections) rather than general statements of principles (value democracy).

At the start of each strategy cycle (perhaps synchronized with the federal budget process), it is useful to gather a diverse team of experts to participate in the goal-setting exercise. The group should discuss likely challenges and opportunities, and brainstorm different end-states that might harmonize with US interests. Policy leaders should solicit ideas widely, and remember that there are no bad ideas during brainstorming.

The next step is to narrow the range of options to the highest priorities. The ITN procedure offers one tool for prioritization: rate each option based on its importance, tractability, and neglectedness. Importance refers to how valuable the solution would be if it worked. Tractibility is the likelihood of success – cost and political viability should be factored in here. Neglectedness is the extent to which others are already working to solve the same problem. For instance, advancing a new nuclear non-proliferation treaty regime might score an 8/10 in importance, a 2/10 in tractability, and a 7/10 in neglectedness, for a total of 17. If a competing policy scored 23, that might be a better option.

How does one know if they have identified the right goals? Goals are, in no small part, an expression of subjective ethics. In a democracy, our elections are the best arbiter of our shared values, and some amount of variation is healthy and natural. Each newly elected leader comes in with a different set of values that has been “endorsed” by the citizenry. (Political appointees, nominated by the president, are an extension of this process). Whereas this stage is largely subjective, the next stage – policy design – asks policymakers to explore as objectively as possible the likely implications of these high-level goals, and craft interventions to achieve the big goals.

Stage Two: Policy Design

The policy design phase breaks down high-level goals into actionable steps. This stage requires more than mere extrapolation; policymakers must carefully evaluate their options to ensure their chosen pathway will most likely achieve their goals.

Good guiding questions for this stage include: What would it look like to take a meaningful step toward achieving our goal over the next few months? How confident are we that this will actually work?

My favorite tool for policy design is the theory of change (ToC). A ToC is a series of if-then statements that detail precisely how change will occur as a result of US action: “If we do X, then Y will occur.” The goal is to reduce ambiguity as much as possible about how high-level goals will be achieved. For example, "Pursue free and fair elections" is a very ambiguous policy design. It would be better to propose, "If we communicate to the host government that all future military assistance will be conditioned on the certification of the Spring 2025 election as free and fair by reputable international observers, then we can help ensure democratic elections in 2026." Additional analysis should explicate exactly how this coercive strategy would operate, including how US military assistance would incentivize democratic growth or how the threat of sanctions could weaken non-democratic impulses.

A strong ToC pushes policymakers to explore the chronological sequence of actions and reactions. For example: “If US publicly conditions military assistance to the government of X upon the allowance of election observers, then the government of X will likely accept election observers. If there are election observers, then the government will respect the legal rights of opposition political parties, resulting in a free and fair election.”

Policy design must strike the right balance between specificity and generality. This entails identifying a limited set of near-term actions that, if pursued over the next 3–12 months, will advance the high-level foreign policy goals. Policies tend to be designed in Washington but implemented abroad, necessitating that the design must provide clear instructions for forward-deployed officials while providing enough flexibility for those closest to the problem to react to obstacles. A well-intentioned policy can be rendered feckless by ambiguity, but there’s also such a thing as too much specificity and micromanagement: “First you will sit down at your computer with a coffee to draft the memo, then…”

The acronym “SMART” helps evaluate whether goals are well designed: every policy goal should be Specific, Measurable, Achievable, Relevant, and Time-Bound. As in the high-level goal phase, brainstorming with an array of experts is an excellent way to develop, evaluate, and prioritize SMART goals.

Policymakers ought to give special attention to whether a policy design is actually “achievable.” When policymakers demand transparency about the set of facts used to support recommendations and invite scrutiny about potential sources of bias, they make better policy in the present. This approach has been pushed by the Foundations of Evidence-Based Policymaking Act, signed into law in 2019, but this sort of rigorous policy evaluation remains uncommon in today’s State Department.

The risk is that a policy may seem logical but fall apart upon inspection. For instance, if financial sanctions are proposed as a tool for pressuring President Putin to end the war in Ukraine, a review of the sanctions literature may illuminate the conditions under which financial coercion has succeeded or failed in the past.

Policy designers should get quite specific about resource requirements at this stage. One must confront the reality that the most effective policy designs may be unaffordable. Competing priorities and limited resources may require narrowing the scope of one’s ambitions. A useful way to evaluate resource allocations is to assign actions to specific offices or team members and estimate financial costs. The ultimate question is always: Which policy design is most likely to be effective and efficient?

The final step of policy design is to develop a plan to continually measure the effectiveness of the policy. Policymaking is an iterative method, not a set-it-and-forget-it microwave meal. A plan to monitor and evaluate implementation will help officials ensure the policy is proceeding according to plan, quickly identify obstacles, and learn more about the impact of their efforts. Feedback is an essential ingredient for the development of expertise over time.

Let’s be clear: there are strong incentives for officials to avoid careful policy design. Telegraphing one’s intentions is like Babe Ruth pointing toward the stadium fence: great if you hit a home run, but you might look like a fool if you strike out. It is much safer for one’s career to set ambiguous goals, promise to appease every stakeholder, and fixate on post hoc signs of progress. I detect these hidden impulses when officials declare their allegiance to “the art of foreign policy.” A culture that punishes any admission of failure will never have a meaningful strategy process.

Stage Three: Policy Implementation

I do not have much to say about how to conduct high-quality implementation beyond the obvious and essential point that policy designs need to be turned into action. As our previous policy engineering article specified:

This entails delivering demarches, negotiating treaties, implementing new projects, or other activities specified in the design stage. With clear policy designs guiding their work, diplomats in the field would be empowered to solve problems that arise during implementation. During implementation, staff would detail their efforts and gather evidence, including objective and subjective judgments, to evaluate progress. Written staff work —primarily in the form of cables from the field, memos to department leadership, and State Department input into interagency meetings — would communicate these judgments, including whether resources are adequate to accomplish the stated policy goal.

If readers have ideas for how to build more effective policy implementation processes, I would love to hear from you. Perhaps some insights from the field of implementation science?

Stage Four: Policy Adaptation

Evidence gathered in the field throughout implementation should guide continuous policy adaptation.

The team responsible for policy adaptation might ask itself the following questions: How did the last quarter go? Where did we actually spend our energy? What were our successes? What worked? What went wrong? What can be expanded, and what needs to be fixed or canceled? How can we adjust the allocation of our finite resources?

Managers might use weekly check-in meetings to monitor progress and quickly troubleshoot obstacles that arise. More fundamental questions about policy design can be addressed in quarterly strategy review meetings.

In this sense, all policies are like experiments. The goal is to collect evidence to evaluate the impact of one’s intervention.

Celebrating success is easy. But a healthy culture must also celebrate failure. Leaders should reward team members who identify problems or discover why a policy is failing to achieve its planned objectives.

There is such a thing as too much adaptation. If a policy design is constantly changing, it suggests that policy designers struck an improper balance between specificity and generality. If high-level goals and policy designs are ultra-specific and overly-choreographed, they will be rendered moot by every small change in external circumstances.

Ultimately, the purpose of reviewing policy is to support the next decision rather than merely evaluate the last decision. This ensures that policy adaptation prioritizes learning rather than scorekeeping.

Conclusion

The policy engineering method accomplishes four objectives: 1) ensures policies will effectively advance the president’s foreign policy objectives, 2) facilitates turning ideas into action, 3) helps the bureaucracy participate in a shared vision of success, and, 4) creates a culture of continuous learning.

The suggestions offered here are not intended as rigid instructions. There is no such thing as a perfect policy process. Offices should regularly adapt policy engineering to their own needs.

Ultimately, I hope that leadership at the State Department will identify policy engineering (or something similar) as a crucial area for improving US foreign policy and invest in building a more rigorous method. This might entail appointing a team to study and design doctrine for diplomacy that would integrate this discipline into the regular business of the Department of State. Until that day, I hope policy engineering can nevertheless be helpful in improving the quality of US foreign policy.

If you have any experience with implementing any of these techniques within the State Department or related institutions, or suggestions about what we missed, we would love to hear from you.

Note: When President Trump announced new additions to the Foreign Terrorist Organization (FTO) list during his March 4, 2025 State of the Union address, my ears perked up: my colleague, fp21 academic fellow Dr. Michael Becker, has just completed a report evaluating the FTO process.

Curious to learn more about the FTO process? The timing of this article is perfect. Michael discovers a number of pathologies in the current FTO process and makes some recommendations about it how it could be improved by adopting more advanced risk management processes. His analysis offers an excellent example of what expert foreign policy could look like in practice. Check it out below.

-Dan

Improving Risk Assessment Processes within the State Department: Lessons from Foreign Terrorist Organization Practices

By: Michael Becker, Ph.D.

In July 2018, the government of Japan executed Shoko Asahara along with several followers. He was the founder of Aum Shinrikyo, a group that carried out chemical weapon attacks in Japan in the 1990s. These executions marked the end of an organization that had not posed a threat to the United States in decades, if indeed it ever had. But despite the group’s collapse beyond a reasonable doubt, it remained on the State Department’s list of Foreign Terrorist Organizations (FTOs) from 1997 until 2022, when it was removed as part of a package of de-listings.

Today, the United States Department of State designates nearly 70 groups as FTOs, but the list does a poor job of reflecting U.S. national security interests. Many groups seem to remain on the FTO list far beyond their “shelf life,” such as Aum Shinrikyo and the Basque group ETA. Other times, groups are added to the FTO list that do not represent the most pressing American national security priorities. For example, in the run-up to the September 11, 2001 terrorist attacks, the United States could have been focusing on addressing the threat from Al-Qaeda and related groups. Instead, it added two Western groups to the FTO list: the New Irish Republican Army and the United Self Defense Forces of Colombia. To take another example, the 1990s designation decision of an anti-Iranian regime organization was characterized by a Clinton administration official as a “goodwill gesture” toward Iran, even though the group had already largely ceased to threaten direct American interests.

A faulty FTO designation process is harmful to U.S. national security in several ways. When an organization is listed as an FTO, multiple organs in the State Department and elsewhere in the United States government are obligated to conduct costly monitoring, interdiction, and disruption operations. Focusing attention on groups that pose little or no threat is a poor use of resources. Additionally, the existing process is too slow to recognize groups of genuine concern. For example, despite indications of its involvement in the 1992 Aden hotel bombing (which targeted U.S. Marines) and the 1993 World Trade Center bombing, Al-Qaeda was not designated until October 1999, two years after the current FTO process was created.

The problem with the FTO process is not poor judgment – of course, specific mistakes are clearer in hindsight: identifying and addressing threats is difficult, and reasonable people can disagree ex ante. The challenge is instead a procedural one, the result of a legalistic process more focused on compliance than risk mitigation.

The Trump administration has signaled a possible change in FTO designation practices. On January 20, 2025, President Donald Trump issued an executive order instructing cabinet officials to consider designating certain drug cartels as FTOs. The administration has also redesignated the Houthis as an FTO, reversing a decision by the Biden administration, which itself had rescinded an FTO designation instituted by the first Trump administration. While the ultimate effects of these decisions remain to be seen, in this essay I argue for a more systematic reform to State Department policy in this area. Specifically, I focus on the current FTO designation process, identify shortcomings, and suggest reforms with a view to improving both counterterrorism and the practice of American foreign policy writ large. In brief, I propose that the FTO process implement systematic risk analysis practices and more granular designation labels. This shift would introduce additional flexibility in the initial designation of an organization while at the same time making it easier to de-list outdated threats.

The current process

The broad structure of this process was outlined in a 2015 Government Accountability Office report. Typically, the process begins with an assessment by the Bureau of Counterterrorism and other bodies within the State Department, which monitor the past and projected future activity of any group deemed a potential threat. Next, there is a consultation process involving stakeholders from multiple agencies and departments. This step, called an “equity check,” increases the number of actors who must agree in order for a designation to occur, and exerts effects on the list’s composition which I discuss later.

According to the State Department, the initial designation requires that a foreign group be deemed to:

engage in terrorist activity, as defined in section 212 (a)(3)(B) of the INA (8 U.S.C. § 1182(a)(3)(B)), or terrorism, as defined in section 140(d)(2) of the Foreign Relations Authorization Act, Fiscal Years 1988 and 1989 (22 U.S.C. § 2656f(d)(2)), or retain the capability and intent to engage in terrorist activity or terrorism.

But this alone is not enough. An additional condition is that such “terrorist activity or terrorism must threaten the security of U.S. nationals or the national security (national defense, foreign relations, or the economic interests) of the United States.” The meeting of both of these conditions permits, but does not compel, a Secretary of State to list an organization. If a listing occurs, the Secretary informs Congress, and the designation takes effect (subject to litigation).

Before 2004, a given designation needed to be actively renewed every two years to remain valid. But under the Intelligence Reform and Terrorism Prevention Act of 2004, the Secretary is empowered to review designations every five years or when an organization files for review. Listed organizations may file for a review of their place on the FTO list after designation and every two years after review decisions thereafter. While federal law allows the Secretary of State to delist a group if they find that the rationale for a designation has changed or if national security demands it, in practice secretaries have tended to maintain the status quo for political and bureaucratic reasons.

Pathologies of the current process

Issues abound throughout the process of FTO designation. Even before the decision to list a group is made, during its initial screening, the State Department sticks to its well-known position that foreign policy is “more art than science,” and that “quantification can be dangerous.” Based on interviews with State Department officials, here, too, analysts are largely reliant on idiosyncratic “good judgment” rather than reproducible scientific methods. Modern risk analysis frameworks are entirely absent from this process.

Once a potentially threatening group has been identified, there are two ways things might go wrong. Statisticians categorize classification errors as “false negatives” and “false positives.” False negatives in the FTO context represent real threats that the nation is unprepared to address and mitigate, while false positives lead to wasted effort and resources on the part of government personnel. Generically, there is a tradeoff between these types of error, but because the current FTO process bifurcates the initial designation and re-designation decisions, the U.S. is currently risking both.

Near the beginning of the process, because of the equity check step, a large number of stakeholders across the federal government have an effective veto over a group’s designation, which they might exercise for any number of bureaucratic, political, or substantive reasons. For instance, there may be pressure not to list an organization with which the U.S. has an enemy in common. This process tends toward a “lowest common denominator” where, as a rule, only the groups everyone agrees are threatening are ultimately listed. Even if the Bureau of Counterterrorism’s initial vetting process recommendation is on the mark, there remains a danger of ultimately excluding organizations whose initial threat level does not generate unanimous alarm.

Another key challenge is that the process is currently driven by legal rather than analytical or strategic considerations; that is, whether lawyers associated with the State Department are satisfied that the statutory conditions for listing can be successfully defended in court. But while State Department policy should, of course, be consistent with the law, such “adversarial legalism” has tended to outsource too much of the policymaking process to lawyers rather than focusing on effective national security policy.

This state of affairs also helps explain why, once on the FTO list, groups tend to stay there for many years after they have ceased to be a threat. In part because of legal inertia, when organizations are ultimately de-listed, it is often in batches that bear little relationship to the threat they pose. Admittedly, maintaining dubious groups on the list may be understandable on political grounds: the State Department does not want to de-list an organization that later attacks the United States. Unfortunately, the expenditure of resources related to a group on the FTO list is potentially considerable (though difficult to measure precisely). A listing, justified or not, engenders a practically whole-of-government response, from observation and/or prosecution of persons associated with a group, to excluding immigrants linked to the group from the country, to a raft of financial monitoring and disruption actions concentrated in the Treasury and Justice Departments.

Of course, ex ante it is never certain that a group constitutes a false positive. “Black swans”—low probability, but high-impact events—are one of the central challenges of counterterrorism and foreign policy. But this reality does not obviate the need for effective risk management, it increases it: the federal government cannot address every conceivable threat, so it must allocate its resources carefully.

It should be noted as important historical context that much of the legislation governing FTO listing was crafted around the early 2000s, in the crucible of the response to 9/11. In this environment, policymakers can be forgiven for failing to design an optimal policy process from a risk management perspective. But, two decades later, some rethinking of FTOs is warranted.

A way forward

The FTO listing process is an excellent environment in which to explore improved risk assessment procedures at the State Department. In a departure from many other challenges in foreign policy, the FTO process has a clear, measurable goal: to prevent terrorist attacks against the United States, its allies, and its interests.

Some suggested solutions that should be explored:

1. Implement a systematic risk analysis procedure. Currently, the designation schema is binary and non-ordinal. A graded or multi-dimensional system should be implemented (internally, if not publicly) to allow for resources to be directed toward the most pressing threats.

This reform could take a number of different forms to account for important, but currently underappreciated, sub-dimensions of risk, such as the directionality of a threat (is it growing, receding, or static?), its probability (how likely is it that the group will conduct an attack?), capacity (how much damage could the group cause?), and relative risk (how does the threat posed by one group compared to another, or to some baseline non-terrorism threat). A group’s level on any one of these component scores could trigger specific policy consequences or contribute to an overall index that provides a broader picture of the threat it poses and what quantity and type of resource attention it merits.

As a part of this reform, the State Department should reform the equity check process so that attempted “vetoes” of a group’s listing are considered holistically in the risk assessment alongside the dangers of not doing so.

Systematic risk analysis (in contrast to an evaluation system that relies on often ad hoc reasoning) would also facilitate more effective evaluation and learning about past decisions. Following unfortunate, but inevitable, instances of terrorist attacks, the Counterterrorism Bureau can review its previous analytical findings and learn from its strengths and weaknesses. After more than two decades of experience since the current FTO system was put into place, there should be plenty of data from which to learn and improve.

Finally, I propose two reforms inspired by the academic literature on nudges and blame avoidance to limit unwarranted retention of non-threatening groups:

2. By default, unless overridden, a designation should sunset after a small number of years. A provisional “probationary” status with a more moderate level of scrutiny and resources could then follow to confirm that a group is transitioning away from threatening Americans, while preserving a pathway for rapid re-designation if necessary.

Although the public tends to overestimate the likelihood of international terrorism, we cannot expect political and agency leaders to disregard the public’s fears, to say nothing of the backlash following an attack. Nudges like changing default behavior can be effective by taking the decision to delist a group out of individuals’ hands, providing some political cover in the unlikely event that a de-listed group later does attack Americans. Such nudges have been successfully applied in other settings where “adding” is easier than “subtracting,” leading to unintended oversupply. This increased flexibility would also free up valuable personnel time that is currently dedicated to continually re-certifying existing FTOs.

3. By default, unless overridden, a probationary designation should sunset to a complete de-listing after a larger number of years

Here, it bears keeping in mind a possible tradeoff. On the one hand, the deterrent power of an FTO designation could be enhanced by some noise and stickiness in the process. On the other hand, at the margin, too much stickiness reduces the incentive to desist from terrorism, because groups already on the list remain there either way. Moreover, relative to the set of threats they must address, U.S. government resources are not so plentiful that they can be expended merely to send a signal.

Conclusion

The process governing the Foreign Terrorist Organization listing process contains a number of pathologies emblematic of broader challenges in the practice of American foreign policy. As such, it contains lessons for the State Department as a whole. For example, the current FTO process reflects outdated attitudes toward resource stewardship, deterrence, and risk assessment that pervade current American foreign policy. Setting priorities and expending resources effectively in a complex geopolitical environment depends on accurate assessments of risk.

FTO designation is only one arrow in the quiver of actions the U.S. government takes to mitigate terrorism. But the State Department list of FTOs is a public good widely looked to by academics, journalists, think tanks, and other institutions of government—at home and abroad—as the authoritative set of violent groups of concern. If the list is inaccurate or incomplete, the national security implications are especially troubling, but the effects extend beyond suboptimal counterterrorism.

The need for counterterrorism practitioners to prioritize the most pressing threats with their resources echoes a broader State Department need for retrenchment faced with the confluence of acute geopolitical challenges and fiscal tightening. Prudent resource use is, of course, a perennial concern, and even a perfect designation process is not a panacea. But as the international landscape appears particularly chaotic for the foreseeable future, the State Department must move to implement state-of-the-art risk assessment and mitigation plans to guard against novel threats from revisionist groups, states, and technologies. Once effectively implemented in the field of counterterrorism, such practices might also extend to other areas, such as pandemic prevention. Issues in counterterrorism provide useful object lessons because terrorism displays many of the features (stochasticity, lack of “return address,” feedback loops) that make other emerging threats so bedeviling.

About the author: Michael Becker, Ph.D. is an academic fellow with fp21: he is a political and data scientist who studies international security, political violence, and foreign policy. His research uses computational, statistical, and qualitative methods to elucidate topics such as the connections between communications and behavior by militant groups, the drivers of lopsided international conflicts, target selection by lone violent offenders, and the dynamics of civil wars. Michael received his Ph.D. from the Department of Politics at New York University.

Give it a listen:

I told her that I believe it's the wrong strategy for supporters of diplomacy to simply defend the status quo without offering viable alternatives. As I've been writing for years, the status quo is not producing sufficiently effective foreign policy and the influence of career foreign policy professionals was trending towards zero.

Instead of playing defense, I suggested that we — those of us who care about national security — must continue to put forward good ideas for reform and look for opportunities to steer this ship in a productive direction.

I know many people (some of whom are friends and colleagues) are scared by what may happen at the State Department. I'm not trying to diminish that fear. But reform is arriving, like it or not, and there will hopefully be plenty of chances for people to make a positive impact.

Here’s a link to the original NPR article and audio:

https://www.npr.org/2025/03/01/nx-s1-5302267/the-state-department-braces-for-looming-cutbacks-by-the-trump-administration

From Strategy to Action: Rethinking How the State Department Works

By Peter Lohman and Dan Spokojny | February 21, 2025

U.S. diplomacy faces a paradox: The State Department is filled with some of the most knowledgeable foreign policy experts in the world, yet its impact is diluted by bureaucratic inefficiencies and lack of focus.

Secretary of State Marco Rubio, in his initial guidance to employees, called for a “more innovative, nimble, and focused State Department.” He lamented the department’s relegation to a “secondary role” in U.S. foreign policy, as other agencies are perceived to move faster and more effectively. Several studies in recent years have identified similar shortcomings.

The State Department’s perceived shortcomings stem from an organizational culture that for too long has relied on educated guesswork and ad hoc implementation. In fulfilling its core duty of advancing U.S. interests overseas, the department has often emphasized the art of foreign policymaking at the expense of disciplined methods. While recent modernization efforts have prioritized expanding staff and expertise, continued gaps in diplomatic tradecraft and culture hinder turning that expertise into policy action.

Suggestions to add rigor to the State Department’s work often collide with two practical realities: First, each new presidential administration brings differing views of U.S. interests. Second, every new challenge to U.S. national security carries unique tradeoffs of risks and benefits.

Yet, the variability of the ends and contexts of U.S. foreign policy should not excuse a lack of structure in its design and implementation. The State Department should treat policymaking — the practice of applying specialized knowledge to the practical solution of public problems — as the discipline that it is: One that includes clear methods for design, implementation, and adaptation. The department should adopt, teach, and refine this method of “policy engineering” and use it to harness the workforce’s tremendous expertise to more effectively advance U.S. interests abroad.

The Gap Between Planning and Doing

The State Department’s traditional method for policymaking is heavy on planning but light on coordinated action. Consider the department’s Managing for Results framework, which mandates every regional and functional bureau, and embassy, outline its high-level goals every four years — what the department calls “strategic planning.” Department staff spend hours poring over the latest National Security Strategy and other strategic guidance to craft over 200 individual plans for virtually every country and transnational issue in the world. The resulting plans provide general direction but fail to incite specific, coordinated action for pursuing U.S. interests.

Take, for example, the fact that very few of these strategic plans influence the department’s day-to-day expenditure of time and resources. While the Managing for Results framework includes recommendations for how to communicate long-term objectives and design foreign assistance programs, it provides little structure for how staff should achieve these long-term goals. Ultimately, the work of following through on the department’s myriad plans is largely ad hoc and voluntary, with limited accountability for progress.

We have experienced this problem firsthand through successive tours in Washington and abroad. Searching for ways to help prioritize among seemingly infinite daily demands, we and our colleagues regarded the department’s formal plans as too amorphous to provide value.

Lacking guidance, staff often defaulted to the most immediate task rather than ones tied to achieving long-term goals. Policy work devolved into a handful of initiatives working at small scale and often cross-purposes — an approach insufficient to tackling the biggest challenges. Resources were misallocated using a “spread-the-peanut-butter” approach whereby time, expertise, and funding went to many problems with the goal of satisfying as many bureaucratic stakeholders as possible.

This ad hoc approach serves as a tax on the department’s already overburdened staff, time, and resources. Absent prioritization, every challenge appears “underfunded,” and few are solved. Congress and presidential administrations become frustrated with the department’s inability to deliver. Worse yet, other parts of the federal government less adept at diplomacy are asked to step in, increasingly marginalizing the State Department over time.

The result is a paradoxical situation whereby the department’s vast foreign policy expertise serves more as a burden than a blessing. Without a method of translating expertise into action, department staff are overwhelmed with information and pulled in divergent directions, challenging everyone’s ability to focus on the most important tasks.

Harvard Business School Professor Frances Frei has a term for large organizations that struggle to prioritize among competing demands, expend enormous staff time focused on inward-facing tasks like email correspondence, and leave their workforce suffering from a general sense of fatigue and aimlessness: “exhausted mediocrity.” She argues that a workforce focused on doing everything rather than the most important things will exhaust itself and disappoint its customers. In our experience, this aptly describes what ails the State Department.

Policy Engineering: Turning Ideas into Action

One effective way to cure the department’s exhausted mediocrity is to fill the space between the formal planning process and the department’s day-to-day work. We call this policy engineering, building on former State Department Counselor Philip Zelikow’s efforts to draw lessons from the field of engineering, which similarly applies specialized knowledge to complex problems. Policy engineering is an iterative method for policymakers to ensure big picture goals connect to action in the field. It is meant to upgrade what Zelikow calls the “software” of public policy: the methods and routines the department uses to solve problems. The U.S. Army’s doctrine for the operations process also provides a useful model.

Policy engineering at the State Department would begin with policy design. Rather than the department’s existing planning process, which burdens staff with writing four-year plans with vague, long-term goals, policy engineering starts with prioritizing action, in coordination with the National Security Council and relevant agencies. Working from direction provided by the National Security Strategy and other high-level plans, bureaus and individual embassies would specify a limited set of near-term objectives that, if pursued over the next 6–12 months, would help achieve the administration’s long-term foreign policy goals. These near-term objectives must be proximate, concrete definitions of success paired with key results that could be reasonably assessed and achieved. The intent of policy design is to keep staff focused on forward-looking objectives amid the demands of the daily grind. In addition, the design phase would involve assigning teams to lead implementation, identifying resources and tools of implementation (i.e., programs, engagements, events), and setting deadlines for review. The policy design phase is largely absent in the State Department’s current organizational culture.

The department’s bureaus and embassies would turn design into action through policy implementation. This entails delivering demarches, negotiating treaties, implementing new projects, or other activities specified in the design stage. With clear policy designs guiding their work, diplomats in the field would be empowered to solve problems that arise during implementation. During implementation, staff would detail their efforts and gather evidence, including objective and subjective judgments, to evaluate progress. Written staff work —primarily in the form of cables from the field, memos to department leadership, and State Department input into interagency meetings — would communicate these judgments, including whether resources are adequate to accomplish the stated policy goal.

These observations would feed continuous policy adaptation, when bureaus and embassies would guide department and interagency changes in design and implementation. Key questions include: What policy interventions have been tried and what has been learned? Should assumptions change? Is the policy sufficiently maximizing the strengths of the United States and its allies and partners while exploiting competitors’ weaknesses? Does design and implementation adequately capture not just the risks of action but also of inaction? Most importantly, how should policy design, implementation, and resourcing change to incorporate these lessons?

The three steps outlined above are intended to serve as a framework rather than formal stages in a sequenced process. Rarely in the messy practice of foreign policymaking do policy responses advance neatly from design to implementation to success, making iterative moves during policy adaptation critical.

Getting Started: Benefits and Recommendations

State Department staff can implement these techniques and begin improving the organization’s policymaking immediately. However, to firmly integrate policy engineering into the culture and practice of the department, institutional upgrades will be needed.

Adopting a standard, department-wide method for policy engineering would be a good starting point (one might call this a cornerstone of “diplomatic doctrine”). The secretary’s Office of Policy Planning could serve as a hub for this work, refining the method over time. The Foreign Service Institute could include classes on policy engineering as part of its new core curriculum, using historical case studies to teach staff foreign policy design, implementation, and adaptation in different contexts and at different levels of the organization.

Policy engineering could also improve accountability and learning. Here, a pilot, perhaps run by the Office of Policy Planning, could work with a small group of bureaus and embassies to test structured semi-annual or annual assessments. These assessments could identify priorities and proximate objectives, lessons learned from implementation, and recommendations for further action, including requests of department leadership to help resource and scale successful policies. Congressional appropriators, in particular, might appreciate the department’s attention to identifying and weeding out poorly performing efforts.

Finally, policy engineering will only reach its full potential if it is a high-level priority. The secretary of state could model prioritization by announcing his or her own proximate objectives each year, including how the department will know if it is making progress in meeting them. Additionally, the secretary could preside over an annual award to recognize the department’s best policy engineering efforts. A future secretary might also link policy engineering with the department’s budget process and use the learning it generates to justify resource requests.

Effective foreign policymaking does not merely spring from the department’s wealth of knowledge and experience. While recent modernization efforts have rightly focused on helping the department develop, acquire, and retain expertise, that expertise needs an organizational culture which prioritizes action. Translating expertise into action is a craft that goes beyond what one can reasonably learn on the job. It should have basic prescriptive steps that can be taught and refined over the course of one’s career.

In a recent report, former Deputy Secretary of State for Management and Resources Richard Verma admits that the department must continue “looking for ways to become more efficient and capable through better strategic planning [and] training… ” We agree. In today’s age of unprecedented peril, with limited resources and in long-term competition with determined foes, the State Department must focus on the handful of initiatives that can make a real difference. Advancing a method for policy engineering is the right next step in service of this endeavor.

Peter Lohman is a State Department foreign service officer. He has served in Chennai, Jerusalem, Jakarta, and Washington. His most recent assignment was as the director for Southeast Asia on the National Security Council. He was a cavalry officer in the U.S. Army before entering the State Department.

Dan Spokojny is the founder of fp21, a nonprofit dedicated to improving the State Department. He previously served in government for more than a decade as a foreign service officer and a legislative staffer in Congress. He teaches U.S. foreign policy at George Washington University, is a member of the Editorial Board of The Foreign Service Journal, and is a former member of the Governing Board of the American Foreign Service Association.

The opinions expressed here are those of the authors and do not represent the views or positions of the U.S. government.

Read part 1 of the M&E series first:

The Opportunities

A mentor once explained that there are two ways for an organization to achieve success: A) achieve its objectives; or, B) fail to achieve its objectives but learn something important in the attempt.

But there is currently little formal feedback to assess the effectiveness of diplomatic strategies and foreign policy, resulting in a culture with a weak understanding of its own tools. Today’s State Department is not dedicated to learning.

Monitoring and evaluation (M&E) practices (or something analogous) could be pivotal for improving the health of the State Department culture.

Creating an effective system of feedback and learning at the State Department would improve its reputation for solving problems, build a culture dedicated to expertise, and help return diplomacy to its rightful role in the center of US national security.

A rigorous sense of “what works” is an essential ingredient for designing useful training the next generation of diplomats, and perhaps even starting to construct diplomatic doctrine. Investments in understanding diplomacy’s successes and failures could also strengthen the merit evaluation system, which is currently derided as a creative writing contest.

One of M&E's strengths is that it forces conversations about exactly what success looks like before the policy is deployed. This is an antidote to the scourge of watered-down, wishy-washy policies that often emerge from the State Department’s maligned clearance process — policies that are calculated to offend nobody rather than create real change.

Strong M&E practices also enhance transparency and accountability. Whereas the State Department has a hard time demonstrating the value of diplomacy to Congress and the American people, M&E can help provide discrete evidence of diplomacy’s impact.

And yet, these benefits are largely theoretical. M&E systems are largely untested for high-level strategy and policymaking at the State Department. While such tools are tried and tested for foreign assistance and program-level work, high-level M&E work presents many new challenges and complications.

The Challenges of Complex, Strategic Environments

M&E is easiest in highly controlled, static environments, but the world of international affairs rarely provides laboratory-like conditions. Diplomacy is not a widget-making factory in which the foreman can easily observe each employee’s productivity. As the scholar James Q. Wilson explains in his book, Bureaucracy, settings where impact is difficult to observe tend to leave managers grasping for control over subordinates by imposing compliance-oriented performance management procedures.

Performance management is not all bad. It is a subtype of M&E that focuses inward on compliance issues: How many hours have we worked? How much money have we spent? Are we doing the activities we said we’d do? Are we following proper procedures? Answering these questions can be useful in certain circumstances, but performance management says nothing about whether a team actually achieved anything.

This situation describes the State Department’s sprawling system of offices and embassies. For example, one of the goals articulated in the 2022-2026 Joint Strategic Plan for the State Department is Promoting a Stable Cyberspace: “By September 30, 2026, sustain and enhance international cooperation to promote the U.S. vision of an open, interoperable, reliable, and secure Internet and a stable cyberspace.”

The Department dutifully reported that it “met” this goal in 2023 because it conducted 175 new or sustained diplomatic engagements on cyber issues – an increase of 171% over the previous year! I’m sure Congress and the Secretary of State were pleased that the team kept busy, but this figure is functionally meaningless; it says nothing about whether all this activity made cyberspace more open and secure.

As fp21 fellow Toby Weed wrote in an excellent article about measurement in foreign policy,

Many officials worry about the unintended consequences of measurement in foreign policy. The scholar Kishan Rana suggests that data-driven management “may misdirect diplomatic activity to the measurable forms and de-valorize the more subtle, long-term work of relationship building.” For instance, a 1983 joint resolution of Congress asked the State Department to produce a report counting how often other countries voted with the United States in the UN General Assembly. The goal was to nudge embassies to invest more in multilateral diplomacy, but embassies were observed “equally weighting ‘throw-away’ policies with real impactful U.S. policy goals.” As the management guru Peter Drucker famously observed, “What gets measured gets managed – even when it’s pointless to measure and manage it, and even if it harms the purpose of the organization to do so.”

It is little wonder, then, that policymakers at State are resistant to such measurement activities. Good policy judgment can become distorted by incentives to maximize meaningless performance metrics. Rigid adherence to top-down directives and compliance frameworks can actually erode the quality of programs and policies. The expertise and judgment of frontline staffers to adapt to new challenges and contexts is an important ingredient for success. This may be especially true in complex strategic settings.

Some go even further, suggesting it is not possible to usefully measure diplomacy. “Diplomacy is fuzzy,” explained a former Ambassador and Dean of the Department’s Foreign Service Institute, “it's almost impossible to quantify what we do, and in fact I think that there's a great danger in trying to quantify it." Again, from Weed’s article:

A core challenge is that “the results of the work of foreign ministries are typically indirect, difficult to measure and not easy to influence directly,” and “whether an expected result is obtained does not depend completely on the actions of the foreign ministry.” (Mathiason, 2007, p. 226). That is, it can be hard to determine causality in diplomacy. Did a leader withdraw their armed forces because of skilled negotiation, or was it due to the opposition’s resolve? Did a civil war end because of the efforts of third party mediators, or because the warring parties simply grew tired of the violence?

A related methodological challenge is that many strategies are often deployed at the same time in a single policy environment. For example, there may be dozens of separate China strategies and policies with complex interdependencies. This confounds methodological approaches that seek to isolate a single, linear cause and effect.

The promise of AI and big data leads many M&E practitioners to surmise that more data is always better. Some programs today may proliferate hundreds of metrics to be tracked. But, more often than not, this attitude is naïve and amounts to little more than a big waste of time. This “M&E overkill” problem may be exacerbated in increasingly complicated strategic settings.

Even if the previous challenges are surmountable, the State Department’s culture strenuously resists critical feedback. If a negative result can harm one’s career, robust M&E practices will rightly be perceived as threatening. This resistance to feedback is political, exacerbated by a climate of distrust between Congress and the State Department. I sometimes joke that “diplomacy is the art of setting ambiguous goals and always claiming success.” A culture that forbids failure is incompatible with effective M&E.

A final challenge is about optics: M&E may also have a branding problem, associated with the scars of past failures rather than as an instrument for success. M&E may also be too closely linked to programmatic work to be accepted by policymakers who see themselves as distinct from (or superior to) foreign assistance work.

What’s Next?

I hope my review of the challenges of strategy-level M&E has not convinced you that I am wasting my time. Instead, my goal here is to be clear-eyed about the challenges facing proponents of high-level M&E for policy and strategy. I am interested in hearing from you: What have I missed? Have I inappropriately minimized any of these challenges? Please send me your thoughts.

My next article will start to flesh out some ideas for how these challenges might be surmounted, and to advance the conversation about creating a more effective culture of policymaking at the State Department.

Stay tuned!

As the newly arrived Foreign Service officer, my assignment was to help lead a multi-million-dollar program to prevent violent extremism in a fragile South Asian country. The project was well-designed. The strategy was to flood the country with pro-peace messages and content. Everything seemed to be going smoothly until a nationwide survey we deployed revealed a startling finding: The more one interacted with our program, the higher support for political violence one demonstrated. The program was likely doing harm!

Counterintuitively, encouraging peace appeared to trigger a negative reaction in the people our program touched. Without a carefully designed plan to study whether the project was working – called monitoring and evaluation (M&E) – we would have continued the program for years while convincing ourselves that we were doing a great job.

How many other foreign policies might be failing to achieve their objectives, or even causing harm?

We don’t really know. But research from other fields offers a stark warning for foreign policy practitioners: most attempts at changing the world fail. In biomedicine, over 90% of drugs fail clinical trials. In education, 90% of the interventions tested failed to achieve impact. In the private sector, 80% of new strategies conducted by Google and Microsoft failed to move the needle. Employment and training programs studied by the Department of Labor failed 75% of the time.

The good news is that M&E is an important component of all foreign assistance programs developed by the State Department and USAID and a lot of smart people are working to improve practices. The bad news is that there is no expectation for M&E when developing a policy or a strategy – any effort that is not considered assistance. The fact is that the State Department does not routinely study the impact of its policy choices.

This is a major weakness in the practice of US foreign policy. The lack of feedback and learning processes helps explain the State Department's poor reputation for policy effectiveness, its weak culture of merit, the absence of sufficient training, and its weak strategy processes.

Strategic-level M&E practices offer a great deal of promise. Applying M&E practices to policy and strategy can help officials ensure programs are proceeding according to plan, understand the impact of their efforts, and learn what works. As this Substack has explored, high-quality feedback is an essential ingredient for the development of expertise.

Over the next few weeks, I plan to explore how the State Department might develop a process to implement M&E for all meaningful strategic and policy initiatives. I have framed my research around some key questions:

1. What is M&E, and what are the potential benefits?

2. What are the drawbacks and pitfalls of M&E practices?

3. What is the current state of the art in M&E practices?

4. Can practices developed for M&E of foreign assistance translate to the complexities of higher-level strategic environments?

5. What are the existing laws and regulations shaping M&E in the government?

6. What kind of organizational features facilitate effective M&E?

I’ll do my best to answer these questions in the coming weeks, but, as always, my goal here is to start a conversation about this complex topic. I would love to hear about your expertise, experience, and perspectives. Comment here, or email me directly, if you have ideas about how (or how NOT) to apply M&E to higher-level strategic policy challenges.