In this post, I will highlight some of the key learnings and takeaways from the course.

The Ones Who Walk Away From Omelas

We began our learning journey in the first week with “The Ones Who Walk Away from Omelas”. This truly inspiring story, written by Ursula K. Le Guin, raises societal questions that go beyond the moral dilemma of weighing the suffering of one against the happiness of many. It encourages readers to reflect on their responsibilities as members of society and to consider the impact of their own choices – to walk away, to stay and accept the status quo, or to stay and fight against existing inequalities.

Algorithmic Decision-Making and Fairness

Next, we explored the topic of algorithmic decision making and fairness, examining different interpretations of fairness, such as providing equal opportunities to individuals, or achieving equal outcomes. Our faculty lecturer, Prof. Mehran Sahami, highlighted the difficulty of optimizing algorithms for fairness due to mutually exclusive definitions of fairness. The Princeton case study “Hiring by Machine” provided us with insightful information on the barriers to implementing fairness in practice, alongside with a series of challenging and thought-provoking questions for the readers. One aspect that stood out for me was the consideration of compensating for the current “lack of moral flexibility or pragmatism” of AI systems compared to humans by incorporating randomness into decision-making processes. This reflection raised the question for me of whether we are limiting ourselves by assuming that replicating human decision-making as we know it is the ultimate goal, or whether the current discussions about algorithmic fairness are simply shedding more light on deeper, existing challenges within our perceptions of fairness.

Data Collection, Privacy, and Civil Liberties

In the course, we also talked about privacy and challenged the common belief that the concept is simply about “hiding a wrong”, as discussed in the essay “I’ve Got Nothing to Hide and Other Misunderstandings of Privacy” by Daniel Solove, published in the San Diego Law Review. The familiar “I’ve got nothing to hide” argument fails to capture the nuanced, pluralistic concept of privacy. For example, collected data could be aggregated or used for secondary purposes, potentially increasing state power. The risks of loss of privacy go beyond surveillance, as illustrated in George Orwell’s “1984”. They can also manifest in a sense of powerlessness, vulnerability or a lack of control, as outlined in Franz Kafka’s “The Trial”. Another common misunderstanding of the concept of privacy was analyzed in the article “What Is Privacy For?” by Lowry Pressly, published in The New Yorker in October 2024. One of the problems outlined in the article is the assumption that privacy is based solely on the principle of keeping already existing information “out of the wrong hands”. As an alternative approach, the author suggests that we should ask whether the main problem is the mere existence of that information. He also emphasizes the right to oblivion, pointing out that “the main interest in life and work is to become someone else that you were not in the beginning”. I strongly agree with this and believe that the use of our past behaviour data by social media platforms can limit our personal development and growth, as well as our ability to learn from mistakes.

Private Platforms and the Political Economy of Technology

In discussing the political economy of technology, we critically examined the business models of today’s big tech companies, which rely on targeted advertising. This approach inherently leads to harmful practices such as surveillance, maximizing user engagement, and the spread of extremist content. The common justification that these practices are essential to provide “free” services for the public can be challenged by proposing a fee-based model including financial support to ensure access for those who cannot afford it. We also discussed the challenges of balancing free speech on the Internet with the goal of tackling hate speech and extremist or violent content. One of the readings was John Stuart Mill’s “On Liberty”, which addresses the risks of social conformity and highlights the importance of allowing diverse perspectives and opinions to avoid confirmation bias. Mill argues that in the pursuit of truth, we must encourage an open discussion and gather and combine even conflicting ideas. Personally, I find the application of these principles challenging in today’s digital age, given the amount of misinformation and harmful content on the Internet and the speed at which it spreads.

Generative AI and the Future of Work

Finally, we explored generative AI and its impact on the labour market. We were encouraged to consider different possible futures: a world in which AI gradually replaces humans through automation, and another in which AI augments human capabilities by providing better information and supporting human decision-making. We also discussed how generative AI is expected to have a significant impact on “knowledge” work, unlike previous technological advances that have primarily affected physical work. Another interesting aspect we learned was that generative AI could particularly benefit average-skilled workers, potentially strengthening the middle class. I was struck by the fact that, according to one study, computer science and mathematics, my two majors at university, are expected to be the areas most complemented by generative AI in the future. Nevertheless, I remain optimistic about the future of work, and one of the quotes from the lecture that stuck with me was that AI will not replace your work, but a person using AI will. I am deeply impressed by the capabilities of generative AI, which can not only increase the productivity of people at work, but also support the learning process. In this context, I highly recommend Google’s NotebookLM. This tool allows users to upload documents, to generate summaries of the content and provide answers with citations. One of the most impressive features for me is the ability to create a podcast with two speakers having a casual conversation about the topics provided in the sources, which is definitely a nice addition when you are discovering complex topics.

Application and Reflection

My learning objective for this course was to deepen my understanding of the ethical dimensions of technology, broadening my focus beyond fairness and non-discrimination in artificial intelligence to a wider range of socially relevant issues in technology. Coming from an engineering background, where the focus is often on finding solutions to problems, it can be challenging to be comfortable with the ambiguity of ethics. However, as I have learnt and practiced throughout the course, the essence of ethics is about asking the right questions rather than seeking immediate solutions.

I would like to thank the course facilitators, cohort leaders, faculty lecturers, and guest speakers for making this course possible, which has reinforced my decision to pursue a career focused on the socially responsible use of technology. In conclusion, I would like to encourage everyone to continue to ask the tough questions, to actively fight against inequality, and to ensureence and mathematics, my two majors at university, are expected to be the areas most complemented by generative AI in the future. Nevertheless, I remain optimistic about the future of work, and one of the quotes from the lecture that stuck with me was that AI will not replace your work, but a person using AI will. I am deeply impressed by the capabilities of generative AI, which can not only increase the productivity of people at work, but also support the learning process. In this context, I highly recommend Google’s NotebookLM. This tool allows users to upload documents, to generate summaries of the content and provide answers with citations. One of the most impressive features for me is the ability to create a podcast with two speakers having a casual conversation about the topics provided in the sources, which is definitely a nice addition when you are discovering complex topics.

Application and Reflection

My learning objective for this course was to deepen my understanding of the ethical dimensions of technology, broadening my focus beyond fairness and non-discrimination in artificial intelligence to a wider range of socially relevant issues in technology. Coming from an engineering background, where the focus is often on finding solutions to problems, it can be challenging to be comfortable with the ambiguity of ethics. However, as I have learnt and practiced throughout the course, the essence of ethics is about asking the right questions rather than seeking immediate solutions.

I would like to thank the course facilitators, cohort leaders, faculty lecturers, and guest speakers for making this course possible, which has reinforced my decision to pursue a career focused on the socially responsible use of technology. In conclusion, I would like to encourage everyone to continue to ask the tough questions, to actively fight against inequality, and to ensure that technology benefits us all and is used to solve the most pressing problems in our world. that technology benefits us all and is used to solve the most pressing problems in our world.

Abstract

In recent years, AI fairness has gained considerable attention, as several machine learning applications have demonstrated systematic discrimination against certain population groups. A major focus of fairness research has been on classical machine learning models and supervised learning tasks, such as classification, particularly in the context of automated decision-making. General-purpose AI models, however, present unique challenges for fairness assessments due to their wide-ranging downstream applications and varied use case scenarios across different domains. It is crucial that both the underlying base model and downstream systems, such as applications built on fine-tuned models, undergo fairness assessments, as new biases can be introduced during the adaptation process. In this article, I propose a comprehensive fairness assessment framework applicable to both downstream applications of general-purpose AI, including LLMs, and classical machine learning tasks. The framework provides a comprehensive step-by-step guide throughout the AI development lifecycle and highlights the essential technical, legal, ethical, and domain-specific skills and responsibilities of the development team required at each stage. The recommended steps begin with a detailed examination of the use case, including the identification of potentially vulnerable groups, followed by the assessment and mitigation of bias in the training, fine-tuning, or retrieval data. Next, it addresses the design and evaluation of fairness-related measures on the trained model, such as a combination of fairness and performance metrics across different groups, along with red teaming to tackle the unique challenges of generative AI. Finally, the framework proposes the implementation of a monitoring platform and the establishment of regular internal and external audits. Three real-world examples of bias illustrate how applying the framework helps uncover a wide range of fairness issues in AI systems.

Motivation

Several AI governance frameworks address the risks of bias and discrimination in AI systems, and highlight the challenges of ensuring fairness in general-purpose AI. In the context of fairness assessments, several of these frameworks remain at a high level, as this approach offers greater flexibility and better adaptability to the rapid developments in AI. Moreover, the context-dependency and complexity of fairness make it challenging to establish universally applicable practical guidelines. However, a lack of sufficiently clear and actionable steps for AI development teams make it difficult to operationalize fairness effectively. Therefore, more concrete guidelines are needed to help development teams integrate fairness into their AI systems in an impactful way.

The designed fairness assessment framework offers a comprehensive guide for AI development teams, easily integrated into the AI development lifecycle. Its uniqueness lies in its applicability to both classical machine learning tasks, such as supervised learning, and general-purpose AI downstream applications, like LLM-based or generative AI applications. Each section of the framework outlines the relevant technical, legal, ethical, and domain-specific skills required, and it provides background information and highlights current gaps in regulation and research.

The fairness assessment framework is grounded in the current literature on fairness and several national and international AI regulations, including the UNESCO Recommendation on the Ethics of Artificial Intelligence and various AI governance frameworks from the United States. These include the NIST AI Risk Management Framework, its companion document on generative AI, and the paper ‘Towards a Standard for Identifying and Managing Bias in Artificial Intelligence.’

Introduction to Fairness

Philosophical background

From a philosophical perspective, fairness is concerned with the morality of an action. Different approaches, such as consequentialism and deontology, offer different views on the definition of fairness. Consequentialism evaluates actions based on their outcomes, while deontological approaches focus on the morality of the actions themselves. The well-known ‘trolley problem’ illustrates the ethical dilemmas that arise in moral decision-making according to conflicting ethical theories, which are also reflected in some of the challenges and controversies in attempting to align AI systems with human values.

Fairness in classical machine learning models

In recent years, AI fairness has gained significant attention due to several prominent cases of bias embedded in AI systems. In 2014, for example, Amazon began building a tool to analyse job applications in order to identify suitable candidates. However, the system was trained on data based on previous applications to the tech company, mainly from men, and the resulting model rejected female applicants more often [1]. In 2018, Dr. Joy Buolamwini found in the Gender Shades Project that the gender classification tools of some leading tech companies performed significantly worse for women and darker-skinned people in terms of classification accuracy [2]. These real-life examples demonstrate, that there are different aspects and nuances within machine learning fairness and that the definition of a ‘fair’ system strongly depends on the context and use case.

Fairness metrics for classical machine learning models

Several so-called fairness metrics have been introduced in the past, which aim to serve as comparable and quantitative measures to assess discrimination in machine learning models or on the training dataset. In general, most fairness metrics for trained machine learning models are defined for supervised learning, in particular, for binary classification tasks, and they are calculated based on error or decision rates. These metrics can be adapted for use in regression or multi-class classification. In total, there are more than 20 fairness metrics, and several of them are interrelated or even incompatible in non-trivial cases. Moreover, each metric reflects different considerations and interpretations of fairness in a given context, and the concept of a ‘fair’ AI system may vary in different societal, legal and cultural settings. It is therefore important to carefully select a fairness metric in a given scenario that best reflects the nuances and subtleties of a setting. For example, in an AI-based hiring tool, fairness may be more related to achieving equal outcome rates across different demographic groups, whereas in an AI-based disease prediction tool, similar error rates are generally more relevant in the context of fairness.

Fairness in general-purpose AI

Unlike classical machine learning models, which are typically limited to specific use cases, general-purpose AI models can be adapted and applied to a wide variety of downstream tasks, making fairness assessments more complex and challenging [3][4]. In general, fairness must be evaluated both at the level of the base model and within each downstream application to account for the specifics of each use case. Common fairness metrics designed for classification can be applied in fairness assessments of specific general-purpose AI downstream applications, particularly when the use case resembles classification tasks. However, generally speaking, research on fairness metrics for general-purpose AI remains limited.

Fairness metrics for large language models

In the context of large language models, there are two main types of fairness metrics: intrinsic fairness metrics for the underlying LLM, and extrinsic fairness metrics, which evaluate the fairness for fine-tuned or adapted downstream tasks. The relationship between these two types of metrics is not fully understood, and some studies suggest that they are not necessarily correlated [5]. Fine-tuning can reduce or even override the fairness of the underlying base model, making it crucial to properly assess the fairness of downstream applications in the context of general-purpose AI.

Limitations of fairness metrics

In general, no single metric, or even combination of metrics, is likely to be able to fully capture all the relevant facets of a given situation. Due to the nuanced and dynamic nature of fairness, it should not be reduced to a purely technical concept or formula. Instead, fairness in AI systems must be assessed from a socio-technical perspective, requiring a multi-stage process that integrates both technical and human considerations. In addition to technical developers, experts from legal, philosophical, social, and domain-specific fields should be involved in the assessment of fairness. This interdisciplinary approach is crucial to account for as many fairness-relevant aspects of the real-world application as possible and to ensure that the complexities of fairness in practice are adequately addressed.

Red teaming

Due to the aforementioned challenges in fairness assessments, particularly in the context of general-purpose AI, red teaming has become a standard approach for identifying fairness-related vulnerabilities and risks beyond the calculation of fairness metrics. The National Institute for Standards in Technology (NIST) defines red teaming as ‘a group of people authorised and organised to emulate a potential adversary’s attack or exploitation capabilities against an organisation’s security posture’ [6]. Originating from cybersecurity, red teaming in the context of fairness testing for general-purpose AI typically involves adversarial testing to challenge the model for discriminatory behavior or harmful content. Counterfactual testing evaluates how changes in inputs, such as those within a protected class, affect outputs and can therefore provide valuable insight into potentially discriminatory behaviour of an AI system.

Limitations of red teaming in fairness assessments

Red teaming offers significant advantages in fairness assessments, as it enables highly targeted tests and uncovers vulnerabilities in AI systems that might otherwise go undetected. However, there is currently a lack of standardized methodologies, and existing approaches are diverse and often vaguely formulated [7]. Therefore, red teaming alone is not a sufficiently reliable indicator of a fair model and should be integrated into a multi-stage process for fairness assessment.

Additional Fairness-Relevant Aspects

In addition to red teaming and the calculation of fairness metrics, the following factors are relevant to fairness assessments:

1. Representative and Diverse Data: Ensuring that the training data, or the data used for fine-tuning or retrieval, is representative of the intended use case and demographically diverse is essential for minimizing bias in AI systems.

2. Usage of Sensitive Attributes: In addition, we must identify potentially sensitive attributes and their proxies, and assess whether their use may be prohibited by law or lead to discriminatory outcomes. On the other hand, in some cases, it may be beneficial to perform disaggregated analysis using sensitive attributes, particularly in medicine under specific circumstances. However, this must be approached with caution to avoid unintended biases or ethical concerns.

3. Respect for Privacy: Protecting the privacy of individuals is fundamental to the ethical development of AI systems, particularly when dealing with sensitive attributes such as race, gender or health data.

4. Monitoring and Auditing of Deployed Models: After deployment, model drift or changes in the data environment can introduce new biases into an AI system. It is essential to implement a monitoring system that continuously tracks the performance and fairness of the model and alerts the user in the event of performance degradation, new biases or other significant changes. In addition, regular audits should be conducted to identify additional vulnerabilities in the system.

5. Accessible Design: Accessibility is a key factor in the design of fair AI systems. Development teams should create systems that are as inclusive as possible, ensuring that they can be used by people of all abilities. In doing so, they can help reduce the digital divide and promote wider access to technology.

6. Diversity in the Development Team: Building a diverse development team is critical to responsible AI development. A diverse team brings a wider range of perspectives, making it easier to identify potential biases and vulnerabilities in the system. This diversity of thought helps mitigate risks that might otherwise go unnoticed.

The Fairness Assessment Framework

Access

You can access the framework here. For best quality, please download the file before viewing.

Stages

The fairness assessment framework consists of the following stages:

1. Use Case Analysis

2. Data Analysis and Debiasing

3. Fairness Metric Design

4. Design of General Tests

5. Red Teaming Test Design

6. Fair Application Development

7. Evaluations and Debiasing

8. Monitoring and Auditing

9. Documentation and Reporting

Use Case Analysis

The first part of the framework aims to provide a detailed analysis of the use case environment. It begins with identifying the stakeholders and assessing the diversity and skills of the individuals involved in the development of the AI system. Next, we recommend investigating whether there are established best practices for the use case, and identifying potentially vulnerable groups, as well as potential areas of discrimination, including biases and stereotypes. Additionally, it is important to evaluate whether the use of sensitive attributes could lead to discrimination or if differentiating between groups within a sensitive attribute is both desirable and essential. Lastly, the framework suggests identifying potentially harmful feedback loops in the model, specifying the societal goals of the application, and addressing inclusivity and accessibility standards. It also emphasizes defining the level of risk and impact of the application.

Data Analysis and Debiasing

The next part begins with a privacy assessment of the data used for training, fine-tuning, or retrieval. This involves ensuring compliance with privacy regulations, examining data collection methods, and potentially anonymizing the data. Based on the results of the use case analysis, it may be necessary to remove sensitive attributes and their proxies from the data or to disaggregate the data within groups. Additionally, we need to evaluate the representativeness, diversity, and inclusiveness of the data, while identifying any historical or social biases embedded within it. If discrepancies are found, they should be addressed by either collecting additional or higher-quality data, or by applying well-established debiasing techniques such as resampling or reweighing.

Identified gaps

• Due to the complexity of fairness and its dependence on cultural and societal factors, there is no clear standard or guidance on how to properly define a ‘diverse’ or ‘representative’ dataset. Simply reflecting real-world distributions may lead to poor outcomes for minority groups, thereby undermining democratic values. More research and the establishment of best practices in this area would be valuable to avoid reinforcing social inequalities through unbalanced data.

• In addition, altering or manipulating real-world data, as some debiasing techniques do, has far-reaching ethical implications. Currently, there is a lack of guidelines or consensus on which methods are permissible and morally acceptable. However, since debiasing techniques are integral to ensuring fair AI systems, it is crucial to provide clear guidance to developers in this area.

Fairness Metric Design

To assess the fairness of the trained model later in the AI development lifecycle, we can apply existing fairness metrics where appropriate, particularly for models based on labeled training data and categorical outputs. For non-categorical outputs, or when a suitable fairness metric does not yet exist, custom fairness metrics can be developed in collaboration with domain experts and potentially vulnerable groups, as recommended in the NIST Risk Assessment Framework [8].

In general, group fairness metrics require that certain values — such as the true positive rate — be the same across groups. However, this is rarely achievable in real-world scenarios. Therefore, as a final step in designing fairness metrics, we need to consider acceptable levels of difference or tolerance that can still indicate a fair system. This threshold should depend on factors such as the level of risk and the impact of the application.

Identified gaps

• Although there are some guidelines for selecting fairness metrics, they are often generalized and primarily focused on automated decision-making applications in (binary) classification. In my Master’s thesis at the University of Oxford, I developed a decision support system for binary classification that helps users select an appropriate metric based on the specifics of the use case and concepts from philosophy and law. However, as AI systems are introduced into more and more sensitive areas of our lives, more research in this area is needed.

• Given the limited current knowledge of fairness metrics for general-purpose AI, I suggest encouraging further research in this area.

• Another gap in the context of fairness metrics is the challenge of defining acceptable tolerance levels or intervals across groups, and determining which debiasing methods are morally acceptable. The 2023 paper ‘The Unfairness of Fair Machine Learning: Levelling Down and Strict Egalitarianism by Default’ by S. Wachter, B. Mittelstadt, and C. Russell highlights that some current debiasing methods result in lower performance or outcome rates for better-performing groups, solely to achieve parity or meet a certain tolerance level across all groups [9]. This approach is highly controversial from a moral standpoint and underscores the need for further research in this area.

• To address the gaps in fairness metrics beyond classification and regression, the NIST AI Risk Management Framework recommends creating and designing custom fairness metrics ‘in collaboration with domain experts and affected communities.’ I propose promoting research and the development of standards or more detailed guidance for designing fairness metrics, as well as potentially establishing a best practice platform, so that AI development teams do not have to design metrics from scratch.

Design of General Tests

In the next step of the framework, we design general tests for the AI model. These include selecting appropriate performance metrics and benchmark datasets, conducting accessibility testing through user experience evaluations, and, where appropriate, field testing to further investigate potential misbehavior of the model with vulnerable groups, as recommended by the NIST AI Risk Management Framework.

Identified gaps

• Over-reliance on the results of publicly available benchmarks carries certain risks. Studies have shown that incorporating benchmark datasets into the training or adaptation process can significantly improve test results when those same benchmarks are later used for bias testing. However, this does not necessarily indicate that the model itself is free of bias [10].

Red Teaming Test Design

Next, we design red teaming tests as an integral part of fairness assessment, particularly for generative AI applications. Adversarial tests are generally aimed at provoking discriminatory behavior and harmful outcomes from the model, and counterfactual tests assess how changes in input affect the outcome, helping to uncover biases in the application. If necessary, we can use generative AI to carefully expand the number of test cases, ensuring human oversight to avoid introducing new biases into the test data.

Identified gaps

• Currently, there are no standards for red teaming general-purpose AI applications to identify discrimination, either in terms of test designs or the people involved in the red teaming process. Poorly designed tests may fail to properly identify risks in the model, and there is no consensus on how to standardize the interpretation of red teaming results [7].

Fair Application Development

If technically feasible, fairness constraints in the form of fairness metrics can be incorporated into the training or adaptation process, or techniques such as fair representation learning or adversarial debiasing can be applied. We recommend training multiple models and selecting the one with the fewest fairness issues. For generative AI applications, extra care should be taken in designing the system prompt with potential fairness concerns in mind. Additionally, content moderation or output filtering can be implemented to prevent harmful content.

Evaluations and Debiasing

This stage focuses on executing all previously designed tests, including the calculation of performance and fairness metrics, as well as the execution of red teaming tests. The results should then be analyzed by experts to determine whether further action is necessary.

Identified gaps

• Currently, there are no guidelines for the use of benchmark datasets to identify bias in generative AI applications.

• Additionally, there is a lack of standards for interpreting the results of performance metrics, fairness metrics, benchmarking, red teaming, or field testing, as mentioned earlier.

• Similar to the data analysis and debiasing stage, there are no clear ethical guidelines regarding the implications of directly modifying the model’s behavior or its outputs.

Monitoring and Auditing

Assessing the fairness of an AI model before its initial deployment is just the first step in identifying risks early in the development lifecycle. Once the application is in production, ongoing monitoring and audits are necessary to address model drift, unforeseen events, or other changes in the system that may lead to bias and discrimination. Additionally, we should aim to regularly update the application to improve fairness as better data becomes available or as other enhancements to the model are identified.

Identified gaps

• Most existing fairness metrics are designed for the development and testing phases of an AI system and rely on labeled training data. However, since fairness in production is crucial, we need to focus more on developing fairness metrics suitable for monitoring AI systems in production.

• Additionally, it may be helpful to establish guidelines for the frequency of AI testing.

Documentation and Reporting

Some AI regulations, such as the EU AI Act, require reporting on safety and fairness outcomes. Therefore, one of the final steps is to report the results in accordance with the requirements of the relevant regulations. As a final step in the framework, we recommend establishing a centralized, monitored best practice platform that contains a detailed description of all the previously defined fairness assessment steps. This platform would allow AI development teams to build on existing work related to fairness, rather than having to create all the metrics and test cases from scratch. However, I strongly recommend that this platform be expert-supervised to ensure high-quality examples and guidance.

Application of the fairness assessment framework to real-life examples of bias from the past

This section discusses three real-life examples of biased AI applications from the past and how the use of the framework created would have helped identify and mitigate discriminatory behaviour.

AMS Berufsinfomat

The ‘Berufsinfomat’ was launched by the Austrian Public Employment Service (AMS) in January 2024 as a GPT-3.5-based chatbot to support jobseekers. Shortly after its launch, several newspapers reported that the chatbot reproduced stereotypes, particularly those based on gender [11]. For example, the bot recommended IT jobs to a young man, but suggested studying gender studies when the same question was asked by a young woman. In response to these reports, AMS made improvements to the bot, including adjusting the system prompt to prevent distinctions based on gender.

This case highlights the need for extensive fairness and safety testing before releasing a high-impact application like the Berufsinfomat. It is likely that the developers would have identified the risk of perpetuating stereotypes in job recommendations had they conducted thorough bias testing. Assuming the application was fine-tuned using employment data from AMS, an early-stage analysis of the data for gender imbalances or job distribution across protected groups, along with acknowledgment of historical inequalities embedded in the data, would have helped detect bias. At the latest, during the red teaming phase, counterfactual testing — such as submitting similar prompts for both a man and a woman — would have revealed the fairness issues within the application.

Predictive Policing

Another example of bias is the predictive policing tool ‘PredPol’ in the US [12]. The tool is a classic machine learning application that uses historical crime data to identify areas where crime is likely to occur. However, historical bias in the data — stemming from over-policed areas and potentially biased policing practices in the past — led to negative feedback loops, further reinforcing bias by sending even more police to those areas. This example illustrates the interconnectedness of technology and society, as well as their dynamic relationship and interdependence.

The fairness framework could have helped identify the risk of racial bias in the historical data during the initial ‘use case analysis’ phase, and recognize the feedback loop created by increasing police presence in certain areas. Additionally, a demographic analysis of the data could have highlighted disparities in high-risk outcomes.

Healthcare

In the third example of bias, researchers at University College London discovered that an algorithm for predicting liver disease failed to detect the disease in women twice as often as in men [13]. Gender bias is a well-known risk in healthcare applications, and the framework would have helped to identify this issue. If the symptoms of a disease vary between genders, it may be necessary to include the sensitive attribute or its proxies as input data. Calculating a fairness metric based on the false negative rates across genders could have further revealed the bias present in the model.

Conclusion and further improvements

The framework developed offers a comprehensive set of measures for identifying discrimination and fairness issues in AI systems. The three real-life examples highlight its strengths in uncovering biases within AI systems. This framework serves as an initial step in supporting AI development teams with the complex task of implementing and operationalizing fairness in their AI applications.

Possible suggestions for the future include:

• Integration of the guidance for fairness metric selection developed in my Master’s thesis into the framework: Integrating the decision support system for fairness metrics, as developed in my Master’s thesis, would provide the development teams of classical machine learning models with a structured approach to choose the most appropriate fairness metrics.

• A clearer distinction between general-purpose AI and classical machine learning in terms of required framework steps: General-purpose AI downstream applications may require additional steps such as field testing and red teaming to address fairness, whereas classical machine learning models, typically built for specific tasks, may need fewer steps. Clearly distinguishing which steps are essential for each type would help development teams focus their efforts where needed.

• Provide additional best practice examples of the framework: Providing concrete, real-world examples of the framework in action would illustrate its application in both classical machine learning and general-purpose AI. Detailed examples of how fairness metrics were applied, how red teaming was designed and executed, and how mitigations were implemented would offer development teams valuable insights.

• Further investigate the identified gaps: Several gaps in fairness assessments, such as the lack of standardized fairness metrics for general-purpose AI or the absence of ethical guidelines for altering model behavior, need further research. Future work should focus on exploring these gaps and proposing practical solutions, standards or best practices.

Final remark

I acknowledge that it is difficult, if not impossible, to fully address all the identified gaps in the framework. Given my technical background, I may also be biased or overly optimistic in attempting to break down the sophisticated concept of fairness into a logical sequence of steps to provide comprehensive guidance. However, I believe it is better to make an effort to transform highly complex socio-technical concepts into actionable steps rather than prematurely conclude that no satisfactory solution exists. If we do not try, we accept the status quo as it is.

References

[1] Insight — Amazon scraps secret AI recruiting tool that showed bias against women | Reuters

[2] Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy dispari ties in commercial gender classification. In Conference on fairness, accountability and transparency, pages 77–91. PMLR, 2018.

[3] Foundation models such as ChatGPT through the prism of the UNESCO Recommendation on the Ethics of Artificial Intelligence — UNESCO Digital Library

[4] Towards a Standard for Identifying and Managing Bias in Artificial Intelligence (nist.gov)

[5] Li, Yingji, et al. “A survey on fairness in large language models.” arXiv preprint arXiv:2308.10149 (2023).

[6] Red Team/Blue Team Approach — Glossary | CSRC (nist.gov)

[7] Feffer, Michael, et al. “Red-Teaming for Generative AI: Silver Bullet or Security Theater?.” arXiv preprint arXiv:2401.15897 (2024).

[8] AI Risk Management Framework | NIST

[9] Mittelstadt, Brent, Sandra Wachter, and Chris Russell. “The Unfairness of Fair Machine Learning: Levelling down and strict egalitarianism by default.” arXiv preprint arXiv:2302.02404 (2023).

[10] Zhou, Kun, et al. “Don’t make your llm an evaluation benchmark cheater.” arXiv preprint arXiv:2311.01964 (2023).

[11] Predictive policing algorithms are racist. They need to be dismantled. | MIT Technology Review

[12] Vorurteile und zweifelhafte Umsetzung: AMS-KI-Chatbot trifft auf Spott und Hohn — Netzpolitik — derStandard.at › Web

[13] Gender bias revealed in AI tools screening for liver disease | UCL News — UCL — University College London

The complete selection guidance is available for download below.

Fairness Metric Selection Guidance

224KB ∙ PDF file

Download

Download