AI Bias / Bias is often defined as a “tendency, inclination, or prejudice for or against something or someone.” Humans constantly experience biases. Our brains are programmed to be biased. Why let’s us quickly explain. We, as a human, have two thinking systems:
- System 1 refers to automatic, fast thinking that operates with little to no voluntary control. This system generates our impressions and intuitions and informs our “gut feelings.”
- System 2 involves our active thinking. It’s a conscious effort, and is linked to you studying or more broadly put ilinked to deliberate action and choice.
We need System 1 thinking to help us organize and manage all the stimuli we constantly encounter, but cognitive biases also emerge from this system. System 1 thinking relies on associations and categories to recognize patterns and is therefore able to make quick and efficient judgments. We learn to create associations and categories based on our personal experiences, education, upbringing, and communities and as a consquence we have stereotypes and norms that come with them.
AI systems, are not bound by System 1 thinking. They can make predictions and decisions that are less biased and even identify human biases. While this is exciting, it is not the whole story. AI systems are human creations and can still contain bias (either because the AI system learned this bias from the data or due to the way it was built). As a result, AI systems can perpetuate and amplify discrimination or marginalization of certain groups, even when they are functioning correctly.
A common example of bias in AI is the use of AI in recruitment processes. Suppose an AI system is trained on data from primarily male candidates, this happens! for example, in 2018, Amazon ecalled its AI-driven hiring tool due to tit’s favor of male candidates over female candidates. The model would penalizing candidates whose resumes included the word “women’s” . Therefore womens were rejected even if these female candidates were equally or better qualified. This is a form of sample bias or selection bias. 1
Why are AI systems biased?
Simply put, AI systems are biased because they are human creations. They are classification technologies and are products of the context in which they are created. Okay that’s still a bit vague, let’s make it more tangible. Bias can enter an AI model during the development and use of a machine learning AI system. It can arise during the generation, collection, labeling, and management of data that the algorithm learns (it’s training data) from, as well as how the algorithm learns, the design, from this data en how these algortihms are evaluated. This is a problem, as this happens often unknowingly, as how AI systems work tend to be a black box. Therefore, always remember that AI can be baised and therefore this can result in inaccurate predictions and/or discriminatory outputs.
Additionally, it matters who develops AI systems. The perspectives and knowledge of those who develop AI systems are integrated into them, while the tech companies and labs creating many large-scale AI systems tend to be predominantly male and white.
When examining the causes of bias in AI, we see three main sources: bias in the training data itself, bias in the models behind, and bias in the use of generative AI itself.
Training Data
Let’s start with the training data. The dataset(s) used to train an algorithm are the first step in making a model like ChatGPT, because AI systems learn to make decisions based on these training datasets. However, there are several points where bias can enter.
First, the composition, what enters as training dataset, often depends on human choices. Which sources should we include, which sources shouldn’t be included and how data from different categories are weighted. Models like ChatGPT are trained on vast datasets, so maybe that is less of a problem (it is not, but let’s say hypothetically it is). An implicit assumption would be that the data points are accurat reflections of the real world, this is not the case, there are significant data gaps. Sometimes, data on specific groups or regions are missing.
In 2017, around 2.3 billion women worldwide do not have any Internet access, and more than 1.7 billion do not own a mobile phonesome 200 million fewer women than men have online access or mobile phones.2
For example, rural communities in low-income countries often generate little to no digital data because of limited access to the internet or smartphones, while urban populations in high-income countries, particularly those in wealthier, predominantly white neighborhoods, tend to generate huge amounts of data due to the digitalization. If certain groups are underrepresented or overrepresented in the training data, a distorted view of reality (in AI land) arises. This skewed representation can lead to distorted and unfair outcomes. A feedback loop can also occur when data from the past reflect injustice and discrimination, causing the AI to repeat these patterns.
A feedback loop can also occur when data from the past reflect inequalities and discrimination, causing the AI to repeat these patterns. For example let’s extent the example mentioned in the introduction , if historical hiring data used to train an AI recruitment tool reflects systemic racism by favoring white candidates over equally qualified Black, Latino, or Indigenous candidates, the AI model is more likely to reinforce this bias, unfairly rejecting applications from people of color. Conversely, if the AI is trained to overcorrect by prioritizing candidates from underrepresented racial groups without considering qualifications, it may create new forms of bias, undermining perceptions of fairness and meritocracy. If predictive policing systems trained on biased historical crime data may disproportionately target Black and Latino neighborhoods, not because these areas are more prone to crime, but because past policing practices unfairly focused on these communities. This can lead to a cycle where certain racial groups are overpoliced, while others are underserved, reinforcing inequities that exist in society.
Natural Language Processing algorithms, the models behind ChatGPT rely on thus on the published text corpora that are available online, and when the first models were trained, the free text corpora on the internet. Often associated with less quality. Word embeddings, how these models represents words in computer lend, can (will) show biased language, For example, around gender and ethnic stereotypes that are replicated when the tool is asked to make new associations. 3 . These word embeddings are numbers, and therefore you can operate arithmetic operations. For example, doctor minus father plus mother, results in …… “nurse” .
at last, Reinforcement Learning from Human Feedback (RLHF) means that a human influence is once again at play within the traning data. The way data is labeled or categorized depends on human annotators, whose own, often unconscious, biases can end up in the dataset (their system 1!). A common example arefacial recognition systems. Annotators may label images in ways that unintentionally prioritize lighter-skinned individuals due to preexisting societal norms or biases. This can lead to systems that work well for white individuals but perform poorly for Black or darker-skinned individuals, resulting in harmful inaccuracies, such as higher rates of misidentification for people of color in criminal justice applications. Similarly, in content moderation systems, annotators might unintentionally flag speech from certain racial or cultural groups as offensive based on their own biases or lack of cultural understanding, leading to disproportionate censorship of marginalized voices.
Different sources of these bias can amplify each other! What if A dataset used to train neural networks to identify skin cancer from photographs had <5% images of dark-skinned individuals4, and this is reinforce with bias from RHLF.
AI Bias asserting from the Model
Second, bias can creep into an algorithm at various points. It can arise when the algorithm is assigned a goal. For instance, AI models that work with binary categories (e.g., “fraud/no fraud” or “male/female”) struggle to recognize nuances. This can lead to distortion, especially for groups or situations that do not fit neatly into these binaries. Bias can also emerge in selecting inputs. An algorithm can inadvertently perpetuate inequality by using indirect variables (so-called “proxies”).
For example, a used healthcare algorithm in the United States falsely concluded that Black patients were healthier than equally sick white patients. The algorithm used healthcare costs as a proxy for health needs, but Black patients with the same level of need tend to spend less money due to unequal access to care, systemic inequities, and other factors.5
AI Bias from usage
Finally, bias can also emerge in the use of AI systems, both on the algorithmic side, such as when an AI system systematically excludes or disadvantages certain groups, and on the user side. If users accept the AI’s results as “normal,” that outcome is reinforced. The AI then continues to learn based on this confirmation, creating a vicious cycle that strengthens biases. Also a “correct” AI result can be misinterpreted if the user themselves approaches the outcome with biases.
The AI Bias Paradox
The AI Bias Paradox refers to the phenomenon that AI systems, designed to be objective, can actually reinforce existing biases because they are trained on human-created data. This creates the illusion of objectivity while reproducing or even amplifying bias.
An example involves algorithms in the criminal justice system, such as the COMPAS system, which is designed to predict the likelihood of recidivism. Research has shown that certain groups are unfairly labeled as higher risk. This distortion is often a direct result of historical or social differences that have entered the dataset and are unintentionally reproduced by the AI. The AI takes over human decisions and insights but scales them into a system deployed on a large scale, leading to amplified inequalities.
This brings us to the core of the AI Bias Paradox: AI systems are intended to provide objective insights but are based on a world historically characterized by bias and unequal treatment. This creates the appearance of neutrality, while these systems can adopt inherent inequalities. This underscores the importance of recognizing that AI does not generate automatic “truth” but operates based on human-formed data and assumptions.
Mitigating AI Bias
In this list, we share practical recommendations to recognize, evaluate, and actively reduce AI bias. The goal is to use AI more consciously and effectively without blindly trusting the generated output. The recommendations are as follows:
- Be skeptical: Do not automatically assume that AI output is correct, and critically assess the generated information. AI models can make mistakes or contain biases.
- Compare outputs from different chatbots: By comparing outputs from multiple AI models (such as ChatGPT, Copilot, Gemini, Claude), you can better understand the strengths, weaknesses, and biases of different systems.
- Use diverse prompts: Experiment with various formulations and questions to see how input affects output. This helps you understand how to limit or avoid bias in the output.
- Analyze personas: Examine how different personas or roles assigned to a chatbot influence the output. Pay attention to implicit assumptions and how they can affect the tone, content, and potential toxicity of the responses.
- Evaluate texts for biases: Reverse the roles and use AI to analyze texts for possible biases, both in your own work and in texts by others. This can help you become more aware of biases.
- Verify information: Check whether AI-generated sources are accurate and reliable. Do not trust information blindly without verifying it.
- Conduct independent research: Use AI as a tool, but continue to think critically, conduct research, and take responsibility for your own analyses. Do not rely entirely on AI output.
Conclusion
AI Bias is inextricably linked to human thinking and human-made systems. AI, often presented as objective, can reproduce and amplify these biases. To minimize this, it is essential to be aware of the various causes of bias, from data to algorithms and usage applications. Human judgment and responsibility remain essential: AI can assist, but not replace the academic thinker!
Want to learn more about bias in AI check out this Breaking Down Bias: Does Your Username Affects Chatbot Responses?
- Dastin, J. (2018). Amazon scraps secret AI recruiting tool that showed bias against women. Reuters. Retrieved from https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias -against-women-idUSKCN1MK08G. ↩︎
- A call to action for gender equality and women’s economic empowerment (2017). UN High Level Panel on Women’s Economic Empowerment. ↩︎
- Garg, N., Schiebinger, L., Jurafsky, D., & Zou, J. (2018). Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of Sciences, 115(16). doi: 10.1073/pnas.1720347115 ↩︎
- Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017). Dermatologist-level classification of skin cancer with deep neural network ↩︎
- Obermeyer, Z., Powers, B. Vogeli, C., Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366 (6464), 447-453. ↩︎
