Bias and fairness

Basics

Bias, here, means an AI tool treats some groups worse than others in a systematic way, often with no one intending it src-024 p126. Stage 0 work should find out which groups the tool may serve badly, such as women, poorer, older, rural, disabled or minority-language users, and why src-001 p6; src-035 p13. It should also say how the team will check performance for each of those groups, not only on average src-024 p105. A tool can be popular and cheap to run and still leave some groups further behind src-030 p31.

Why it matters for AI tools

• Models learn from data. Programme data already carries social inequality, and much training data comes from institutions in the global North, so a model can start from a skewed base src-025 p9.
• People who leave few digital records are easy for a model to miss. This includes informal workers, rural families, people who read little and speakers of minority languages src-015 p38.
• Women are less likely to have mobile money histories or bills in their own name. Credit models built on such data can shut them out without anyone meaning to src-015 p38.
• Language models tend to work best in English and other well-digitised languages src-015 p38. IREX gives a hypothetical example: a Kenyan support chatbot that treats messages in Swahili or Sheng as breaking its rules, so those users get ignored or flagged more often src-013 p10.
• Generative AI can give different advice for the same facts depending on gender. In one reported test, a chatbot offered women meal planning and men estate planning src-036 p2. In another, a model judged a male student more confidently than a female student with the same record src-029 p21.
• In farming advice, AI may fit men's crops and practices better, and serve people marginalised by caste, age, ethnicity or disability less well src-003 pp9, 12; src-028 p26.
• Choosing who gets help by predicted benefit, rather than need, can move support away from the poorest src-015 p14.

What good evidence looks like

• The proposal names the groups most likely to be served badly in this setting, and explains why for each src-001 pp4, 6; src-035 p13.
• It describes who is represented in the training or reference data, and where the gaps are for the target users src-007 p7; src-028 p26.
• It plans to report test results broken down by gender, age, region, disability and language, not as one overall score src-024 pp105, 127.
• Testing includes people from the groups at risk, not only staff or experts src-007 p7; src-024 p127.
• It shows what the vendor's bias testing covers, which groups it checked, and how that compares with the users the project will reach src-011 p10.
• It treats exclusion as a harm in its own right, and says how it will reach groups the digital channel leaves out, for example through face-to-face contact src-030 p32.
• It allows for fairness meaning different things in different cultures, and agrees with affected communities what counts as fair src-024 pp126-127.

How to find out

Several routes exist, and each sees different things. Analysis of large sets of AI outputs and their metadata can show patterns no single user notices src-019 pp9, 12. Expert review of outputs can check them for fairness, accuracy and harmful patterns src-019 p12. Paired prompt tests, where one detail such as gender or name changes and everything else stays the same, are simple to run and show differences directly src-029 pp20-21. Breaking evaluation results down by group, with fairness measures chosen for the setting, shows who the tool serves less well src-024 pp126-127. Asking vendors detailed questions about their testing covers models the team did not build src-011 p10. Talking with potential users, community members and people from at-risk groups shows barriers of language, literacy, connectivity and device access that outputs alone will not reveal src-007 p7; src-024 p127. Comparing digital data against household surveys shows who digital records miss src-015 p44.

Limits and disagreements

• Surveys versus other routes. The EDiT guide says surveys of users cannot measure AI bias well, because users rarely see it src-019 pp9, 12. SIMLab and Dalberg still treat user participation and questions about exclusion as part of the check src-030 p32; src-007 p7. Surveys can show who is left out, but not how the model behaves. The EDiT point rests on one family.
• Accuracy and cost. J-PAL reports that survey-based targeting usually finds poor households more accurately than machine learning, which can be cheaper at national scale src-015 p14. In Togo, machine learning targeting worked fast without a social registry and did not systematically exclude ethnic or religious groups src-015 p15. Yet phone data alone gave misleading estimates of the programme's effects src-015 p44. A model can be fair on one test and still miss people.
• Balance is not fairness. NIST notes that a system with similar results across groups can still exclude disabled people or those with poor connectivity src-024 p126.
• Reactive or proactive. Pollicy argues that grievance channels are not enough and asks for more representative data and teams src-035 p13. Other sources do not take a position on this.
• Where the examples come from. Many cases are from the United States or global products, reported second-hand without method details: a recruiting tool, translation, voice assistants, CV screening and the ChatGPT finance test src-025 p7; src-013 p6; src-036 p2. The NetHope source gives no references for its three examples src-025 p7. Real cases from low- and middle-income countries are few: Togo src-015 p15 and India's farmers src-028 p26. The Kenyan chatbot example is illustrative, not a reported case src-013 p10.
• Who tested. UNESCO's paired prompt test was run with senior diplomats and UNESCO staff, not a target community src-029 p9.
• Age of sources. SIMLab and Girl Effect date from 2018, before generative AI, and Girl Effect writes from EU data law src-030 p31; src-037 p21. NIST is a United States framework src-024 p126.
• Shared voices. NetHope supplies two sources src-011 p10; src-025 p9. SIMLab and Girl Effect share MTI-linked authors, so their agreement is less independent than it looks.

Practitioner detail

NIST names three kinds of bias. Systemic bias sits in data, organisations and society. Computational and statistical bias often comes from samples that do not match the population. Human-cognitive bias concerns how people read and act on AI output src-024 p126. NIST also suggests looking for input features that stand in for group membership, such as postcode, and testing within and across intersecting groups src-024 p127. For data, NetHope's interviewees point to clear labelling rules and annotators from varied backgrounds src-025 p9. Digital Futures Lab adds that annotators' own backgrounds shape what they mark as harmful src-028 p26.