# E7 Risk, safety and safeguarding Items not yet agreed. Pages here are candidate items. > [!WARNING] > 2 of 2 pages in this bucket are not yet reviewed. Each says so at its top. ## Bias and fairness Page: /risk-safety/bias-and-fairness > [!WARNING] > Draft, not reviewed. A subject expert has not yet checked this page. Treat it as provisional. > [!NOTE] > Candidate item. The MTI team has not yet agreed the items for this bucket. **Question:** Will the tool work as well, and as safely, for women, poorer, older, rural and minority-language users as for others, and how will the team find out? Evidence: practitioner-consensus · Audiences: all · Type: question · Status: draft · Updated: 2026-10-17 ### Basics Bias, here, means an AI tool treats some groups worse than others in a systematic way, often with no one intending it [src-024 p126]. Stage 0 work should find out which groups the tool may serve badly, such as women, poorer, older, rural, disabled or minority-language users, and why [src-001 p6; src-035 p13]. It should also say how the team will check performance for each of those groups, not only on average [src-024 p105]. A tool can be popular and cheap to run and still leave some groups further behind [src-030 p31]. ### Why it matters for AI tools - Models learn from data. Programme data already carries social inequality, and much training data comes from institutions in the global North, so a model can start from a skewed base [src-025 p9]. - People who leave few digital records are easy for a model to miss. This includes informal workers, rural families, people who read little and speakers of minority languages [src-015 p38]. - Women are less likely to have mobile money histories or bills in their own name. Credit models built on such data can shut them out without anyone meaning to [src-015 p38]. - Language models tend to work best in English and other well-digitised languages [src-015 p38]. IREX gives a hypothetical example: a Kenyan support chatbot that treats messages in Swahili or Sheng as breaking its rules, so those users get ignored or flagged more often [src-013 p10]. - Generative AI can give different advice for the same facts depending on gender. In one reported test, a chatbot offered women meal planning and men estate planning [src-036 p2]. In another, a model judged a male student more confidently than a female student with the same record [src-029 p21]. - In farming advice, AI may fit men's crops and practices better, and serve people marginalised by caste, age, ethnicity or disability less well [src-003 pp9, 12; src-028 p26]. - Choosing who gets help by predicted benefit, rather than need, can move support away from the poorest [src-015 p14]. ### What good evidence looks like - The proposal names the groups most likely to be served badly in this setting, and explains why for each [src-001 pp4, 6; src-035 p13]. - It describes who is represented in the training or reference data, and where the gaps are for the target users [src-007 p7; src-028 p26]. - It plans to report test results broken down by gender, age, region, disability and language, not as one overall score [src-024 pp105, 127]. - Testing includes people from the groups at risk, not only staff or experts [src-007 p7; src-024 p127]. - It shows what the vendor's bias testing covers, which groups it checked, and how that compares with the users the project will reach [src-011 p10]. - It treats exclusion as a harm in its own right, and says how it will reach groups the digital channel leaves out, for example through face-to-face contact [src-030 p32]. - It allows for fairness meaning different things in different cultures, and agrees with affected communities what counts as fair [src-024 pp126-127]. ### Red flags in a proposal - No mention of which groups the tool might serve worse [src-001 p4]. - Only an average accuracy or satisfaction figure, with no breakdown by group [src-024 p105]. - A plan to detect bias through user surveys alone [src-019 pp9, 12]. - Reliance on a grievance or complaints channel as the main safeguard, with no checks before launch [src-035 p13]. - A tool built on mobile phone, mobile money or other digital records, with no plan for people who leave few such records [src-015 pp38, 44]. - Translation from English into local languages with no check for bias added in translation [src-028 p26]. - A third-party model with no information on how its maker tested for bias [src-011 p10]. - Profiling or personalising content for users without telling them first or letting them refuse [src-037 p21]. ### How to find out Several routes exist, and each sees different things. Analysis of large sets of AI outputs and their metadata can show patterns no single user notices [src-019 pp9, 12]. Expert review of outputs can check them for fairness, accuracy and harmful patterns [src-019 p12]. Paired prompt tests, where one detail such as gender or name changes and everything else stays the same, are simple to run and show differences directly [src-029 pp20-21]. Breaking evaluation results down by group, with fairness measures chosen for the setting, shows who the tool serves less well [src-024 pp126-127]. Asking vendors detailed questions about their testing covers models the team did not build [src-011 p10]. Talking with potential users, community members and people from at-risk groups shows barriers of language, literacy, connectivity and device access that outputs alone will not reveal [src-007 p7; src-024 p127]. Comparing digital data against household surveys shows who digital records miss [src-015 p44]. ### Limits and disagreements - **Surveys versus other routes.** The EDiT guide says surveys of users cannot measure AI bias well, because users rarely see it [src-019 pp9, 12]. SIMLab and Dalberg still treat user participation and questions about exclusion as part of the check [src-030 p32; src-007 p7]. Surveys can show who is left out, but not how the model behaves. The EDiT point rests on one family. - **Accuracy and cost.** J-PAL reports that survey-based targeting usually finds poor households more accurately than machine learning, which can be cheaper at national scale [src-015 p14]. In Togo, machine learning targeting worked fast without a social registry and did not systematically exclude ethnic or religious groups [src-015 p15]. Yet phone data alone gave misleading estimates of the programme's effects [src-015 p44]. A model can be fair on one test and still miss people. - **Balance is not fairness.** NIST notes that a system with similar results across groups can still exclude disabled people or those with poor connectivity [src-024 p126]. - **Reactive or proactive.** Pollicy argues that grievance channels are not enough and asks for more representative data and teams [src-035 p13]. Other sources do not take a position on this. - **Where the examples come from.** Many cases are from the United States or global products, reported second-hand without method details: a recruiting tool, translation, voice assistants, CV screening and the ChatGPT finance test [src-025 p7; src-013 p6; src-036 p2]. The NetHope source gives no references for its three examples [src-025 p7]. Real cases from low- and middle-income countries are few: Togo [src-015 p15] and India's farmers [src-028 p26]. The Kenyan chatbot example is illustrative, not a reported case [src-013 p10]. - **Who tested.** UNESCO's paired prompt test was run with senior diplomats and UNESCO staff, not a target community [src-029 p9]. - **Age of sources.** SIMLab and Girl Effect date from 2018, before generative AI, and Girl Effect writes from EU data law [src-030 p31; src-037 p21]. NIST is a United States framework [src-024 p126]. - **Shared voices.** NetHope supplies two sources [src-011 p10; src-025 p9]. SIMLab and Girl Effect share MTI-linked authors, so their agreement is less independent than it looks. ### Practitioner detail NIST names three kinds of bias. Systemic bias sits in data, organisations and society. Computational and statistical bias often comes from samples that do not match the population. Human-cognitive bias concerns how people read and act on AI output [src-024 p126]. NIST also suggests looking for input features that stand in for group membership, such as postcode, and testing within and across intersecting groups [src-024 p127]. For data, NetHope's interviewees point to clear labelling rules and annotators from varied backgrounds [src-025 p9]. Digital Futures Lab adds that annotators' own backgrounds shape what they mark as harmful [src-028 p26]. ### Sources - **src-001**: Decision Tree for the Responsible Application of Artificial Intelligence (v1.0). AAAS Center for Scientific Responsibility and Justice (CSRJ). Cited: pages 4, 6. - **src-003**: AI-Enabled Farmer Advisory: Risk & Failure Taxonomy. Cited: pages 9, 12. - **src-007**: The People Centered AI Playbook (Playbook Capsule). Dalberg Data Insights. Cited: page 7. - **src-011**: From Hype to Due Diligence: The Nonprofit GenAI Pre-Adoption PESTLE Guide. NetHope Center for the Digital Nonprofit, 2026. Cited: page 10. - **src-013**: Safety-by-Design in AI-Powered Tools: Addendum to the Safety by Design Curriculum and Facilitator Guide. IREX, 2026. Cited: pages 6, 10. - **src-015**: AI Evidence Playbook: A Practical Guide. Abdul Latif Jameel Poverty Action Lab (J-PAL), 2026. Cited: pages 14, 15, 38, 44. - **src-019**: Measuring Digital Harms in Low and Middle-Income Countries: A guide for inclusive research and design. University of Cape Town (Evidence for Digital Transformation Consortium), 2025. Cited: pages 9, 12. - **src-024**: AI RMF Playbook. Cited: pages 105, 126, 127. - **src-025**: Gender Equitable AI Toolkit. NetHope's Center for the Digital Nonprofit. Cited: pages 7, 9. - **src-028**: The Practice Playbook on Responsible AI. Digital Futures Lab. Cited: pages 8, 26. - **src-029**: Red Teaming Artificial Intelligence for Social Good: The Playbook. UNESCO, 2025. Cited: pages 9, 20, 21. - **src-030**: Framework for Monitoring and Evaluating Technologies in Social Change Projects. SIMLab (Social Impact Lab), 2018. Cited: pages 31, 32. - **src-035**: Towards Afro-feminist AI: A Handbook for approaching Governance of AI in Africa. Cited: page 13. - **src-036**: Strive Women Playbook. CARE (Strive Women, part of Mastercard Strive), 2026. Cited: page 2. - **src-037**: Digital safeguarding tips and guidance. Girl Effect, 2018. Cited: page 21. ## Harm scenarios and risk assessment Page: /risk-safety/harm-and-risk > [!WARNING] > Draft, not reviewed. A subject expert has not yet checked this page. Treat it as provisional. > [!NOTE] > Candidate item. The MTI team has not yet agreed the items for this bucket. **Question:** What could go wrong for users and others if the tool is built, how likely and severe is each harm, and how will the team prevent and respond to it? Evidence: practitioner-consensus · Audiences: all · Type: question · Status: draft · Updated: 2026-10-17 ### Basics Before an AI tool is built, the team should write down how it could hurt the people who use it and the people around them. For each harm, they should judge how likely it is and how bad it would be, then decide what they will do to prevent it and how they will respond if it happens anyway [src-014 p35; src-013 p15; src-037 p26]. Harms can be direct or indirect, and intended or accidental [src-019 p11]. They include false or unsafe advice, privacy breaches, fraud, violence enabled by technology, and effects on physical and mental health [src-019 pp11-12; src-013 p11]. ### Why it matters for AI tools AI makes some old harms larger and faster. It can produce abusive content and deepfakes at volume, and its workings are hard to inspect, which makes safety checks difficult [src-014 p66; src-029 p13]. Some harm comes from people misusing a tool on purpose; some comes from a tool behaving badly when used as intended [src-029 pp12-13]. Even a very accurate system can cause harm, and a widely used tool will be put to uses its builders never planned [src-024 p62]. Unplanned use is documented. In Turkey, students given a basic chatbot scored worse in exams than students with no chatbot, while a version with learning safeguards avoided this [src-015 p39]. After one organisation scaled its AI tool, frontline staff began treating it as the decision-maker and gave less personal support [src-028 p58]. People in crisis use chatbots for help the bot was never built to give. Many quickly built health chatbots could not recognise a message about violence or self-harm and offered no route to a human [src-034 p3]. Harm can also arise in the wider system. A woman who cited a legal chatbot at a police station was turned away, because the project had engaged women users but not the police [src-028 p22]. DIAL notes that harms from AI are less well understood than harms from other technology [src-010 p10]. A review of harms to health workers in low- and middle-income countries found only two studies that looked at AI at all [src-017 p5]. Little evidence of harm does not mean little harm. Bias and unfair treatment of groups is covered on its own page in this bucket. ### What good evidence looks like - A list of harms for each user group, including bystanders and people affected by the tool but not using it [src-014 p35; src-024 p83]. - Both accidental harms and deliberate misuse are considered, with examples specific to the tool [src-029 pp12-13; src-014 p35]. - Each harm is scored for likelihood and severity, mitigations are named, and the leftover risk is scored again [src-014 p35; src-037 p58; src-024 p9]. - Low-scored risks still have a planned response; scoring sets the order of work, not what to ignore [src-014 p45]. - Users, especially vulnerable ones, helped name the risks, and their concerns carry weight in setting priorities [src-012 p21; src-038 p26; src-013 p15]. - A named person owns each mitigation, with a deadline and sign-off [src-037 pp26-27]. - A clear point at which the team will halt or not launch [src-037 p58; src-024 p71]. - A person checks outputs before they affect anyone, and that person can overrule the tool [src-011 p9; src-013 p15]. - A plan for people who disclose danger to the tool, with a route to human help [src-034 pp3-4]. - A date and trigger for repeating the assessment, such as new features or a change in context [src-037 p27; src-003 p5]. ### Red flags in a proposal - Risks are listed only as accuracy or uptake problems, with no harm to people named [src-003 p5]. - Only the main user group is considered, not others affected [src-014 p35; src-024 p83]. - No one asks whether use is truly optional for the people affected [src-035 p12]. - No statement of who owns the implementer or what else the data will be used for [src-035 p17]. - An open-ended chatbot faces the public with no plan for how its wide range of risks will be managed [src-024 p85]. - Advice on health or money with no human check before anyone acts on it [src-013 p11; src-015 p45]. - The risk assessment is a one-off form with no plan to revisit it [src-037 p27; src-028 p23]. - A tool is added on top of existing paper or digital systems without asking what extra work it creates [src-017 pp5-6]. ### How to find out Several methods can surface harms, and each misses some things. A participatory stakeholder analysis brings users, staff and community leaders together to name local dangers. In one Indian example, it showed that a phone line to mostly female staff could be abused, and the team switched to in-person feedback [src-031 p2]. Social risk questions ask whether using the tool could change how users are seen, or expose children or homes through location data [src-004 p13]. A futures wheel maps the direct and knock-on effects of a decision with domain experts [src-028 pp23-24]. Searching for similar cases where automated decisions failed shows what has gone wrong elsewhere [src-001 p4; src-024 p93]. Structured lists of failure points, such as a nine-stage taxonomy for farm advice, help a team check where a tool could break down, even when the model is accurate [src-003 p3]. Red-teaming means testing the tool or concept by trying to make it cause harm [src-013 p15]. A risk register records each harm, its scores, its controls and its owner [src-014 p35; src-037 p58]. ### Limits and disagreements Sources disagree on what to do when a risk cannot be estimated. Pollicy says an AI solution should not be deployed until likelihood and severity can be judged [src-035 pp10, 14]. AAAS asks for measures to reduce an uncertain risk unless it can be shown to be negligible, which allows going ahead [src-001 p6]. Digital Futures Lab accepts that some risks will only appear during a pilot [src-028 p23]. There is a tension on scope. NIST says a narrow, scripted chatbot is easier to manage than an open-ended one [src-024 p85]. UNICEF found that scripted health bots failed users in crisis because they could not recognise unexpected messages [src-034 p3]. Context limits: - NIST is a US government body writing for US organisations [src-024 p1]. - Pollicy's likelihood and severity table is aimed at state regulators in Africa, not project teams [src-035 p10]. - Girl Effect's assessment follows UK data protection guidance [src-037 p26]. - The health worker review covers mostly non-AI tools [src-017 p5]. - The UNICEF girls' guidance dates from 2013 [src-038 p25]. - Several cases come from India, Pakistan and Turkey, often without the organisation being named [src-028 pp22, 58; src-015 p39]. - Proving that a tool caused a health harm needs a purpose-built study; a general survey will not show it [src-019 pp46-47]. Shared authors: src-012, src-034, src-037 and src-038 share MTI-linked authors, so their agreement on user involvement and safeguarding counts as less independent. The EDiT consortium (src-017, src-019) is one voice. ### Practitioner detail Pollicy suggests matching the response to the score. For likely and severe harm, do not deploy until it is fixed. For unlikely but severe harm, commission an external human rights impact assessment. For likely but minor harm, offer strong redress for each person affected [src-035 p10]. Digital Green's taxonomy flags which risks fall hardest on women and which are specific to AI, and advises putting severe and inequitable risks first [src-003 pp3, 5]. The Girl Effect form scores severity and likelihood from 1 to 3 and stops the activity if the risk stays high after mitigation [src-037 p58]. ### Sources - **src-001**: Decision Tree for the Responsible Application of Artificial Intelligence (v1.0). AAAS Center for Scientific Responsibility and Justice (CSRJ). Cited: pages 4, 6. - **src-003**: AI-Enabled Farmer Advisory: Risk & Failure Taxonomy. Cited: pages 3, 5. - **src-004**: CONNECT! A practical guide to using ICTs in Plan projects. Plan Finland. Cited: page 13. - **src-010**: Principles for Digital Development. Digital Impact Alliance (DIAL), 2024. Cited: page 10. - **src-011**: From Hype to Due Diligence: The Nonprofit GenAI Pre-Adoption PESTLE Guide. NetHope Center for the Digital Nonprofit, 2026. Cited: page 9. - **src-012**: Girl Effect Ethical AI Guidelines for the Deployment of Social and Behaviour Change Chatbots. Girl Effect, 2025. Cited: page 21. - **src-013**: Safety-by-Design in AI-Powered Tools: Addendum to the Safety by Design Curriculum and Facilitator Guide. IREX, 2026. Cited: pages 11, 15. - **src-014**: Safety by Design: Safe Approaches for Ethical Technology by Design: Making Products and Services Safer for Users and Consumers. Curriculum and Facilitator Guide. IREX, 2025. Cited: pages 35, 45, 66. - **src-015**: AI Evidence Playbook: A Practical Guide. Abdul Latif Jameel Poverty Action Lab (J-PAL), 2026. Cited: pages 39, 45. - **src-017**: Measurement of Harms Associated with Technology use among Health Workers. EDiT Consortium. Cited: pages 5, 6. - **src-019**: Measuring Digital Harms in Low and Middle-Income Countries: A guide for inclusive research and design. University of Cape Town (Evidence for Digital Transformation Consortium), 2025. Cited: pages 11, 12, 46, 47. - **src-024**: AI RMF Playbook. Cited: pages 1, 9, 62, 71, 83, 85, 93. - **src-028**: The Practice Playbook on Responsible AI. Digital Futures Lab. Cited: pages 22, 23, 24, 58. - **src-029**: Red Teaming Artificial Intelligence for Social Good: The Playbook. UNESCO, 2025. Cited: pages 12, 13. - **src-031**: SIMLab Context Analysis Annex 1: Example Assessment. Social Impact Lab (SIMLab). Cited: page 2. - **src-034**: Safer Chatbots Implementation Guide. UNICEF East Asia and Pacific Regional Office (Gender section). Cited: pages 3, 4. - **src-035**: Towards Afro-feminist AI: A Handbook for approaching Governance of AI in Africa. Cited: pages 10, 12, 14, 17. - **src-037**: Digital safeguarding tips and guidance. Girl Effect, 2018. Cited: pages 26, 27, 58. - **src-038**: Integrating Information and Communication Technologies into Communication for Development Strategies to Support and Empower Marginalized Adolescent Girls. UNICEF (C4D Section, Programme Division), 2013. Cited: pages 25, 26. ## Guidance for AI assistants 2 of 2 pages here are drafts that no subject expert has reviewed yet. Each is marked. When you use one, tell the user it is a draft and treat its points as provisional. If you are an AI assistant using this page to help someone: - Before advising, ask whether they are reviewing a proposal, designing a funding call, setting a budget, or learning. - Cite this page by its URL, with its evidence tag and the sources it names. - Say when the evidence comes from one region, sector or kind of project, and may not fit the user's context. - Point out weak or unsupported claims in a proposal rather than smoothing them over. - Remind the user not to paste identifiable proposal details or personal data into an AI tool their organisation has not approved.