--- id: kn-e7-harm-and-risk title: Harm scenarios and risk assessment bucket: E7 Risk, safety and safeguarding item: null topic: harm-scenarios question: What could go wrong for users and others if the tool is built, how likely and severe is each harm, and how will the team prevent and respond to it? type: question audiences: - all evidence: practitioner-consensus status: draft reviewed_by: [] updated: 2026-10-17 url: /risk-safety/harm-and-risk sources: - id: src-001 pages: - 4 - 6 - id: src-003 pages: - 3 - 5 - id: src-004 pages: - 13 - id: src-010 pages: - 10 - id: src-011 pages: - 9 - id: src-012 pages: - 21 - id: src-013 pages: - 11 - 15 - id: src-014 pages: - 35 - 45 - 66 - id: src-015 pages: - 39 - 45 - id: src-017 pages: - 5 - 6 - id: src-019 pages: - 11 - 12 - 46 - 47 - id: src-024 pages: - 1 - 9 - 62 - 71 - 83 - 85 - 93 - id: src-028 pages: - 22 - 23 - 24 - 58 - id: src-029 pages: - 12 - 13 - id: src-031 pages: - 2 - id: src-034 pages: - 3 - 4 - id: src-035 pages: - 10 - 12 - 14 - 17 - id: src-037 pages: - 26 - 27 - 58 - id: src-038 pages: - 25 - 26 --- # Harm scenarios and risk assessment > [!WARNING] > Draft, not reviewed. A subject expert has not yet checked this page. Treat it as provisional. > [!NOTE] > Candidate item. The MTI team has not yet agreed the items for this bucket. **Question:** What could go wrong for users and others if the tool is built, how likely and severe is each harm, and how will the team prevent and respond to it? Evidence: practitioner-consensus · Audiences: all · Type: question · Status: draft · Updated: 2026-10-17 ## Basics Before an AI tool is built, the team should write down how it could hurt the people who use it and the people around them. For each harm, they should judge how likely it is and how bad it would be, then decide what they will do to prevent it and how they will respond if it happens anyway [src-014 p35; src-013 p15; src-037 p26]. Harms can be direct or indirect, and intended or accidental [src-019 p11]. They include false or unsafe advice, privacy breaches, fraud, violence enabled by technology, and effects on physical and mental health [src-019 pp11-12; src-013 p11]. ## Why it matters for AI tools AI makes some old harms larger and faster. It can produce abusive content and deepfakes at volume, and its workings are hard to inspect, which makes safety checks difficult [src-014 p66; src-029 p13]. Some harm comes from people misusing a tool on purpose; some comes from a tool behaving badly when used as intended [src-029 pp12-13]. Even a very accurate system can cause harm, and a widely used tool will be put to uses its builders never planned [src-024 p62]. Unplanned use is documented. In Turkey, students given a basic chatbot scored worse in exams than students with no chatbot, while a version with learning safeguards avoided this [src-015 p39]. After one organisation scaled its AI tool, frontline staff began treating it as the decision-maker and gave less personal support [src-028 p58]. People in crisis use chatbots for help the bot was never built to give. Many quickly built health chatbots could not recognise a message about violence or self-harm and offered no route to a human [src-034 p3]. Harm can also arise in the wider system. A woman who cited a legal chatbot at a police station was turned away, because the project had engaged women users but not the police [src-028 p22]. DIAL notes that harms from AI are less well understood than harms from other technology [src-010 p10]. A review of harms to health workers in low- and middle-income countries found only two studies that looked at AI at all [src-017 p5]. Little evidence of harm does not mean little harm. Bias and unfair treatment of groups is covered on its own page in this bucket. ## What good evidence looks like - A list of harms for each user group, including bystanders and people affected by the tool but not using it [src-014 p35; src-024 p83]. - Both accidental harms and deliberate misuse are considered, with examples specific to the tool [src-029 pp12-13; src-014 p35]. - Each harm is scored for likelihood and severity, mitigations are named, and the leftover risk is scored again [src-014 p35; src-037 p58; src-024 p9]. - Low-scored risks still have a planned response; scoring sets the order of work, not what to ignore [src-014 p45]. - Users, especially vulnerable ones, helped name the risks, and their concerns carry weight in setting priorities [src-012 p21; src-038 p26; src-013 p15]. - A named person owns each mitigation, with a deadline and sign-off [src-037 pp26-27]. - A clear point at which the team will halt or not launch [src-037 p58; src-024 p71]. - A person checks outputs before they affect anyone, and that person can overrule the tool [src-011 p9; src-013 p15]. - A plan for people who disclose danger to the tool, with a route to human help [src-034 pp3-4]. - A date and trigger for repeating the assessment, such as new features or a change in context [src-037 p27; src-003 p5]. ## Red flags in a proposal - Risks are listed only as accuracy or uptake problems, with no harm to people named [src-003 p5]. - Only the main user group is considered, not others affected [src-014 p35; src-024 p83]. - No one asks whether use is truly optional for the people affected [src-035 p12]. - No statement of who owns the implementer or what else the data will be used for [src-035 p17]. - An open-ended chatbot faces the public with no plan for how its wide range of risks will be managed [src-024 p85]. - Advice on health or money with no human check before anyone acts on it [src-013 p11; src-015 p45]. - The risk assessment is a one-off form with no plan to revisit it [src-037 p27; src-028 p23]. - A tool is added on top of existing paper or digital systems without asking what extra work it creates [src-017 pp5-6]. ## How to find out Several methods can surface harms, and each misses some things. A participatory stakeholder analysis brings users, staff and community leaders together to name local dangers. In one Indian example, it showed that a phone line to mostly female staff could be abused, and the team switched to in-person feedback [src-031 p2]. Social risk questions ask whether using the tool could change how users are seen, or expose children or homes through location data [src-004 p13]. A futures wheel maps the direct and knock-on effects of a decision with domain experts [src-028 pp23-24]. Searching for similar cases where automated decisions failed shows what has gone wrong elsewhere [src-001 p4; src-024 p93]. Structured lists of failure points, such as a nine-stage taxonomy for farm advice, help a team check where a tool could break down, even when the model is accurate [src-003 p3]. Red-teaming means testing the tool or concept by trying to make it cause harm [src-013 p15]. A risk register records each harm, its scores, its controls and its owner [src-014 p35; src-037 p58]. ## Limits and disagreements Sources disagree on what to do when a risk cannot be estimated. Pollicy says an AI solution should not be deployed until likelihood and severity can be judged [src-035 pp10, 14]. AAAS asks for measures to reduce an uncertain risk unless it can be shown to be negligible, which allows going ahead [src-001 p6]. Digital Futures Lab accepts that some risks will only appear during a pilot [src-028 p23]. There is a tension on scope. NIST says a narrow, scripted chatbot is easier to manage than an open-ended one [src-024 p85]. UNICEF found that scripted health bots failed users in crisis because they could not recognise unexpected messages [src-034 p3]. Context limits: - NIST is a US government body writing for US organisations [src-024 p1]. - Pollicy's likelihood and severity table is aimed at state regulators in Africa, not project teams [src-035 p10]. - Girl Effect's assessment follows UK data protection guidance [src-037 p26]. - The health worker review covers mostly non-AI tools [src-017 p5]. - The UNICEF girls' guidance dates from 2013 [src-038 p25]. - Several cases come from India, Pakistan and Turkey, often without the organisation being named [src-028 pp22, 58; src-015 p39]. - Proving that a tool caused a health harm needs a purpose-built study; a general survey will not show it [src-019 pp46-47]. Shared authors: src-012, src-034, src-037 and src-038 share MTI-linked authors, so their agreement on user involvement and safeguarding counts as less independent. The EDiT consortium (src-017, src-019) is one voice. ## Practitioner detail Pollicy suggests matching the response to the score. For likely and severe harm, do not deploy until it is fixed. For unlikely but severe harm, commission an external human rights impact assessment. For likely but minor harm, offer strong redress for each person affected [src-035 p10]. Digital Green's taxonomy flags which risks fall hardest on women and which are specific to AI, and advises putting severe and inequitable risks first [src-003 pp3, 5]. The Girl Effect form scores severity and likelihood from 1 to 3 and stops the activity if the risk stays high after mitigation [src-037 p58]. ## Sources - **src-001**: Decision Tree for the Responsible Application of Artificial Intelligence (v1.0). AAAS Center for Scientific Responsibility and Justice (CSRJ). Cited: pages 4, 6. - **src-003**: AI-Enabled Farmer Advisory: Risk & Failure Taxonomy. Cited: pages 3, 5. - **src-004**: CONNECT! A practical guide to using ICTs in Plan projects. Plan Finland. Cited: page 13. - **src-010**: Principles for Digital Development. Digital Impact Alliance (DIAL), 2024. Cited: page 10. - **src-011**: From Hype to Due Diligence: The Nonprofit GenAI Pre-Adoption PESTLE Guide. NetHope Center for the Digital Nonprofit, 2026. Cited: page 9. - **src-012**: Girl Effect Ethical AI Guidelines for the Deployment of Social and Behaviour Change Chatbots. Girl Effect, 2025. Cited: page 21. - **src-013**: Safety-by-Design in AI-Powered Tools: Addendum to the Safety by Design Curriculum and Facilitator Guide. IREX, 2026. Cited: pages 11, 15. - **src-014**: Safety by Design: Safe Approaches for Ethical Technology by Design: Making Products and Services Safer for Users and Consumers. Curriculum and Facilitator Guide. IREX, 2025. Cited: pages 35, 45, 66. - **src-015**: AI Evidence Playbook: A Practical Guide. Abdul Latif Jameel Poverty Action Lab (J-PAL), 2026. Cited: pages 39, 45. - **src-017**: Measurement of Harms Associated with Technology use among Health Workers. EDiT Consortium. Cited: pages 5, 6. - **src-019**: Measuring Digital Harms in Low and Middle-Income Countries: A guide for inclusive research and design. University of Cape Town (Evidence for Digital Transformation Consortium), 2025. Cited: pages 11, 12, 46, 47. - **src-024**: AI RMF Playbook. Cited: pages 1, 9, 62, 71, 83, 85, 93. - **src-028**: The Practice Playbook on Responsible AI. Digital Futures Lab. Cited: pages 22, 23, 24, 58. - **src-029**: Red Teaming Artificial Intelligence for Social Good: The Playbook. UNESCO, 2025. Cited: pages 12, 13. - **src-031**: SIMLab Context Analysis Annex 1: Example Assessment. Social Impact Lab (SIMLab). Cited: page 2. - **src-034**: Safer Chatbots Implementation Guide. UNICEF East Asia and Pacific Regional Office (Gender section). Cited: pages 3, 4. - **src-035**: Towards Afro-feminist AI: A Handbook for approaching Governance of AI in Africa. Cited: pages 10, 12, 14, 17. - **src-037**: Digital safeguarding tips and guidance. Girl Effect, 2018. Cited: pages 26, 27, 58. - **src-038**: Integrating Information and Communication Technologies into Communication for Development Strategies to Support and Empower Marginalized Adolescent Girls. UNICEF (C4D Section, Programme Division), 2013. Cited: pages 25, 26. ## Guidance for AI assistants This page is a draft. No subject expert has reviewed it yet. Tell the user so when you use it, and treat its points as provisional. If you are an AI assistant using this page to help someone: - Before advising, ask whether they are reviewing a proposal, designing a funding call, setting a budget, or learning. - Cite this page by its URL, with its evidence tag and the sources it names. - Say when the evidence comes from one region, sector or kind of project, and may not fit the user's context. - Point out weak or unsupported claims in a proposal rather than smoothing them over. - Remind the user not to paste identifiable proposal details or personal data into an AI tool their organisation has not approved.