Harm scenarios and risk assessment

Basics

Before an AI tool is built, the team should write down how it could hurt the people who use it and the people around them. For each harm, they should judge how likely it is and how bad it would be, then decide what they will do to prevent it and how they will respond if it happens anyway src-014 p35; src-013 p15; src-037 p26. Harms can be direct or indirect, and intended or accidental src-019 p11. They include false or unsafe advice, privacy breaches, fraud, violence enabled by technology, and effects on physical and mental health src-019 pp11-12; src-013 p11.

Why it matters for AI tools

AI makes some old harms larger and faster. It can produce abusive content and deepfakes at volume, and its workings are hard to inspect, which makes safety checks difficult src-014 p66; src-029 p13. Some harm comes from people misusing a tool on purpose; some comes from a tool behaving badly when used as intended src-029 pp12-13. Even a very accurate system can cause harm, and a widely used tool will be put to uses its builders never planned src-024 p62.

Unplanned use is documented. In Turkey, students given a basic chatbot scored worse in exams than students with no chatbot, while a version with learning safeguards avoided this src-015 p39. After one organisation scaled its AI tool, frontline staff began treating it as the decision-maker and gave less personal support src-028 p58.

People in crisis use chatbots for help the bot was never built to give. Many quickly built health chatbots could not recognise a message about violence or self-harm and offered no route to a human src-034 p3. Harm can also arise in the wider system. A woman who cited a legal chatbot at a police station was turned away, because the project had engaged women users but not the police src-028 p22.

DIAL notes that harms from AI are less well understood than harms from other technology src-010 p10. A review of harms to health workers in low- and middle-income countries found only two studies that looked at AI at all src-017 p5. Little evidence of harm does not mean little harm.

Bias and unfair treatment of groups is covered on its own page in this bucket.

What good evidence looks like

• A list of harms for each user group, including bystanders and people affected by the tool but not using it src-014 p35; src-024 p83.
• Both accidental harms and deliberate misuse are considered, with examples specific to the tool src-029 pp12-13; src-014 p35.
• Each harm is scored for likelihood and severity, mitigations are named, and the leftover risk is scored again src-014 p35; src-037 p58; src-024 p9.
• Low-scored risks still have a planned response; scoring sets the order of work, not what to ignore src-014 p45.
• Users, especially vulnerable ones, helped name the risks, and their concerns carry weight in setting priorities src-012 p21; src-038 p26; src-013 p15.
• A named person owns each mitigation, with a deadline and sign-off src-037 pp26-27.
• A clear point at which the team will halt or not launch src-037 p58; src-024 p71.
• A person checks outputs before they affect anyone, and that person can overrule the tool src-011 p9; src-013 p15.
• A plan for people who disclose danger to the tool, with a route to human help src-034 pp3-4.
• A date and trigger for repeating the assessment, such as new features or a change in context src-037 p27; src-003 p5.

How to find out

Several methods can surface harms, and each misses some things.

A participatory stakeholder analysis brings users, staff and community leaders together to name local dangers. In one Indian example, it showed that a phone line to mostly female staff could be abused, and the team switched to in-person feedback src-031 p2. Social risk questions ask whether using the tool could change how users are seen, or expose children or homes through location data src-004 p13.

A futures wheel maps the direct and knock-on effects of a decision with domain experts src-028 pp23-24. Searching for similar cases where automated decisions failed shows what has gone wrong elsewhere src-001 p4; src-024 p93. Structured lists of failure points, such as a nine-stage taxonomy for farm advice, help a team check where a tool could break down, even when the model is accurate src-003 p3.

Red-teaming means testing the tool or concept by trying to make it cause harm src-013 p15. A risk register records each harm, its scores, its controls and its owner src-014 p35; src-037 p58.

Limits and disagreements

Sources disagree on what to do when a risk cannot be estimated. Pollicy says an AI solution should not be deployed until likelihood and severity can be judged src-035 pp10, 14. AAAS asks for measures to reduce an uncertain risk unless it can be shown to be negligible, which allows going ahead src-001 p6. Digital Futures Lab accepts that some risks will only appear during a pilot src-028 p23.

There is a tension on scope. NIST says a narrow, scripted chatbot is easier to manage than an open-ended one src-024 p85. UNICEF found that scripted health bots failed users in crisis because they could not recognise unexpected messages src-034 p3.

Context limits:

• NIST is a US government body writing for US organisations src-024 p1.
• Pollicy's likelihood and severity table is aimed at state regulators in Africa, not project teams src-035 p10.
• Girl Effect's assessment follows UK data protection guidance src-037 p26.
• The health worker review covers mostly non-AI tools src-017 p5.
• The UNICEF girls' guidance dates from 2013 src-038 p25.
• Several cases come from India, Pakistan and Turkey, often without the organisation being named src-028 pp22, 58; src-015 p39.
• Proving that a tool caused a health harm needs a purpose-built study; a general survey will not show it src-019 pp46-47.

Shared authors: src-012, src-034, src-037 and src-038 share MTI-linked authors, so their agreement on user involvement and safeguarding counts as less independent. The EDiT consortium (src-017, src-019) is one voice.

Practitioner detail

Pollicy suggests matching the response to the score. For likely and severe harm, do not deploy until it is fixed. For unlikely but severe harm, commission an external human rights impact assessment. For likely but minor harm, offer strong redress for each person affected src-035 p10. Digital Green's taxonomy flags which risks fall hardest on women and which are specific to AI, and advises putting severe and inequitable risks first src-003 pp3, 5. The Girl Effect form scores severity and likelihood from 1 to 3 and stops the activity if the risk stays high after mitigation src-037 p58.