Risk, safety and safeguarding

Bias, here, means an AI tool treats some groups worse than others in a systematic way, often with no one intending it. Stage 0 work should find out which groups the tool may serve badly, such as women, poorer, older, rural, disabled or minority-language users, and why. It should also say how the team will check performance for each of those groups, not only on average. A tool can be popular and cheap to run and still leave some groups further behind.

Before an AI tool is built, the team should write down how it could hurt the people who use it and the people around them. For each harm, they should judge how likely it is and how bad it would be, then decide what they will do to prevent it and how they will respond if it happens anyway. Harms can be direct or indirect, and intended or accidental. They include false or unsafe advice, privacy breaches, fraud, violence enabled by technology, and effects on physical and mental health.