Standards › Standard 2D › Standard operating procedures (SOPs)
Standard operating procedure (SOP) · AI-SOP-001 · Version 1.1 · Approved by the ASF Expert Committee
Supports: 2D.5
At a glance
- Audit a sample of AI-assisted decisions each quarter for over-diagnosis and missed diagnosis.
- Track every complaint or concern about AI-assisted care.
- Act on any finding: retrain, restrict use, or stop.
- Record each audit on AI-REG-001.
- Stop and get help straight away if: an audit shows harm or a pattern of error, and use continues without review.
1. Purpose
To monitor AI systems after introduction for over-diagnosis, missed diagnosis, complaints and other unintended effects, and to act on findings [E-011, EC].
2. Scope
| Applies to | Clinical leads, quality lead, IT lead |
| Locations | All clinical areas using AI systems at the facility |
| When | Quarterly audit; continuous complaint tracking |
| Does not apply to | General clinical audit, covered by SOP-N26 |
3. Definitions
| Term | Meaning |
|---|---|
| Over-diagnosis | A condition recorded or treated that would not otherwise have needed it |
| Missed diagnosis | A condition the AI system or its use failed to detect |
| Audit sample | A set of AI-assisted decisions selected for checking against the clinical outcome |
4. Background
An AI tool can perform well in testing and poorly in use. Criterion 2D.5 requires ongoing checks for over-diagnosis, missed diagnosis and other unintended effects, not an assumption that the tool works as intended.
5. Responsibilities
| Role | Responsibility | Competence |
|---|---|---|
| Clinical lead for each system | Runs the quarterly audit | Clinical |
| Quality lead | Reviews findings and trends | Quality (SOP-N26) |
| IT lead | Checks performance data and system changes | Health informatics (EC) |
6. Materials and equipment
| Item | Requirement | Condition |
|---|---|---|
| Audit sample list (random, 10 per quarter, EC) | Current | Generated each quarter |
| AI-REG-001 Monitoring Register | Current | Quality file |
7. Safety
- Patients affected by a suspected error are reviewed clinically at once, not left until the next audit.
8. Before you start
- Confirm the system version and the audit period.
9. Procedure
Step 1. Select a sample.
- Select a random sample of AI-assisted decisions each quarter (EC size).
- Include decisions where the clinician agreed and where they overrode the AI output.
Key point: Cases where clinicians overrode the AI are a useful source of error.
Step 2. Check against outcome.
- For each case, compare the AI output with the clinical outcome or expert review.
- Record over-diagnosis, missed diagnosis, and other effects.
Key point: The outcome, not the agreement of the clinician, is the check.
Step 3. Track complaints.
- Record every complaint or concern about AI-assisted care on INC-POL-002, flagged as AI-related.
- Review each one with the clinical lead.
Key point: Patients often notice errors first.
Step 4. Act on findings.
- For each finding, decide: no action, retrain, add a check, restrict use, or stop use.
- Record the decision and its owner on AI-REG-001.
Key point: A finding with no action is a finding that repeats.
Step 5. Report.
- Report results each quarter to the quality committee, and to the governing body yearly.
Key point: Reporting keeps the monitoring visible.
10. Decision points
- If the audit shows a pattern of missed diagnosis, use is restricted at once while the cause is found.
11. Special situations
| Situation | Action |
|---|---|
| Vendor update | Re-run the audit sample after the update before full use. |
12. Escalation
- Patient harm linked to an AI system is reported as an incident at once (INC-POL-001).
13. Monitoring
| Measure | Target | Frequency | Owner | If not met |
|---|---|---|---|---|
| Quarterly audit completed | 100% | Quarterly | Clinical lead | Escalate to quality lead |
| AI complaints tracked and reviewed | 100% | Quarterly | Quality lead | Escalate |
| Findings with decision recorded | 100% | Quarterly | Quality lead | Escalate to chief executive |
14. Records
| Record | Kept in | Retention |
|---|---|---|
| AI-REG-001 AI Monitoring Register | Quality file | the period set by the facility’s records policy years |
15. Training
- Clinical auditors are trained in the method at appointment (EC).
16. Related documents
| ID | Title | Use |
|---|---|---|
| AI-REG-001 | AI Monitoring Register | Used in steps 2 to 4 |
| SOP-N26 | Internal Audit, Quality Oversight and Learning Policy (generic) | General audit |
| AI-POL-001 | Responsible Use of AI in Care Policy | Parent policy |
17. Evidence
| Reference | Note |
|---|---|
| E-011 | WHO (2021), Ethics and governance of artificial intelligence for health. Used as the basis for AI governance. Link and wording to verify before publication. |
| E-009 | ASF Expert Committee consensus. Sample size and thresholds to be agreed. |
18. Change history
| Version | Date | What changed | Approval |
|---|---|---|---|
| 1.0 | the date of approval | First version | Approved by the ASF Expert Committee |
| 1.1 | the date of approval | Internal quality review: corrected cross-references, status wording and document control | Approved by the ASF Expert Committee |
E-learning for this document
References
- World Health Organization (2021). Ethics and governance of artificial intelligence for health: WHO guidance. Geneva: WHO. Open source (E-011)
- Accréditation Sans Frontières (2026). ASF Expert Committee consensus on Standard 1. (E-009)