Risk–Benefit Assessment Tool for Using AI in Evidence Syntheses

Definitions

Managing Team/Workflow

Organising project tasks, timelines, and team coordination.

Searching / Scoping

Developing search strategies to find studies.

Screening

Selecting studies based on relevance.

Data Extraction

Collecting relevant information from included studies.

Data Synthesis

Combining and summarising the extracted data.

Critical Appraisal

Assessing the methodological quality of studies.

Reporting

Documenting methodology and results.

Visualisation of Results

Creating graphs or visuals.

Exploration

AI used for ideation, drafting, or brainstorming.

Expedition

AI accelerates or supports human-led processes.

Endorsement

AI helps phrase or generate conclusions (highest scrutiny).

Severity of Risk

Impact if AI introduces error.

Likelihood of Risk

Probability that AI introduces that error.

Net Score = Benefit − (Severity × Likelihood ÷ 5)

Stage	Potential Risks	Potential Benefits
Managing Team/Workflow	Oversimplifies complex project dependencies or misunderstands team capacity and expertise.	Automates scheduling, creates structured protocols, and generates task templates.
Searching / Scoping	Misses important terminology or grey literature sources.	Suggests additional keywords or new sources.
Screening	Misapplies inclusion criteria or omits relevant studies.	Prioritises records likely to be relevant for faster review.
Data Extraction	Extracts incorrect or inconsistent information.	Identifies and retrieves structured data automatically.
Data Synthesis	Aggregates findings inappropriately or overgeneralises differences.	Suggests grouping strategies or summary statements.
Critical Appraisal	Misses methodological limitations or applies bias inconsistently.	Flags missing information or supports checklist-based evaluation.
Reporting	Generates vague or biased language.	Helps structure summaries or standardised sections.
Visualisation of Results	Creates misleading visuals or uses unsuitable formats.	Generates tables, maps, or plots from structured data.

Evidence synthesis stage: Type of AI use: