Selection is part of the method, not invisible preparation
Inclusions and exclusions determine what evidence later becomes visible. Terms such as relevant, high quality or suitable cannot be repeated without an operational definition. A log translates them into rules that can be applied to an item.
Criteria follow from the research question, design, resources and protection requirements. They must not be designed to produce a favoured finding or remove inconvenient cases after inspection. Piloting may expose ambiguity, provided changes remain transparent.
The process works for literature screening, document corpora, samples, interview material and datasets. It imposes no method or minimum quantity.
Write rules that two people can apply
Each rule specifies the feature, threshold, evidence source and outcome when met. “Recent” becomes a justified date window; “scholarly” becomes named document types or review features. Avoid wording that already assumes the preferred conclusion.
Keep inclusion and exclusion distinct. An item can meet an inclusion feature yet fail an independent protection or quality rule. Define priority so the same item does not receive a different outcome merely because criteria were checked in a different order.
Pilot rules on a small, varied set. Pilot items do not automatically belong in the final sample. Record whether they were reassessed under the locked rule set.
Do not bury boundary cases in residual labels
A boundary case partly meets a rule, contains conflicting metadata or exposes an unanticipated gap. Give it an ID and reason rather than “other exclusion”. Cite the passage or metadata on which the decision rests.
With multiple reviewers, the more senior person should not decide silently. Define a resolution route: independent first decisions, comparison, reasoned consensus or third review. Different initial views can remain in the log.
Unclear is a valid temporary state. An unavailable file differs from content-based exclusion. Record retrieval attempt, deadline and eventual disposition separately.
Version changes to the criteria
When a rule changes, retain old and new wording, date, reason and decision maker. Identify previously screened items affected and reassess them consistently. Applying a change only to later items introduces systematic bias.
Distinguish clarification from substantive expansion. Defining “online date” as the date of the version studied may clarify a rule; adding another country expands the search space. Both need documentation, but their consequences differ.
Explicitly label changes made after results were seen. That does not automatically invalidate the project, but it limits claims that selection was fully specified in advance.
Turn case decisions into transparent reporting
Report the starting set or search yield, duplicates, exclusions by distinct reason, boundary cases and final set. Totals should reconcile. If several reasons can apply to one item, use a primary reason or explain multiple counting.
Do not reveal protected identities or confidential full content. Aggregated totals, criterion definitions and anonymised cases can provide traceability while detailed access remains controlled.
The methods parameter check integrates the log into reporting. The coding guide structures later analysis. Browse the research methods hub; the single foundation route is how verifiable checking works.
Run a decision-audit sample
Select included, excluded and unresolved items from different screening stages. A second person applies only the recorded rule and compares outcome and reason. Disagreement indicates ambiguity in wording, evidence or reviewer preparation.
An agreement statistic may help some designs, but it is not universally required and does not replace examination of cases. Report only measures whose calculation and purpose fit the project.
Before analysis, lock the criteria version, export the log and link both to the material version. Later corrections receive a new version instead of silent overwriting. Check that identifiers survive export and that every final item has one inspectable decision trail.
Inspect the order of dependent rules. If language is checked before document type, early exclusion may leave later features unknown. Record “not assessed” rather than “not met”. Assign each item one primary final outcome for flow totals while retaining secondary observations. This permits arithmetic reconciliation without erasing complexity.
For machine-assisted prescreening, report query, fields, version, threshold and human review. An automated filter does not assume methodological responsibility. Where the design includes a sample of automatically rejected material, record its selection, findings and any resulting rerun rather than quietly changing the system.