Cherry-picking can begin before a source is cited
A biased result can be built into the query. Searching only for the expected effect makes criticism and null findings harder to retrieve. Distortion can also occur during screening, when a promising title receives special treatment, or during writing, when only the convenient outcome from a multi-outcome study is mentioned. Date limits, language restrictions, database choice and a preference for easily available full text may all alter the evidence base.
A bounded search is not automatically cherry-picking. A dissertation needs a manageable scope and can reasonably focus on a population, method or period. The question is whether limits match the research question, are stated before individual results are known and are applied consistently. Later changes may be defensible, provided that their date, reason and impact are recorded.
Specify the question and eligibility rules first
Break the question into topic, population or material, context, and any relevant method or outcome concepts. List synonyms, discipline-specific vocabulary and spelling variants. Decide which designs, years, languages and publication types can be assessed. Explain each restriction. A language limit imposed by available competence may be honest, but it should be named as a possible source of missing evidence.
Eligibility must not depend on whether a result agrees with the thesis. “Studies students in formal education” is a usable criterion; “finds a benefit” is not. Plan how to handle duplicate records, multiple reports from one study, retractions, protocols and unavailable full texts. Decide which bibliographic and decision fields will be retained so that the selection can be checked later.
Screen against rules rather than the headline
Review titles and abstracts with a short eligibility checklist. Preserve a decision for every screened record, not only the included ones. At full-text stage, assign a specific exclusion reason. “Not relevant” is difficult to audit; “wrong population”, “no empirical outcome”, “outside date scope” or “duplicate report with no additional data” is more informative.
If possible, ask another person to screen a sample, or repeat a sample yourself after a pause without looking at the first decision. Disagreement reveals vague criteria. A lone researcher can flag borderline records and resolve them with a supervisor. Record the reasoning rather than quietly moving a difficult paper into the convenient category.
Test whether the search can retrieve challenge
Add terms for replication, criticism, null findings, adverse effects, limitations and alternative explanations where these are meaningful in the field. Follow references backwards and citations forwards from central sources. Include protocols, registered reports, theses or institutional repositories when grey literature is relevant and falls within the defined scope.
The purpose is not to find a token opponent. It is to check that the design is capable of finding disagreement. If no robust counter-evidence emerges, report the route and residual limits. “Not located” is not the same as “does not exist”; databases differ in coverage, terminology and update frequency.
Separate eligibility from critical appraisal
First ask whether a record addresses the question; then assess how much weight its method supports. Use discipline-specific appraisal and apply it in the same way regardless of direction. A weak supportive study should not receive a softer review than a weak contradictory one. Capture design, population, measurement, uncertainty and conflicts without inventing a universal score.
Several papers may analyse the same sample. Do not count them as independent replications. Conversely, one article may contain multiple genuinely separate studies. The unit of evidence follows the underlying data and method, not the number of entries in the bibliography.
Report the boundaries of the search
State the databases, date span, search dates, principal concepts, eligibility rules and scale of screening. A narrative dissertation review may use a concise description; a systematic review must follow the relevant reporting and methods standards. Identify amendments and sources likely to be missed. This is more credible than implying exhaustive coverage from a limited exercise.
Connect the selected records to an evidence matrix before drafting the conclusion. Place conflicting and uncertain findings beside the claims they qualify. Where the literature is heterogeneous, the conclusion may remain conditional. The goal is a defensible answer, not a perfectly smooth story.
Six signs that the chapter needs another audit
- Every source supports the thesis despite visible controversy in the field.
- Queries and databases cannot be reconstructed.
- Exclusion reasons shift with the direction of the result.
- A convenient secondary outcome replaces the study’s wider findings.
- Several reports from one dataset are presented as independent evidence.
- The prose claims completeness that the search did not achieve.