Give source, interim and clean data distinct roles
Raw data is the earliest reliably preserved state obtained from an instrument, participant, source or external provider. It is the reference point even when it contains inconsistent labels, technical columns or suspected mistakes. Interim data supports import, conversion, exploratory checks and staged processing. Clean data is a reviewed release in which stated correction, recoding and exclusion rules have been applied.
These labels describe function within your project rather than universal quality. Adding “clean” to a filename cannot validate it. The state becomes auditable only when its source, processing rule, execution and outcome are recorded. A platform export may already contain hidden vendor processing; in that case, call it the earliest retained export and document the device, interface, export time and known preprocessing. Do not claim access to an unprocessed original you never received.
Capture the source before exploring it
Move incoming files into a controlled intake area and record their origin, date, format, size and checksum where appropriate. Open tabular sources read-only or work on a separate copy. Spreadsheet applications may reinterpret dates, decimal separators, missing values or character encodings at import. For manually entered records, retain the authorised source forms or scans separately from the entered table so that transcription questions can be checked.
Confidential and personal data require approved storage, access control and transfer. An unchanged source is not automatically suitable for indefinite retention or publication. Consent, contracts, data-protection requirements and institutional schedules define what may be kept. Separate re-identification keys from content and assign them an appropriate security classification. This workflow cannot replace formal ethics or data-protection review.
Write rules before inspecting the desired outcome
Define permitted ranges, missing-value codes, duplicate handling, format conversions and exclusion criteria as early as possible. Some rules will arise during quality control; mark their date, trigger and rationale. Do not replace an unusual measurement solely because it is inconvenient. Consult the collection record and plausibility checks, then document whether the value was corrected, flagged, excluded or retained.
Use scripts or repeatable queries for operations that can be automated. Where manual correction is unavoidable, record the variable or object, old value, new value, reason and reviewer. Never put participant names or sensitive content into a broadly accessible log. Use controlled record identifiers and keep the history under the same safeguards as the underlying data.
Freeze one clean input for each reported analysis
Before analysis, validate structure, row counts, variable types, allowed ranges, missingness and the effect of every cleaning rule. Assign a version and checksum. Analysis scripts should read that release and write outputs elsewhere. If an error later requires correction, issue a new clean version and preserve the old one; silent replacement makes past results impossible to diagnose.
A stable input also supports sensitivity analysis. You can compare whether a conclusion depends on an exclusion or imputation rule and report material differences. Neat folders do not guarantee good methods, but they stop copy confusion from disguising methodological changes. The report should identify the data version used for each final table or model.
Account for external, qualitative and image data
For third-party data, the untouched download is your local source state. Retain the landing-page citation, release, licence, access date and checksum. The provider may already have curated it, so do not describe it as original observation. In qualitative work, recording, first transcript, corrected transcript and de-identified analysis copy are separate objects. Define what corrections and redactions mean and record who performed them.
Image and sensor workflows may preserve a lossless master beside smaller working derivatives. Log conversion, calibration, cropping and any loss of embedded metadata. A format change is a transformation even when the picture looks similar. Decide whether timestamps, colour profiles, coordinates and device fields are evidence, sensitive information or disposable technical detail before altering them.
Six questions for a clean handover
- Is each source capture unchanged, uniquely identified and appropriately protected?
- Can any script or application accidentally write to the raw-data location?
- Does every cleaning action have a rule, rationale, execution record and effect check?
- Is exactly one clean release named as input to the reported analysis?
- Do record counts and checksums agree across the inventory and logs?
- Can another researcher rebuild the clean state from source and documented code?