dqcheckr checks recurring data deliveries: each time a file arrives, one function call verifies it against a per-dataset configuration, records a snapshot, and renders an HTML report. The workflow is deliberately plain:
- configs are files — generated once, then owned by your hand edits;
- running is code — one R call per action, no application to operate;
- outputs are files — self-contained HTML reports you open from disk, plus a SQLite snapshot history.
This vignette walks the whole loop on a small example: generate the configs, read what was generated, edit a rule, validate, run, list the history, and compare two deliveries.
A delivery arrives
Our example delivery has a wrinkle that real files often have: the
header repeats a column name (Amount appears twice).
writeLines(c(
"Date,Amount,Currency,Amount,Status",
"2026-07-01,100.00,AUD,15.00,settled",
"2026-07-02,250.50,AUD,0.00,settled",
"2026-07-03,80.25,NZD,8.10,pending"
), "orders.csv")Bootstrap: two generator calls
A brand-new deployment needs exactly two calls: one for the shared global options, one per dataset. Both are create-only — they will never overwrite a config that already exists.
library(dqcheckr)
generate_global_config(config_dir = "config")
#> [dqcheckr] Global config written: config/dqcheckr.yml
#> Relative snapshot_db/report_output_dir resolve against the working directory -- run from the deployment root.
generate_dataset_config("orders.csv", config_dir = "config")
#> [dqcheckr] Config written: config/orders.yml
#> Review the commented options, then check with validate_config("orders", config_dir = "config").(The detection itself is also available standalone:
sniff_dataset("orders.csv") returns everything the
generator is about to write — format, delimiter, encoding, column names
and types, key-column candidates — without touching disk, if you want to
inspect a delivery before committing to a config.)
Read what was generated
The dataset config is fully-optioned and self-documenting: every key the package understands appears exactly once — detected values live, optional settings commented out with their defaults — each with the same description the validator uses. Note three things as you read it:
- the duplicated
Amountarrived renamed positionally (Amount_2, with a# was "Amount"note), andcsv_skip: 1drops the file’s own header so the renamed list replaces it; - the
col_nameslist sits under a warning: it is positional, so you must never comment out a single entry — every entry below it would shift onto the wrong column; - the sniffer proposes
key_columnscandidates commented out — columns unique in the sample, for you to confirm, never enforced silently.
writeLines(readLines(file.path("config", "orders.yml")))
#> # dqcheckr dataset config -- generated by generate_dataset_config()
#> # from: orders.csv
#> # Detected values are set live; optional settings are shown commented out
#> # with their defaults. Uncomment and edit only what you want to change.
#> # Check your edits any time with: validate_config("orders")
#>
#> ## Identity of the dataset. Written into every config by convention;
#> ## dqcheckr itself takes the name as a function argument and matches it to
#> ## the config filename.
#> dataset_name: "orders"
#>
#> ## Free-text description of the dataset, for humans reading the config. Not
#> ## read by the checks; the GUI wizard has always written it.
#> # description: "describe this dataset"
#>
#> ## Explicit path to the current delivery. Required unless folder is set;
#> ## takes precedence over folder.
#> current_file: "orders.csv"
#>
#> ## Directory holding the deliveries; the two most recently modified files
#> ## become current and previous. Required unless current_file is set.
#> # folder: "path/to/deliveries/"
#>
#> ## Explicit path to the previous delivery for comparison checks. Only
#> ## honoured alongside current_file.
#> # previous_file: "path/to/previous.csv"
#>
#> ## File format of the delivery. Defaults to CSV; "fwf" switches to the
#> ## fixed-width reader and makes fwf_widths required.
#> format: "csv"
#>
#> ## Text encoding of the delivery. ASCII and its aliases are read as UTF-8
#> ## (lossless superset); a declared UTF-8 is validity-scanned before parsing.
#> encoding: "UTF-8"
#>
#> ## Field separator for CSV files.
#> delimiter: ","
#>
#> ## Quote character for CSV files, passed through to the reader.
#> # quote_char: "\""
#>
#> ## POSITIONAL LIST -- always keep one entry per physical column, in file
#> ## order. NEVER comment out a single entry: every entry below it would
#> ## shift onto the wrong column. To exclude a column from checks, leave it
#> ## here and omit it from expected_columns / rules instead.
#> ## Explicit column names replacing the file's own header, in physical column
#> ## order. Omit to use the file's header. POSITIONAL: always list every
#> ## column; pair with csv_skip: 1 when the file has a header row being
#> ## replaced.
#> col_names:
#> - "Date"
#> - "Amount"
#> - "Currency"
#> - "Amount_2" # was "Amount"
#> - "Status"
#>
#> ## Leading lines to drop from a CSV before reading, e.g. the original header
#> ## row when col_names supplies replacement names.
#> csv_skip: 1
#>
#>
#> ## Column widths for fixed-width files, in physical order. Required when
#> ## format is "fwf". POSITIONAL: always list every column; never comment out
#> ## an entry.
#> # fwf_widths: [5, 10]
#>
#> ## Column names for fixed-width files, matching fwf_widths entry-for-entry.
#> ## POSITIONAL.
#> # fwf_col_names:
#> # - "col_1"
#> # - "col_2"
#>
#> ## Leading lines to drop from a fixed-width file before reading.
#> # fwf_skip: 0
#>
#> ## Columns the delivery is expected to contain; missing or unexpected
#> ## columns are flagged. Name-keyed: commenting an entry out safely drops
#> ## that column from the expectation.
#> expected_columns: ["Date", "Amount", "Currency", "Amount_2", "Status"]
#>
#> ## Columns forming the row identity, checked for uniqueness and missingness.
#> # key_columns: ["Date", "Amount", "Amount_2"] # candidates unique in the sample -- confirm before uncommenting
#>
#> ## Per-column type pins (character, numeric, or date) overriding type
#> ## inference.
#> column_types:
#> "Date": date
#> "Amount": numeric
#> "Currency": character
#> "Amount_2": numeric
#> "Status": character
#>
#> ## Per-column quality rules; see the rule vocabulary for the keys valid
#> ## inside each column's map.
#> # column_rules:
#> # some_column:
#> # max_missing_rate: 0.05
#> # min_value: 0
#> # pattern: "^[A-Z0-9]+$"
#>
#> ## Dataset-level overrides merged over the global default_rules; see the
#> ## rule vocabulary.
#> # rule_overrides:
#> # max_missing_rate: 0.05
#> # min_row_count: 1
#>
#> ## Path to a user-supplied R file of custom check functions run after the
#> ## built-in checks.
#> # custom_checks_file: "custom_checks.R"
#>
#> ## SQLite snapshot database recording every run. Dataset value overrides the
#> ## global one; relative paths resolve against the working directory.
#> # snapshot_db: "data/snapshots.sqlite"
#>
#> ## Directory receiving rendered HTML reports. Dataset value overrides the
#> ## global one; relative paths resolve against the working directory.
#> # report_output_dir: "reports/"Edit a rule
Editing a config is editing a text file. Here we do with code what
you would do in an editor: uncomment the rule_overrides
block and require at least one data row.
cfg_path <- file.path("config", "orders.yml")
lines <- readLines(cfg_path)
lines <- sub("^# rule_overrides:", "rule_overrides:", lines)
lines <- sub("^# min_row_count: 1", " min_row_count: 1", lines)
writeLines(lines, cfg_path)Validate after every edit
validate_config() reports all findings
in one pass — vocabulary, types and ranges, positional-list consistency
— and, when the delivery file is reachable, cross-checks the config
against the file’s header (never its body, so this is cheap even for
very large files).
validate_config("orders", config_dir = "config")
#> dqcheckr config validation: orders -- VALID (tier: config+header)
#> No findings.Typos get a suggestion rather than a silent no-op. Watch what happens if we misspell a key:
writeLines(c(lines, 'delimitter: ";"'), cfg_path)
validate_config("orders", config_dir = "config")
#> dqcheckr config validation: orders -- VALID (tier: config+header)
#> [warning] orders.yml: Unknown key 'delimitter' in the dataset config. Did you mean 'delimiter'?
writeLines(lines, cfg_path) # undoFindings follow one rule worth understanding, because it decides what can stop a run:
-
Config mistakes are errors, and errors block. A
wrong value type, a broken positional list, a missing file source —
things only an edit can cause — abort
run_dq_check()before any snapshot is written. You cannot accidentally run an unrunnable config. -
Delivery-facing findings are warnings, and warnings are
recorded. A key column absent from the file, a
col_namescount that no longer matches — these can mean a config typo or the supplier changed the delivery, and drift must be recorded, not crash the run. The run proceeds, and each warning is persisted into the run’s results as aVC-01check — visible in the report, the WARN counts, and the history — as well as printed to the console. Unknown keys (your own annotations round-trip safely) are warnings too.
Run the check
result <- run_dq_check("orders", config_dir = "config", open_report = FALSE)
#> [dqcheckr] orders: PASS - 0 warning(s), 0 failure(s). Report: /tmp/RtmplB8uWb/dq-demo/reports/orders_20260726_032053_1.html
result$status
#> [1] "PASS"Each run appends a snapshot to the SQLite database and renders a
self-contained HTML report into reports/ (when the Quarto
CLI is available; open_report = TRUE, the default, opens it
for you in interactive use). The report is a plain file — open it from
the file manager, mail it, archive it.
The run history
The history is one call away, newest first. The id
column is what the drift comparison takes.
runs <- list_runs("orders", config_dir = "config")
runs[, c("id", "run_timestamp", "overall_status",
"check_pass_count", "check_warn_count", "check_fail_count")]
#> id run_timestamp overall_status check_pass_count check_warn_count
#> 1 1 2026-07-26T03:20:53Z PASS 19 0
#> check_fail_count
#> 1 0list_runs() is per-dataset and resolves the right
database from the configs (honouring a per-dataset
snapshot_db override). For the whole deployment at once —
every dataset sharing a database — drop to the primitive and name the
file yourself:
list_snapshots(db_path = file.path("data", "snapshots.sqlite"))[
, c("id", "dataset_name", "run_timestamp", "overall_status")]
#> id dataset_name run_timestamp overall_status
#> 1 1 orders 2026-07-26T03:20:53Z PASSA second delivery, and drift
The next delivery arrives — with a suspiciously changed
Amount profile and a missing status. Run the check again,
then compare the two snapshots.
writeLines(c(
"Date,Amount,Currency,Amount,Status",
"2026-08-01,900.00,AUD,15.00,settled",
"2026-08-02,1250.50,AUD,0.00,",
"2026-08-03,880.25,NZD,8.10,pending"
), "orders.csv")
result2 <- run_dq_check("orders", config_dir = "config", open_report = FALSE)
#> [dqcheckr] orders: FAIL - 0 warning(s), 1 failure(s). Report: /tmp/RtmplB8uWb/dq-demo/reports/orders_20260726_032058_2.htmlcompare_snapshots() defaults to the two most recent
runs; pass two ids from list_runs() to compare
any historical pair. With report = TRUE (the default) it
renders a drift report HTML like the run report; here we just
compute.
cmp <- compare_snapshots("orders", config_dir = "config",
report = FALSE, open_report = FALSE)
#> [dqcheckr] drift: orders snapshot #1 vs #2
names(cmp)
#> [1] "dataset_name" "snap_prev" "snap_curr"
#> [4] "table_drift" "schema_changes" "missing_rate_changes"
#> [7] "non_numeric_changes" "mean_shifts" "spread_changes"
#> [10] "distinct_changes" "report_path"Judging spread changes
The drift result also carries each numeric column’s
spread (min, max, standard deviation). These are always
reported — but never judged until you set a threshold,
so a config that predates the rule gains information without gaining new
failure modes. The Amount column’s spread more than doubled
with the second delivery, yet exceeds stays
FALSE:
cmp$spread_changes[, c("Column", "numeric_sd_prev", "numeric_sd_curr",
"numeric_sd_shift_pct", "numeric_sd_exceeds")]
#> Column numeric_sd_prev numeric_sd_curr numeric_sd_shift_pct
#> 1 Amount 93.117645 208.296813 1.236921
#> 2 Amount_2 7.507996 7.507996 0.000000
#> numeric_sd_exceeds
#> 1 FALSE
#> 2 FALSEActivating the judgement is the usual edit — uncomment the rule in
the global config (one default for every dataset; a per-column value in
column_rules would win for its column):
gcfg_path <- file.path("config", "dqcheckr.yml")
glines <- readLines(gcfg_path)
glines <- sub("^# default_rules:", "default_rules:", glines)
glines <- sub("^# max_numeric_sd_shift_pct:.*",
" max_numeric_sd_shift_pct: 0.2", glines)
writeLines(glines, gcfg_path)
cmp2 <- compare_snapshots("orders", config_dir = "config",
report = FALSE, open_report = FALSE)
#> [dqcheckr] drift: orders snapshot #1 vs #2
cmp2$spread_changes[, c("Column", "numeric_sd_shift_pct", "numeric_sd_exceeds")]
#> Column numeric_sd_shift_pct numeric_sd_exceeds
#> 1 Amount 1.236921 TRUE
#> 2 Amount_2 0.000000 FALSESpecial cases worth knowing
Fixed-width files. The sniffer guesses column
boundaries from blank gutters and always emits a character-position
ruler comment above fwf_widths so you can
correct a mis-split by counting. A packed file — columns
touching, no gutters — gets explicit TODO widths that
validate_config() flags as an error, so a run refuses to
start until you fill them in against the ruler:
writeLines(c("AB12XY0099QQWW2026",
"CD34ZW0100RRTT2026"), "ledger.txt")
generate_dataset_config("ledger.txt", config_dir = "config")
#> [dqcheckr] Config written: config/ledger.yml
#> Review the commented options, then check with validate_config("ledger", config_dir = "config").
validate_config("ledger", config_dir = "config")
#> dqcheckr config validation: ledger -- INVALID (tier: config+header)
#> [error] ledger.yml: 'fwf_widths' still contains the generator's TODO placeholder. Fill in the column widths (see the ruler comment in the config) before running.The never-overwrite rule. Generators create; humans edit. Re-running a generator against an existing config is refused, and the file is untouched — your hand-tuning can never be silently clobbered:
generate_dataset_config("orders.csv", config_dir = "config")
#> Error in `generate_dataset_config()`:
#> ! Config already exists and will not be overwritten: config/orders.yml
#> Generated configs are created once and then owned by hand edits (the never-overwrite rule). To re-sniff, pass a different dataset_name or config_dir and diff the result.To re-sniff a structurally changed delivery, generate under another name (or into a scratch directory) and diff.
Where everything lives
deployment-root/
├── config/ # one YAML per dataset + dqcheckr.yml -- the dataset list
├── data/
│ └── snapshots.sqlite
├── reports/ # one self-contained HTML per run
└── orders.csv # deliveries (or point configs at their real location)The config/ folder is the dataset list:
add a dataset by generating a config, remove one by deleting its file,
inspect one by opening it. The whole workflow is these calls:
| Action | Call |
|---|---|
| New deployment | generate_global_config("config") |
| New dataset | generate_dataset_config("file.csv", config_dir = "config") |
| Check an edit | validate_config("name", config_dir = "config") |
| Run | run_dq_check("name", config_dir = "config") |
| History | list_runs("name", config_dir = "config") |
| Drift | compare_snapshots("name", config_dir = "config") |