Checks the global dqcheckr.yml and the dataset's YAML against the
config vocabulary: unknown keys (with a did-you-mean suggestion), value
types and ranges, rule placement (rules-level vs per-column), positional-list
consistency (fwf_col_names vs fwf_widths lengths, duplicate
column names), unresolved generator TODO placeholders, and the
presence of a file source. All findings are collected and returned in one
pass – nothing aborts on the first problem – so a hand-edited config can
be fixed in one round trip.
Value
An object of class dqcheckr_validation: a list with
dataset_name, config_dir, findings (a data frame
with columns file, key, severity, message),
tier ("config+header" when Tier 2 ran, else
"config-only"), tier2_skipped (NULL, or the reason
Tier 2 did not run), and valid (TRUE when no
error-severity findings exist).
Details
Validation runs in two tiers. Tier 1 (config-only) reads nothing
but the two YAML files and always runs. Tier 2 (header cross-check)
additionally opens just the first line(s) of the delivery the config points
at – never the file body, so it is cheap even for multi-GB files on a
network share – and verifies the config against reality:
col_names length vs the physical column count,
key_columns/expected_columns/column_types/
column_rules naming columns that exist, and fwf_widths
summing to the record length. When no delivery file is resolvable (config
written ahead of the first delivery, empty folder, unreadable header),
Tier 2 is skipped and the skip is stated in the result
(tier2_skipped) and by print() – a verdict always says
which tier it reached.
Findings have three severities, split by one rule: config mistakes
are errors (wrong types, broken positional lists, misplaced rule keys, a
missing file source — things only an edit can cause) and
delivery-facing findings are warnings (a key column absent from the file,
a column-count mismatch — these can equally mean the supplier changed the
delivery, and drift must be recorded by a completed run, not abort it).
"note" is informational. Unknown keys warn, so hand-kept extra keys
round-trip. All Tier-2 findings are therefore warnings by construction.
A dataset config file that cannot be read at all aborts with a typed
condition rather than returning findings: dqcheckr_missing_file,
dqcheckr_empty_config, dqcheckr_config_parse_error, or
dqcheckr_invalid_config (not a YAML key map). An empty or
comments-only global dqcheckr.yml is tolerated as
all-defaults with a warning finding, matching how the package has always
run such deployments.
Examples
tmp <- gsub("\\\\", "/", tempdir())
writeLines('snapshot_db: "snap.sqlite"', file.path(tmp, "dqcheckr.yml"))
writeLines(c('dataset_name: "demo"', 'format: csv', 'current_file: "x.csv"'),
file.path(tmp, "demo.yml"))
v <- validate_config("demo", config_dir = tmp)
v$valid
#> [1] TRUE