IngestConfig
Entry pointLanguage-independent config for reading a data file and normalizing it to Parquet. A delimited file or a spreadsheet carries no usable type information and Parquet does, so this is the one step that decides types -- everything after it reads a declaration rather than inferring one. `format` is what the file is, not a preference, so it has no default and a config that disagrees with its file is an error. `format` selects the variant and its settings are siblings of it, validated against the variant's own schema below. Anything not set takes rtemis's default, so an object holding only `format` is that variant with every default.
Variants
The value of format selects which schema applies.
delimitedDelimitedIngestConfigparquetParquetIngestConfigxlsxXLSXIngestConfigrdsRDSIngestConfigdtaDTAIngestConfigarffARFFIngestConfig
Properties
formatrequiredstringWhat the file is.
one of
"delimited""parquet""xlsx""rds""dta""arff"columnsobject | nullDeclared column types, keyed by column name, in the `profile/v1` vocabulary. A column named here is converted after reading rather than inferred; one absent is left as the reader gave it.
character2factorbooleanRead character columns as factors. Supervised learning needs categorical predictors as factors, and most file formats cannot say which columns those are.
clean_colnamesbooleanNormalize column names on read.
remove_duplicatesbooleanDrop rows that repeat an earlier row exactly.