csv_source
csv_source
¶
A CSV file read once, so its typed and text forms come from the same bytes.
Classes:
| Name | Description |
|---|---|
CsvSource |
A CSV location and its raw bytes, read once. |
Attributes:
| Name | Type | Description |
|---|---|---|
TEXT_READ_EXCLUDED_LOAD_ARGS |
|
TEXT_READ_EXCLUDED_LOAD_ARGS = frozenset({'converters', 'dtype', 'keep_default_na', 'na_filter', 'na_values', 'parse_dates'})
module-attribute
¶
pd.read_csv options dropped from the text read, because they would override its every-cell-as-text options.
CsvSource(location, content, load_args=dict())
dataclass
¶
A CSV location and its raw bytes, read once.
read parses the file with type inference, as pd.read_csv does.
read_text parses the same bytes again with every cell kept as the text
in the file, so a ZIP code "02054" stays "02054" and "N/A"
stays "N/A". Keeping the bytes rather than a second dataframe means
the text form costs no memory until it is needed, and the file can change
on disk in between without the two forms disagreeing.
Local paths, file:// URLs, and http(s) URLs are read once with
load. Other locations, such as s3:// URLs, keep no bytes:
read and read_text each let pandas open the location, so the
text form is only as current as the second read.
Example
source = CsvSource.load("patients.csv.gz", {"sep": ";"})
typed = source.read()
text = source.read_text() # Parsed from the same bytes as typed.
Methods:
| Name | Description |
|---|---|
load |
Read the raw bytes of |
read |
Parse the file with type inference and the configured |
read_text |
Parse the file with every cell as the string written in it, including blanks and |
Attributes:
| Name | Type | Description |
|---|---|---|
location |
str
|
Path or URL the file was read from. |
content |
bytes | None
|
Raw file bytes, or |
load_args |
Mapping[str, Any]
|
Extra |
location
instance-attribute
¶
Path or URL the file was read from.
content = field(repr=False)
class-attribute
instance-attribute
¶
Raw file bytes, or None when pandas reads location directly.
load_args = field(default_factory=dict)
class-attribute
instance-attribute
¶
Extra pd.read_csv options for the typed read.
load(location, load_args=None)
classmethod
¶
Read the raw bytes of location once.
HTTP(S) URLs are fetched with httpx, following redirects, and a
storage_options mapping is sent as request headers, as pandas does.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
location
|
str
|
Local path, |
required |
load_args
|
Mapping[str, Any] | None
|
Extra |
None
|
Raises:
| Type | Description |
|---|---|
FileNotFoundError
|
If a local |
DataError
|
If an HTTP(S) request fails or returns an error status. The message names the
URL without its credentials or query string (see |
Source code in src/nemo_safe_synthesizer/data_processing/csv_source.py
read()
¶
read_text()
¶
Parse the file with every cell as the string written in it, including blanks and "N/A".
Options in TEXT_READ_EXCLUDED_LOAD_ARGS are dropped; separators,
encodings, and other parsing options still apply.
Source code in src/nemo_safe_synthesizer/data_processing/csv_source.py
redact_url(url)
¶
Return url without credentials, query string, or fragment, which can hold an access token.
Example
"https://host/people.parquet?token=abc" returns "https://host/people.parquet".