Skip to content

csv_source

csv_source

A CSV file read once, so its typed and text forms come from the same bytes.

Classes:

Name Description
CsvSource

A CSV location and its raw bytes, read once.

Attributes:

Name Type Description
TEXT_READ_EXCLUDED_LOAD_ARGS

pd.read_csv options dropped from the text read, because they would override its every-cell-as-text options.

TEXT_READ_EXCLUDED_LOAD_ARGS = frozenset({'converters', 'dtype', 'keep_default_na', 'na_filter', 'na_values', 'parse_dates'}) module-attribute

pd.read_csv options dropped from the text read, because they would override its every-cell-as-text options.

CsvSource(location, content, load_args=dict()) dataclass

A CSV location and its raw bytes, read once.

read parses the file with type inference, as pd.read_csv does. read_text parses the same bytes again with every cell kept as the text in the file, so a ZIP code "02054" stays "02054" and "N/A" stays "N/A". Keeping the bytes rather than a second dataframe means the text form costs no memory until it is needed, and the file can change on disk in between without the two forms disagreeing.

Local paths, file:// URLs, and http(s) URLs are read once with load. Other locations, such as s3:// URLs, keep no bytes: read and read_text each let pandas open the location, so the text form is only as current as the second read.

Example

source = CsvSource.load("patients.csv.gz", {"sep": ";"}) typed = source.read() text = source.read_text() # Parsed from the same bytes as typed.

Methods:

Name Description
load

Read the raw bytes of location once.

read

Parse the file with type inference and the configured load_args.

read_text

Parse the file with every cell as the string written in it, including blanks and "N/A".

Attributes:

Name Type Description
location str

Path or URL the file was read from.

content bytes | None

Raw file bytes, or None when pandas reads location directly.

load_args Mapping[str, Any]

Extra pd.read_csv options for the typed read.

location instance-attribute

Path or URL the file was read from.

content = field(repr=False) class-attribute instance-attribute

Raw file bytes, or None when pandas reads location directly.

load_args = field(default_factory=dict) class-attribute instance-attribute

Extra pd.read_csv options for the typed read.

load(location, load_args=None) classmethod

Read the raw bytes of location once.

HTTP(S) URLs are fetched with httpx, following redirects, and a storage_options mapping is sent as request headers, as pandas does.

Parameters:

Name Type Description Default
location str

Local path, file:// URL, or other URL that pandas can read.

required
load_args Mapping[str, Any] | None

Extra pd.read_csv options.

None

Raises:

Type Description
FileNotFoundError

If a local location does not exist.

DataError

If an HTTP(S) request fails or returns an error status. The message names the URL without its credentials or query string (see redact_url).

Source code in src/nemo_safe_synthesizer/data_processing/csv_source.py
@classmethod
def load(cls, location: str, load_args: Mapping[str, Any] | None = None) -> CsvSource:
    """Read the raw bytes of ``location`` once.

    HTTP(S) URLs are fetched with ``httpx``, following redirects, and a
    ``storage_options`` mapping is sent as request headers, as pandas does.

    Args:
        location: Local path, ``file://`` URL, or other URL that pandas can read.
        load_args: Extra ``pd.read_csv`` options.

    Raises:
        FileNotFoundError: If a local ``location`` does not exist.
        DataError: If an HTTP(S) request fails or returns an error status. The message names the
            URL without its credentials or query string (see ``redact_url``).
    """
    args = dict(load_args or {})
    return cls(location, _read_bytes(location, args.get("storage_options")), args)

read()

Parse the file with type inference and the configured load_args.

Source code in src/nemo_safe_synthesizer/data_processing/csv_source.py
def read(self) -> pd.DataFrame:
    """Parse the file with type inference and the configured ``load_args``."""
    return self._read_csv(self.load_args)

read_text()

Parse the file with every cell as the string written in it, including blanks and "N/A".

Options in TEXT_READ_EXCLUDED_LOAD_ARGS are dropped; separators, encodings, and other parsing options still apply.

Source code in src/nemo_safe_synthesizer/data_processing/csv_source.py
def read_text(self) -> pd.DataFrame:
    """Parse the file with every cell as the string written in it, including blanks and ``"N/A"``.

    Options in ``TEXT_READ_EXCLUDED_LOAD_ARGS`` are dropped; separators,
    encodings, and other parsing options still apply.
    """
    args = {key: value for key, value in self.load_args.items() if key not in TEXT_READ_EXCLUDED_LOAD_ARGS}
    return self._read_csv({**args, "dtype": str, "na_filter": False})

redact_url(url)

Return url without credentials, query string, or fragment, which can hold an access token.

Example

"https://host/people.parquet?token=abc" returns "https://host/people.parquet".

Source code in src/nemo_safe_synthesizer/data_processing/csv_source.py
def redact_url(url: str) -> str:
    """Return ``url`` without credentials, query string, or fragment, which can hold an access token.

    Example:
        ``"https://host/people.parquet?token=abc"`` returns ``"https://host/people.parquet"``.
    """
    if "://" not in url:
        return url
    parsed = urlparse(url)
    return parsed._replace(netloc=parsed.netloc.rpartition("@")[2], query="", fragment="").geturl()