knmidata.opendata package¶
Submodules¶
Module contents¶
KNMI Open Data API client components.
- class knmidata.opendata.AsyncOpenDataClient(*, transport: AsyncTransport, auth: Auth)¶
Bases:
objectClient for the KNMI Open Data API (file-based bulk downloads).
Covers the whole Open Data workflow: list_files (catalog with cursor pagination and time filtering), get_download_url (presigned S3 URL per file), stream_file/download_files (atomic concurrent downloads to disk), open_dataset (open files already on disk into an xarray.Dataset), and get_dataset (list + download-if-missing + open in one call). Access it via client.opendata, or instantiate directly with your own transport/auth.
- Parameters:
transport (AsyncTransport) – Shared HTTP transport and concurrency semaphore.
auth (httpx.Auth) – Authentication strategy for the Open Data gateway.
- auth: Auth¶
- async download_files(dataset_name: str, dataset_version: str, filenames: list[str], destination_dir: str | Path | None = None) list[Path]¶
Download multiple files concurrently.
- Parameters:
dataset_name (str) – Name of the KNMI dataset.
dataset_version (str) – Dataset version identifier.
filenames (list[str]) – List of filenames to download.
destination_dir (PathLike | None, default=None) – Directory to save files into. Uses a temporary directory if None.
- Returns:
Local paths of downloaded files, in the same order as filenames.
- Return type:
list[Path]
- Raises:
KNMIError – If one or more files could not be downloaded.
- async get_dataset(dataset_name: str, dataset_version: str, start_time: str | Timestamp | datetime | date | None = None, end_time: str | Timestamp | datetime | date | None = None, cache_dir: str | Path | None = None, file_pattern: str | None = None, max_files: int | None = None, engine: str = 'h5netcdf', combine: Literal['by_coords', 'nested', 'no_conflicts'] = 'by_coords', concat_dim: str | None = None, **open_dataset_kwargs: Any) Dataset¶
Get matching archive files via Open Data and assemble an xarray.Dataset.
If cache_dir is given, files already on disk (matching the catalog size) are reused instead of re-downloaded, and the dataset is opened lazily from disk via open_dataset. If cache_dir is None, files are downloaded to a temporary directory, loaded into memory, and removed again.
- Parameters:
dataset_name (str) – KNMI dataset name.
dataset_version (str) – Dataset version identifier.
start_time (TimeLike | None, default=None) – Start timestamp (ISO-8601 string, pandas Timestamp, datetime, or date; naive values are assumed UTC).
end_time (TimeLike | None, default=None) – End timestamp (TimeLike, see start_time).
cache_dir (PathLike | None, default=None) – Local folder to cache downloaded NetCDF/HDF5 files in. If None, files are downloaded to a temporary directory and loaded into memory.
file_pattern (str | None, default=None) – Optional regular expression to keep only matching file names.
max_files (int | None, default=None) – Maximum number of files to download (after listing, ascending order).
engine (str, default="h5netcdf") – Xarray backend engine used to open each file (see files_to_dataset).
combine (CombineMethod, default="by_coords") – How to combine the per-file datasets (see files_to_dataset).
concat_dim (str | None, default=None) – Dimension to concatenate along when combine=’nested’.
**open_dataset_kwargs (Any) – Additional keyword arguments passed to xarray.open_dataset per file.
- Returns:
Combined multidimensional dataset.
- Return type:
xr.Dataset
- Raises:
NoFilesFoundError – If the query matches no files.
- async get_download_url(dataset_name: str, dataset_version: str, filename: str) str¶
Retrieve a temporary S3 presigned download URL for a file.
- Parameters:
dataset_name (str) – Name of the KNMI dataset.
dataset_version (str) – Dataset version identifier.
filename (str) – File name inside the dataset.
- Returns:
Temporary presigned download URL.
- Return type:
str
- async list_files(dataset_name: str, dataset_version: str, start_time: str | Timestamp | datetime | date | None = None, end_time: str | Timestamp | datetime | date | None = None, max_keys: int = 1000, sorting: str | None = None, order_by: str | None = None, file_pattern: str | None = None, max_files: int | None = None) list[str]¶
Fetch matching file names from the catalog, handling pagination automatically.
- Parameters:
dataset_name (str) – Name of the KNMI dataset (e.g., ‘rad_nl25_rac_mfbs_01h’).
dataset_version (str) – Dataset version identifier (e.g., ‘2.0’).
start_time (TimeLike | None, default=None) – Start timestamp filter (ISO-8601 string, pandas Timestamp, datetime, or date; naive values are assumed UTC). Requires server-side ordering by modification time, so order_by defaults to ‘lastModified’ when a time filter is given.
end_time (TimeLike | None, default=None) – End timestamp filter.
max_keys (int, default=1000) – Number of keys to request per pagination batch. Clamped to the API maximum of 1000.
sorting (str | None, default=None) – ‘asc’ or ‘desc’.
order_by (str | None, default=None) – Attribute to order/filter by: ‘filename’ (default), ‘lastModified’, or ‘created’. Set to ‘lastModified’ (the default when start_time or end_time is given) to make begin/end act as ISO-8601 timestamps; with the default ‘filename’ ordering, begin/end are compared as filename strings instead.
file_pattern (str | None, default=None) – Optional regular expression applied client-side to keep only matching file names.
max_files (int | None, default=None) – Stop paginating once this many matching files have been collected (after client-side pattern filtering).
- Returns:
List of unique file names available for download.
- Return type:
list[str]
- async open_dataset(source: str | Path | Sequence[str | Path], engine: str = 'h5netcdf', combine: Literal['by_coords', 'nested', 'no_conflicts'] = 'by_coords', concat_dim: str | None = None, **open_dataset_kwargs: Any) Dataset¶
Open already-downloaded files from disk into an xarray.Dataset (no network).
- Parameters:
source (PathLike | FilePaths) – A directory (all files in it, sorted by name — e.g. a cache_dir previously filled by get_dataset) or an explicit list of file paths.
engine (str, default="h5netcdf") – Xarray backend engine used to open each file (see files_to_dataset).
combine (CombineMethod, default="by_coords") – How to combine the per-file datasets (see files_to_dataset).
concat_dim (str | None, default=None) – Dimension to concatenate along when combine=’nested’.
**open_dataset_kwargs (Any) – Additional keyword arguments passed to xarray.open_dataset per file.
- Returns:
Combined multidimensional dataset, opened lazily from disk.
- Return type:
xr.Dataset
- async stream_file(url: str, target_path: Path, expected_size: int | None = None) Path¶
Stream download a file from a presigned URL in bounded memory.
The file is written to a
.partsibling and atomically moved into place, so an interrupted download is never mistaken for a complete file. Existing complete files are skipped.- Parameters:
url (str) – Direct presigned download URL.
target_path (Path) – Destination path on local disk.
expected_size (int | None, default=None) – Expected file size in bytes; used to validate the download and to decide whether an existing file can be skipped.
- Returns:
Path to the downloaded file.
- Return type:
Path
- transport: AsyncTransport¶
- knmidata.opendata.files_to_dataset(file_paths: Sequence[str | Path], engine: str = 'h5netcdf', combine: Literal['by_coords', 'nested', 'no_conflicts'] = 'by_coords', concat_dim: str | None = None, **open_dataset_kwargs: Any) Dataset¶
Assemble local NetCDF/HDF5 files into an xarray.Dataset.
Implemented with xr.open_dataset + xr.combine_by_coords so that no dask installation is required.
- Parameters:
file_paths (FilePaths) – Paths to the downloaded NetCDF files.
engine (str, default="h5netcdf") – Xarray backend engine.
combine (CombineMethod, default="by_coords") – How to combine the per-file datasets: ‘by_coords’ (combine_by_coords), ‘nested’ (concat along concat_dim), or ‘no_conflicts’ (merge).
concat_dim (str | None, default=None) – Dimension to concatenate along when combine=’nested’.
**open_dataset_kwargs (Any) – Additional options forwarded to xr.open_dataset.
- Returns:
Combined multidimensional dataset.
- Return type:
xr.Dataset
- Raises:
KNMIError – If file_paths is empty, or combine=’nested’ without concat_dim.