knmidata.opendata package

Submodules

Module contents

KNMI Open Data API client components.

class knmidata.opendata.AsyncOpenDataClient(*, transport: AsyncTransport, auth: Auth)

Bases: object

Client for the KNMI Open Data API (file-based bulk downloads).

Covers the whole Open Data workflow: list_files (catalog with cursor pagination and time filtering), get_download_url (presigned S3 URL per file), stream_file/download_files (atomic concurrent downloads to disk), open_dataset (open files already on disk into an xarray.Dataset), and get_dataset (list + download-if-missing + open in one call). Access it via client.opendata, or instantiate directly with your own transport/auth.

Parameters:
  • transport (AsyncTransport) – Shared HTTP transport and concurrency semaphore.

  • auth (httpx.Auth) – Authentication strategy for the Open Data gateway.

auth: Auth
async download_files(dataset_name: str, dataset_version: str, filenames: list[str], destination_dir: str | Path | None = None) → list[Path]

Download multiple files concurrently.

Parameters:
  • dataset_name (str) – Name of the KNMI dataset.

  • dataset_version (str) – Dataset version identifier.

  • filenames (list[str]) – List of filenames to download.

  • destination_dir (PathLike | None, default=None) – Directory to save files into. Uses a temporary directory if None.

Returns:

Local paths of downloaded files, in the same order as filenames.

Return type:

list[Path]

Raises:

KNMIError – If one or more files could not be downloaded.

async get_dataset(dataset_name: str, dataset_version: str, start_time: str | Timestamp | datetime | date | None = None, end_time: str | Timestamp | datetime | date | None = None, cache_dir: str | Path | None = None, file_pattern: str | None = None, max_files: int | None = None, engine: str = 'h5netcdf', combine: Literal['by_coords', 'nested', 'no_conflicts'] = 'by_coords', concat_dim: str | None = None, **open_dataset_kwargs: Any) → Dataset

Get matching archive files via Open Data and assemble an xarray.Dataset.

If cache_dir is given, files already on disk (matching the catalog size) are reused instead of re-downloaded, and the dataset is opened lazily from disk via open_dataset. If cache_dir is None, files are downloaded to a temporary directory, loaded into memory, and removed again.

Parameters:
  • dataset_name (str) – KNMI dataset name.

  • dataset_version (str) – Dataset version identifier.

  • start_time (TimeLike | None, default=None) – Start timestamp (ISO-8601 string, pandas Timestamp, datetime, or date; naive values are assumed UTC).

  • end_time (TimeLike | None, default=None) – End timestamp (TimeLike, see start_time).

  • cache_dir (PathLike | None, default=None) – Local folder to cache downloaded NetCDF/HDF5 files in. If None, files are downloaded to a temporary directory and loaded into memory.

  • file_pattern (str | None, default=None) – Optional regular expression to keep only matching file names.

  • max_files (int | None, default=None) – Maximum number of files to download (after listing, ascending order).

  • engine (str, default="h5netcdf") – Xarray backend engine used to open each file (see files_to_dataset).

  • combine (CombineMethod, default="by_coords") – How to combine the per-file datasets (see files_to_dataset).

  • concat_dim (str | None, default=None) – Dimension to concatenate along when combine=’nested’.

  • **open_dataset_kwargs (Any) – Additional keyword arguments passed to xarray.open_dataset per file.

Returns:

Combined multidimensional dataset.

Return type:

xr.Dataset

Raises:

NoFilesFoundError – If the query matches no files.

async get_download_url(dataset_name: str, dataset_version: str, filename: str) → str

Retrieve a temporary S3 presigned download URL for a file.

Parameters:
  • dataset_name (str) – Name of the KNMI dataset.

  • dataset_version (str) – Dataset version identifier.

  • filename (str) – File name inside the dataset.

Returns:

Temporary presigned download URL.

Return type:

str

async list_files(dataset_name: str, dataset_version: str, start_time: str | Timestamp | datetime | date | None = None, end_time: str | Timestamp | datetime | date | None = None, max_keys: int = 1000, sorting: str | None = None, order_by: str | None = None, file_pattern: str | None = None, max_files: int | None = None) → list[str]

Fetch matching file names from the catalog, handling pagination automatically.

Parameters:
  • dataset_name (str) – Name of the KNMI dataset (e.g., ‘rad_nl25_rac_mfbs_01h’).

  • dataset_version (str) – Dataset version identifier (e.g., ‘2.0’).

  • start_time (TimeLike | None, default=None) – Start timestamp filter (ISO-8601 string, pandas Timestamp, datetime, or date; naive values are assumed UTC). Requires server-side ordering by modification time, so order_by defaults to ‘lastModified’ when a time filter is given.

  • end_time (TimeLike | None, default=None) – End timestamp filter.

  • max_keys (int, default=1000) – Number of keys to request per pagination batch. Clamped to the API maximum of 1000.

  • sorting (str | None, default=None) – ‘asc’ or ‘desc’.

  • order_by (str | None, default=None) – Attribute to order/filter by: ‘filename’ (default), ‘lastModified’, or ‘created’. Set to ‘lastModified’ (the default when start_time or end_time is given) to make begin/end act as ISO-8601 timestamps; with the default ‘filename’ ordering, begin/end are compared as filename strings instead.

  • file_pattern (str | None, default=None) – Optional regular expression applied client-side to keep only matching file names.

  • max_files (int | None, default=None) – Stop paginating once this many matching files have been collected (after client-side pattern filtering).

Returns:

List of unique file names available for download.

Return type:

list[str]

async open_dataset(source: str | Path | Sequence[str | Path], engine: str = 'h5netcdf', combine: Literal['by_coords', 'nested', 'no_conflicts'] = 'by_coords', concat_dim: str | None = None, **open_dataset_kwargs: Any) → Dataset

Open already-downloaded files from disk into an xarray.Dataset (no network).

Parameters:
  • source (PathLike | FilePaths) – A directory (all files in it, sorted by name — e.g. a cache_dir previously filled by get_dataset) or an explicit list of file paths.

  • engine (str, default="h5netcdf") – Xarray backend engine used to open each file (see files_to_dataset).

  • combine (CombineMethod, default="by_coords") – How to combine the per-file datasets (see files_to_dataset).

  • concat_dim (str | None, default=None) – Dimension to concatenate along when combine=’nested’.

  • **open_dataset_kwargs (Any) – Additional keyword arguments passed to xarray.open_dataset per file.

Returns:

Combined multidimensional dataset, opened lazily from disk.

Return type:

xr.Dataset

async stream_file(url: str, target_path: Path, expected_size: int | None = None) → Path

Stream download a file from a presigned URL in bounded memory.

The file is written to a .part sibling and atomically moved into place, so an interrupted download is never mistaken for a complete file. Existing complete files are skipped.

Parameters:
  • url (str) – Direct presigned download URL.

  • target_path (Path) – Destination path on local disk.

  • expected_size (int | None, default=None) – Expected file size in bytes; used to validate the download and to decide whether an existing file can be skipped.

Returns:

Path to the downloaded file.

Return type:

Path

transport: AsyncTransport
knmidata.opendata.files_to_dataset(file_paths: Sequence[str | Path], engine: str = 'h5netcdf', combine: Literal['by_coords', 'nested', 'no_conflicts'] = 'by_coords', concat_dim: str | None = None, **open_dataset_kwargs: Any) → Dataset

Assemble local NetCDF/HDF5 files into an xarray.Dataset.

Implemented with xr.open_dataset + xr.combine_by_coords so that no dask installation is required.

Parameters:
  • file_paths (FilePaths) – Paths to the downloaded NetCDF files.

  • engine (str, default="h5netcdf") – Xarray backend engine.

  • combine (CombineMethod, default="by_coords") – How to combine the per-file datasets: ‘by_coords’ (combine_by_coords), ‘nested’ (concat along concat_dim), or ‘no_conflicts’ (merge).

  • concat_dim (str | None, default=None) – Dimension to concatenate along when combine=’nested’.

  • **open_dataset_kwargs (Any) – Additional options forwarded to xr.open_dataset.

Returns:

Combined multidimensional dataset.

Return type:

xr.Dataset

Raises:

KNMIError – If file_paths is empty, or combine=’nested’ without concat_dim.