Skip to content

Building downstream libraries on DataCache

See the API reference for signatures, defaults, return values, and errors.

Use the public Cache, fetch_file, inspection, registry and database APIs for new integrations. Private helpers can change between releases, except the compatibility entry points named under notes for specific libraries.

  1. Let callers choose a cache root. Do not create it merely to check whether a dataset is installed: use expected_path, Cache.local_path, or inspection.
  2. Keep presence separate from integrity. exists answers whether a path is present; inspect and validate_file check readability and supplied trusted hash/size. Inspect every required member of a release.
  3. Pass trusted metadata for installed bytes, after transformation. Reuse the same expectations across concurrent writers to a destination.
  4. Make installation and repair explicit. Do not automatically force a refresh merely because a cache is corrupt, inaccessible, or unavailable offline.
  5. Forward timeout, retry settings, progress_callback, and show_progress where your library exposes downloads. Keep logging configuration in the top-level application.
  6. Use database versions to identify data/schema changes. Close connections when finished; use separate connections for independent concurrent work.
  7. Let original network and filesystem exceptions propagate, or retain them as causes when adding domain-specific context. FileValidationError.path and .reason provide structured validation details.
  8. Use install_archive when a released tar file owns a directory tree. Treat only inspect_archive(...).status == "available" as installed and resolve member paths from the returned bundle directory, never from the store.

Reusable downstream toolkit

Downstream libraries should keep domain catalogues, CLI wording and generated indexes, while delegating acquisition and source-store lifecycle to DataCache:

Downstream shape DataCache API Shared behavior
One established file path, as in PyEnsembl annotation/FASTA sources fetch_file or VersionedFileRegistry Atomic download, decompression, retries, resume, progress, integrity and provenance
A released tar tree, as in MHCflurry weights/data VersionedArchiveRegistry or install_archive Ordered parts, safe extraction, consumer receipts, all-or-nothing publication and status
Several separately published files that belong together VersionedDatasetRegistry or install_bundle Pinned versions, all-or-nothing publication and inspection

All explicit download methods forward show_progress, callback, timeout and bounded retry options to the shared transport. Registry construction, status, inspection and local-path resolution are offline. Applications can therefore offer consistent list/info/download behavior without maintaining private network or extraction implementations.

Shared OpenVax cache

OpenVax packages share one cache so that published data, such as test datasets, is downloaded once for all of them. Select its root with get_cache_root, listing the package's own variable before the shared one:

root = datacache.get_cache_root("openvax", "OSTEOSARC_CACHE", "OPENVAX_DATA_CACHE")

OPENVAX_DATA_CACHE, when set, is the root itself; otherwise the platform cache directory named openvax is used. Store content-addressed files under <root>/objects/sha256/<sha256><original suffixes>, as osteosarc and Vaxrank do, so identical files are shared. Keep each package's own receipts and derived files in a subdirectory named after the package. Importing datacache does not import pandas, numpy, or requests, so command-line tools can resolve the root cheaply; those are loaded only by the functions that need them.

Compatibility and scope

Existing positional download and database arguments remain supported. New settings are keyword-only. Filename normalization and archive-selection rules remain compatible; see the download reference.

Cache hits preserve existing bytes, names, schemas, versions, and permissions. New-build validation runs only when a database actually needs rebuilding. The upgrade guide explains legacy source reuse and deliberate repair of previously corrupted data. Optional make_file_readable maintenance lets file owners share selected old 0600 files without changing them during normal cache access.

The existing pandas dependency range remains unchanged. CI exercises pandas 1.4.4, 1.5.3, and current versions; older versions in the historical dependency range are not all tested on modern Python. Nullable dtype support depends on the installed pandas version.

DataCache caches by path or database version, not by tracking remote content changes. Refreshing a URL does not invalidate arbitrary downstream artifacts. Libraries own dataset versions and their dependency relationships.

tqdm is installed by default; enable displays with show_progress=True. HTML table conversion requires the html extra. The library does not install logging handlers or change the process umask. Download retry counts and waits are bounded, while timeout limits connection/read inactivity per attempt, not total elapsed time.

Use versioned bundles when named files must be installed together, or archive installation when a tar archive defines the complete tree. Both work on Linux, macOS and Windows; resumable downloads need POSIX. Individual atomic downloads do not provide multi-file transactions or distributed locking. Unhandled termination can leave staging files behind. SQLite operations depend on SQLite's filesystem locking and journaling. New database publication needs hard-link support. Validate deployment-specific network filesystems and ACL requirements in the consuming application.

The runnable offline example exercises the public download, transform, inspection, CSV, and database APIs together.

Notes for specific libraries

  • MHCflurry: use VersionedArchiveRegistry (or install_archive) for released .tar.bz2 trees and ordered historical parts. Its store_path callback preserves release selection and existing paths. Resolve member paths from the returned bundle; preserve DOWNLOAD_INFO.csv with extra_files. Do not enumerate model files as bundle assets or treat the managed store's existence as a completed download.
  • hitlist / tsarina: use VersionedFileRegistry to retain fixed paths, single-Path returns and legacy root manifests without moving old caches. For a deliberate migration to bundles, the existing {filename, urls, default_version} mapping is accepted with verified=False, as is the cache_dir root callable. This is mapping compatibility, not a drop-in filesystem or return-value migration: download returns asset paths, local_path requires an installed bundle, and status rows contain a BundleInspection. A downstream adapter can preserve its public return types and errors. Keep old paths readable during migration; install into a new managed root instead of overwriting legacy directories.
  • mhcseqs: express each records/manifest pair or multi-file source bundle as one version's assets. Keep schema checks and biological validation in mhcseqs. Generated outputs stay outside the source store. The library no longer needs to compose lock, backup, rename and rollback helpers.
  • Vaxrank / Isovar / Varcode / Topiary: share a root, dataset name and data revision to share a complete installation. Existing digest-addressed individual assets can remain under <root>/objects/sha256/; these APIs do not relocate them or change prediction-cache identities.
  • pyensembl: keeps its fixed paths and derived SQLite/FASTA indexes while using fetch_file; record_provenance=True and inspect_file show where files came from. Bundles are optional for references with trustworthy multi-file metadata. Existing calls to the private _download_and_decompress_if_necessary keep their legacy literal-URL inference when no transform flags are passed (explicit flags use the public parsed-URL rules), with the current permissions and retry behavior.