Maintain the assembly catalogue¶

pyensembl/assembly_catalog.json is an offline snapshot of assembly identity, separate from cache naming. assembly.py is the shared resolver used by EnsemblRelease and CLI selection. The catalogue ships in both the wheel and source distribution.

Sources and update procedure¶

Numbered records come from the first 4 KiB of each supported species/release GTF, using genome-version, genome-build, and genome-build-accession. Where headers are absent or omit an accession, the capture script reads the small meta.txt.gz in that release's core database dump. It records the source URL and uses assembly.default, assembly.name, and assembly.accession. Each capture includes the compressed GTF prefix's SHA-256.

Dated records use accession-specific entries in the new platform's species.json, filtered to registered/current and historical assemblies. The GRCh37 2013_09 directory is independently verified and retained even if the bulk catalogue omits it. Species-level breed/strain labels are not copied to every assembly. Registered breed accessions identify their own species entries; general dog/mouse constraints may match a known breed/strain.

Run from a development checkout with its dependencies installed:

python scripts/capture_assembly_metadata.py /tmp/ensembl-assembly-captures
python scripts/build_assembly_catalog.py /tmp/ensembl-assembly-captures
./lint.sh
./test.sh

Capture is an explicit online maintenance command. It never uses the user's genome cache, and never downloads full GTFs, DNA, or sequence indexes. Existing captures are reused, so use a new directory for a completely fresh audit. Inspect upstream errors and the generated diff before committing an update. Metadata acquisition must not become an import-time or constructor operation.

The current snapshot audits 1,311 numbered combinations across 43 registered species and contains 52 dated assembly/provider records. Every numbered entry has a build and full assembly name. Sources do not provide a verified accession for these 75 older combinations:

Species	Releases with an unknown accession
Human	54
Mouse	54–60
Zebrafish	48–60
Worm	48–70
Macaque (`macaca_mulatta`)	75–85
Guinea pig	75–89
Malaria parasite	Ensembl Genomes 40–44

These gaps are represented in metadata_gaps, and the corresponding assembly_accession is None. A missing accession must not be replaced by the accession of a later assembly. Existing datasets remain usable offline. The guinea pig release 89 core metadata URL was also absent; its GTF still provides its assembly name.

New API verification¶

EnsemblRelease.from_ensembl explicitly calls the new service's /genome/{accession}/explain, /genome/{uuid}/details, and /genome/{uuid}/ftplinks. UUID input skips accession discovery. It validates the returned identity and parses the annotation date/provider from the actual geneset path. Website release labels and genome UUIDs are saved separately.

Human and dog responses captured on 2026-10-20 are in tests/data/assembly_metadata/. Tests also use independently captured GTF headers and JSON/pickle objects generated by the published 2.28.3 wheel. Refresh those fixtures deliberately; do not regenerate old-version fixtures with the implementation being tested.

Compatibility checks¶

Test the inferred and explicit forms, ambiguous breed dates, numbered patch transitions, missing metadata, unknown accessions, API errors, and malformed or conflicting API links. The resolver must keep ordinary construction offline.

New serialization stores the resolved metadata and cache-layout choice. Deserialization restores that snapshot even when a newer catalogue disagrees. Old states retain their prior species selection and receive metadata only in memory. The accession inferred for an ordinary call must never cause its deserializer to switch to the explicit-accession cache layout.

reference_name remains the installed display/cache label. An upstream name change cannot rename it. The unused EnsemblAnnotation class was removed in 2.30.0; its historical cache files remain reusable through EnsemblRelease. Keep existing EnsemblRelease signatures and serialized objects supported. All URL generation and DNA preparation belong to EnsemblRelease.

For release validation, reopen actual old-version installed caches with downloads and parsing forbidden and compare their contents, paths, permissions, inodes, and modification/change timestamps. Run the full suite against an isolated copy of the integration data, never by migrating a user's cache.

Annotation comparisons¶

pyensembl/release_comparisons.json records ten scoped comparisons from twenty complete GTF sources captured on 2026-10-23. It ships with wheels and sdists. To reproduce the comparison in an isolated directory:

python scripts/capture_release_comparisons.py --output /tmp/ensembl-gtf-comparisons

This command downloads approximately 1.2 GiB, with a 1 GiB limit per source. It does not touch installed annotation caches. It verifies dated GTF MD5s, records SHA256 and manifest provenance, hashes gene/transcript ID.version sets, and hashes the sorted multiset of complete normalized GTF rows. Headers and input ordering are excluded; repeated attributes and duplicate rows are retained. A missing version is recorded as unknown, never guessed. Set comparisons include unversioned IDs so missing versions cannot be mistaken for entirely unrelated genes. No annotation or sequence cache is shared based on these comparisons.

Review release_comparisons.json in the output directory before copying it into the package. --summaries-only recomputes relationships from existing complete summaries and saved identifier sets without network access. For a fresh upstream audit use a new directory. The tiny independent excerpts in tests/data/release_comparisons/ test parser behavior, not the complete-source biological claims. A separate regression indexes the same real multi-isoform GTF grouped, coordinate-sorted, and with all child features first.

The preferred list is reserved for reviewed future ordering that supersedes a frozen numbered selection. Every entry must name an exact release, accession, provider, coverage, and evidence-based reason. New directory listings alone must never add these preferences. Keep latest opt-in and serialize its resolved concrete selection.