RNA, ribosome and protein measurements¶

The quantitative layer of orfdata stores measurements of genes, transcripts, reference ORFs, proteins, protein groups, peptides and source-defined regions. A measurement stays at the resolution the experiment supports. Parent-gene RNA abundance is useful context for an ORF; it is not a separate measurement of each ORF in that gene.

Released datasets, each with its own guide:

Human, mouse and dog protein abundance (Pfizer/Oziolor atlas)
Dog and mouse RNA abundance (Pfizer/Oziolor atlas)
Matched human tissue RNA and protein (Wang et al.)
Yeast protein and RNA abundance (CoPY)
Yeast RNA and ribosome occupancy (Ingolia et al.)

See released coverage and gaps; the data roadmap tracks further imports. The bundled human ORF evidence catalogue remains available offline. Its pooled PSM counts describe spectra, not sample-level protein abundance.

Discover, download and query¶

Install orfdata[data] to manage datasets. Creating a registry and listing releases reads metadata only:

from orfdata import DatasetRegistry, ScopedTarget

registry = DatasetRegistry()  # Metadata only; no download.
releases = registry.list(species="dog", assay="digest_ms")

For a registered release:

Call	What it does
`registry.info(name, version)`	Returns its immutable manifest.
`registry.download(name, version)`	Fetches and verifies it.
`registry.open(name, version)`	Reads it offline.
`registry.inspect(name, version)`	Reports missing, available or invalid files without downloading or repairing them.
`registry.download_metadata(name, version)`	Returns the artifact's access terms and file hashes and sizes.

An invalid cache requires an explicit download(..., force=True); old reader paths survive. Python download options such as timeout, retries and progress go to DataCache.

Query an opened dataset by an exact, reference-scoped target. Here transcript is a PyEnsembl Transcript in the dataset's exact reference context:

dataset = registry.open(name, version)
target = ScopedTarget.from_pyensembl(
    transcript, kind="transcript", namespace="ensembl:110"
)
for measurement in dataset.measurements(target=target):
    print(measurement.value, measurement.unit, measurement.status)

Use ScopedTarget.from_orf(bound_reference_orf, namespace=source_release) for a validated reference ORF. Namespaces and identifiers must match the source's mapping table. The default PyEnsembl target uses its unversioned id; request versioned=True when the source uses versioned identifiers. There is no implicit mapping across annotation releases, gene isoforms, ORFs or species. A variant or fusion product does not inherit a native reference protein measurement.

Use ScopedTarget(reference, "region", source_id, source_namespace) for a quantified UTR, intron or other source-defined window. Keep its native feature type and target length in measurement provenance. A region ID does not assert known genomic coordinates or a translated ORF. RNA and Ribo measurements of the same region can be compared through an explicit same_target relationship; such a density ratio describes that region, not a particular uORF inside it. Regions do not automatically bind to PyEnsembl or participate in gene/ORF orthology.

For native sources with unreported genomic metadata, use explicit source references and mapping views to retain the original observation alongside its mapping to a PyEnsembl target.

Choose a native series explicitly when a source supplies several units:

values = dataset.measurements(quantity="protein_abundance", unit="iBAQ", scale="linear")
dataset.export_parquet("ibaq.parquet", unit="iBAQ", scale="linear")

unit, scale and log_base also work with mapped_measurements and existing storage v1-v4 releases. Units are exact and case-sensitive. Selecting scale="log", log_base=2 returns values already on that scale; nothing is transformed. An unavailable series returns no observations. Contradictory linear/log requests raise an error. Omitting selectors preserves all native series.

For example, Pfizer iBAQ and rIBAQ are two series from the same source rows. Selecting or exporting both does not create additional donors, spectra or independent detections. Source row/sample/assay lineage stays attached to each measurement.

The CLI uses the same operations:

orfdata datasets list --species yeast
orfdata datasets info DATASET --version RELEASE
orfdata datasets download DATASET --version RELEASE
orfdata datasets inspect DATASET --version RELEASE
orfdata datasets query DATASET --version RELEASE --quantity protein_abundance

Queries stream JSON lines. --reference ref:SHA256 selects an exact context; target queries also require --target, --kind and --namespace. Unqualified sample/assay ID queries across multiple references are rejected. Existing catalogue commands such as orfdata --gene TPST1 retain their behavior.

What each value means¶

Measurement retains the dataset/version, reference-scoped target, sample and assay IDs, value, native unit, linear/log scale and log base, uncertainty and its definition, detection limit, status, origin and provenance. Measured zero, below detection, not measured, not mapped and missing are distinct. Censored or missing values use None; native sentinels can remain in provenance.

RNA counts/TPM/FPKM, ribosome occupancy, source-defined translation efficiency, LFQ/iBAQ/TMT values, peptide intensity, PSM counts, copies per cell and turnover remain distinct. No conversion is automatic. Derived/predicted values name a method and retain source/input provenance. An estimate from calibration remains an estimate; an intensity/TPM ratio does not acquire molecule-ratio units.

Sample records biological units, pools and source labels. Assay separately records technical replicates, runs, fractions, TMT plex/channel/bridge, normalization and calibration. A sample's strain/breed belongs in its attributes; it need not be the strain used to assemble its reference genome. TargetMapping retains all candidate assignments or an unmapped reason. A protein-group representative does not become a uniquely measured protein. See pairing and orthology comparisons for explicit experimental relationships and derived RNA/protein ratios.

Importing a source¶

Define a DatasetManifest with exact reference contexts, raw source URLs, SHA-256 hashes and byte sizes, citations, file-specific access/redistribution terms, searched/measurable coverage, processing software/version/parameters and row accounting. Unavailable metadata stay unknown. Article licensing does not substitute for the data files' terms.

An importer calls QuantitativeDataset.create(path, manifest, samples=..., assays=..., measurements=..., mappings=...). These inputs may be generators; measurements are written incrementally.

Every observation names a source asset or earlier input measurements. Derived inputs precede outputs in the same dataset release. A derived measurement names its inputs in one of two forms, which cannot be mixed:

input_measurements: IDs within the output reference.
input_references: MeasurementReference records with each input's dataset, version, reference and ID.

Cross-species derived abundance is rejected. Duplicate IDs, broken foreign keys, wrong dataset releases and cross-reference links fail the import. Failed builds do not replace data.

Use build_dataset(manifest, builder, ...) for a managed local import. builder(source_paths, output_path) reads the verified inputs and creates that SQLite dataset. DataCache publishes it atomically with input/output checksums and a processing receipt. source_paths={asset_id: local_file} supports manually acquired sources. Importing locally does not grant redistribution rights.

A DatasetRegistry(definitions, cache_root=...) registers releases as:

definitions = {
    name: {
        "default_version": version,
        "versions": {version: {
            "manifest": manifest.to_dict(),
            "distribution": {"access_terms": terms, "redistributable": False},
            "assets": {"dataset.sqlite": {
                "url": url, "sha256": sha256, "size": size, "decompress": False,
            }},
        }},
    }
}

CLI users can supply this JSON with orfdata datasets --registry FILE ....

Rebuilding a release from source¶

Rebuild supported releases from their pinned original inputs with registry.build(name, version, source_paths={"asset_id": "/local/file"}) or orfdata datasets build NAME --source-path ASSET=PATH. Local overrides still require the published hashes. Rebuilds use a separate import cache and preserve installed releases. Unsupported source importers fail explicitly.

Use registry.open_build(name, version) to reopen a source rebuild offline, or registry.inspect_build(name, version) to inspect it. Normal open still selects the published artifact. The CLI makes the same choice explicit:

orfdata datasets build DATASET --version RELEASE
orfdata datasets inspect DATASET --version RELEASE --built
orfdata datasets query DATASET --version RELEASE --built --unit counts

pairs --built also uses the local rebuild. Read operations never acquire or repair data.

Build inspection checks source pins, transform identity and output bytes against the saved receipt, even after local source overrides have been removed. Its verified field remains false because a build receipt is not an independent published output checksum. Both cache origins retain their original paths and provenance; explicit force rebuilding preserves existing readers.

Cache location and storage¶

The cache root is ORFDATA_CACHE, then OPENVAX_DATA_CACHE, then the platform's OpenVax cache. orfdata owns its orfdata/ child. Release cache identities include manifest/reference and content hashes.

Datasets are SQLite files. Every table is partitioned by reference key, with composite foreign keys, so human, mouse, dog, yeast and arbitrary other taxa use the same schema. Compact storage and compatibility describes new SQLite v4 writes, legacy readers and actual-source size/query benchmarks.

The core API needs neither pandas nor PyArrow. The optional orfdata[columnar] extra enables dataset.export_parquet(path). Its query columns accompany the full measurement JSON, preserving exact values and provenance even when a floating-point analysis column cannot represent an integer exactly.