Development Notes¶

These notes record the curation decisions represented by the current pipeline.

Dataset Scope¶

The canonical object is named CTRPv2.
CTRPv2.0 is the canonical source for sample records, treatment records, experiments, published curve profiles, and raw dose-response values.
CTRPv2.1 supplies the published AUC companion table and the microarray and copy-number molecular profiles.
CTRPv2.2 is a narrower ACME analysis package from 2015, while CTRPv2.1 is the 2016 molecular-correlation release. Release numbering therefore does not establish analytical precedence.
CTRPv2.1 contains 326,148 AUC rows, 835 cell lines, 481 compounds, and 319,395 unique compound-cell pairs. CTRPv2.2 contains 260,496 rows, 645 cell lines, the same 481 compounds, and 253,433 unique pairs.
Every CTRPv2.1 AUC row matches the canonical CTRPv2.0 curve table on experiment_id, master_cpd_id, master_ccl_id, and AUC. Among 245,121 singly represented compound-cell pairs shared by v2.1 and v2.2, 244,390 (99.70%) have equal AUC values. The remaining material differences are concentrated in NCIH524/JHH6 and BFTC905/BFTC909 assignment inconsistencies in v2.2.
CTRPv2.2 contributes 821 AUC pairs outside v2.0/v2.1. Of these, 411 are tied to those assignment inconsistencies. The remaining 407 COV318 and three AGS values lack corresponding v2.0 raw dose-response experiments, so they cannot participate in the required PharmacoGx outcome recomputation.
ACME hotspots and mutation annotations are study-specific analysis products and belong in a dedicated reproduction of the 2015 ACME study.

master_ccl_id is the canonical sample identifier and master_cpd_id is the canonical treatment identifier. Raw names and useful release-specific treatment identifiers remain as provenance columns.
Molecular colData contains a structural sampleid field with the same values as master_ccl_id, as required by MultiAssayExperiment.
tipifarnib-P2 (master_cpd_id 417416) and tipifarnib-P1 (master_cpd_id 417417) remain separate source treatments. legacy_treatment_group records their shared Tipifarnib comparison group.
Drug and cell-line enrichment queries AnnotationDB first and AnnotationGx for unresolved records or missing key fields. In.AnnotationDB and AnnotationGx.Mapped record coverage.
AnnotationDB.Population preserves the AnnotationDB source string. AnnotationDB.Population.Parsed groups those strings as White, Black/African, East Asian, South Asian, Asian unspecified, or Multiracial; unrecognized nonmissing strings remain Other/unclassified and missing values remain NA. The parsed field is the final column of public sample metadata.
Compound and cell-line AnnotationDB misses are written under data/procdata/annotation_audit/ with source identifiers and available AnnotationGx fields.
Failed external annotation requests preserve all source identifiers and produce unresolved annotation fields.

Gene canonicalization uses GENCODE v50 / Ensembl 116 / GRCh38.p14 and human NCBI Gene records.
The resolver updates discontinued Entrez IDs through NCBI Gene history and accepts a unique current gene2ensembl candidate first. X/Y pseudoautosomal pairs with the same current GENCODE symbol and HGNC ID use the chromosome X ENSG as the representative for the source's collapsed Entrez-level measurement. A unique GENCODE Entrez candidate is accepted next, followed by a unique exact current GENCODE gene name.
gene_annotation.tsv is a source-feature reference. It retains unresolved genes and multiple source genes that resolve to the same current Ensembl ID. Resolver-derived columns use GENCODE.* and NCBI.* names.
Molecular assays require unique current Ensembl gene IDs as rownames. A source feature without a unique current Ensembl mapping, or a secondary source feature for an already represented Ensembl ID, is recorded under data/procdata/annotation_audit/.
gene_annotation_misses.tsv records unresolved source names, Entrez IDs, assay membership, mapping status, candidate Ensembl IDs, and current NCBI Gene IDs.
Per-assay molecular audits are grouped under data/procdata/annotation_audit/molecular_features/{microarray,cnv}/ as unmapped.tsv, ambiguous.tsv, and duplicates.tsv.

microarray.gene_expression_log2 contains Affymetrix-derived expression with an internal assay named exprs. Source Affymetrix probeset IDs, Entrez IDs, and gene symbols remain in rowData; probeset IDs are preserved exactly as supplied.
The microarray source value column is mrna_expression_avg_log2. That source label describes the measured expression values and does not indicate RNA sequencing.
cnv.gene_log2 contains gene-level copy-number values with an internal assay named copy_number.
Assay rowData carries source and reference annotations. Assay colData carries the curated sample metadata, with categorical site, histology, AnnotationDB category, and sex fields represented as factors.

TRE colData contains master_ccl_id; TRE rowData contains experiment and treatment attributes. Physical profiles and sensitivity assays contain response measurements without repeated cell-line annotations.
ic50_recomputed is relative IC50, defined at half of the fitted response span. ac50_recomputed is the concentration at absolute 50% viability. Degenerate zero-span fits have both thresholds set to NA.
published_ec50_umol_raw preserves the source value. Analysis-ready published_ec50_umol treats nonpositive values and values above 1e6 as source sentinels.
PharmacoGx computes AUC and AAC from its fitted four-parameter curve over the tested log-concentration range for each response profile. CTRP published AUC instead integrates its constrained three-parameter sigmoid, or a conditional two-parameter refit when the initial apparent EC50 exceeds the highest tested concentration. The source method is described by Rees et al. (https://doi.org/10.1038/nchembio.1986). Fit warnings and endpoint validity fields are stored in recomputed_profile_qc.tsv.
Dose-response experiments are assigned deterministically to the configured 96 chunks by a stable hash of response_id. Snakemake cores determine how many one-thread fit jobs run concurrently.
PSet construction requires response-level QC coverage and at least one successful or degenerate fit before writing public outputs.
The Hill-slope upper bound and other scientific fit settings are defined in config/pipeline.yaml.

The public PharmacoSet @annotation slot contains constructor-level portable metadata and sessionInfo.
The @curation slot contains compact sample, tissue, and treatment identifier mappings.
molecularProfiles(pset) is a MultiAssayExperiment containing the microarray and copy-number SummarizedExperiment objects.
treatmentResponse(pset) is a TreatmentResponseExperiment containing the published and recomputed profile metrics plus observed dose-response values.

pixi run setup installs these development refs: