Skip to content

GOAL-12: Alpha Validation, Robustness and Research Approval Framework #41

Description

@RyanLu0203

GOAL-12: Alpha Validation, Robustness and Research Approval Framework

Objective

GOAL-11 established the research-only quantitative foundation:

  • PIT-safe feature construction;
  • interpretable alpha scoring;
  • fixed chronological linear-ranking baseline;
  • deterministic research artifacts;
  • evaluation primitives and governance audits;
  • production, recommendation, paper-trading, trading, broker and model-promotion locks preserved.

GOAL-12 must now determine whether any GOAL-11 feature or alpha signal has credible, reproducible, out-of-sample cross-sectional predictive evidence.

This goal is not allowed to assume that a factor is useful merely because the pipeline runs. It must actively try to falsify candidate signals, distinguish statistical noise from repeatable evidence, and produce an auditable research decision.

The intended transition is:

GOAL-11: research infrastructure exists
                  |
                  v
GOAL-12: signal validity is tested rigorously
                  |
                  v
Future GOAL-13: risk and portfolio intelligence only if evidence warrants it

Authoritative dependency

Repository:

RyanLu0203/A_share_premarket_core

Authoritative branch:

project-current

GOAL-11 merged through PR #40 with merge commit:

25273bb3d3cf9d6eb6c21caf1317c5c56f086489

Before any implementation:

  1. fetch origin;
  2. verify latest origin/project-current contains the GOAL-11 merge commit;
  3. resolve any legitimate newer movement on project-current;
  4. do not continue from the merged GOAL-11 branch;
  5. create one dedicated branch:

codex/goal12-alpha-validation-robustness

Do not modify project-current directly.


Non-negotiable boundary

This goal remains research-only and non-actionable.

It must not create or unlock:

  • BUY / SELL / HOLD outputs;
  • target prices;
  • target weights;
  • position sizes;
  • portfolio allocations;
  • order quantities;
  • broker connectivity;
  • paper trading;
  • live trading;
  • production model promotion;
  • automated daily recommendations;
  • user-facing actionable recommendations;
  • equity curves presented as investable performance;
  • claims that any factor is production-ready.

All candidate outputs must remain explicitly labeled as one of:

  • research_supported_candidate;
  • research_weak_evidence;
  • research_unstable;
  • research_rejected;
  • research_insufficient_data.

Every result must also carry:

production_ready = false

No candidate may bypass the existing recommendation, trading, broker, production, factor-mining, paper-trading, DQN or RL locks.


Stage 0 — Architecture, data and sufficiency audit

Before adding code, audit the merged GOAL-11 implementation and current governed data.

Review at minimum:

  • feature schema and feature versions;
  • source snapshot contracts;
  • PIT rules;
  • historical panel coverage;
  • symbol coverage by date;
  • missingness patterns;
  • qfq adjustment contract;
  • amount-null contract;
  • label-generation capability;
  • existing chronological split and ranking baseline;
  • existing risk/recommendation/backtest locks;
  • artifact and checksum policy;
  • current canonical interface and API topology.

Create:

docs/quant/GOAL12_ALPHA_VALIDATION_ARCHITECTURE.md

The document must define:

  1. exact research questions;
  2. data eligibility rules;
  3. label definitions;
  4. train/validation/test chronology;
  5. statistical tests;
  6. null hypotheses and negative controls;
  7. multiple-testing control;
  8. factor-decision taxonomy;
  9. artifact policy;
  10. hard boundaries that prevent actionability.

Data sufficiency gate

Calculate and document:

  • number of eligible dates;
  • number of symbols per eligible date;
  • total eligible symbol-date rows;
  • missingness by feature;
  • horizon-specific usable sample counts;
  • cross-sectional breadth by date;
  • corporate-action exclusions or adjustments;
  • earliest and latest eligible feature dates;
  • earliest and latest realizable label dates.

Define minimum sample thresholds before inspecting final results.

If the available history is insufficient for responsible inference, the goal must still implement and validate the framework, but the final scientific conclusion must be research_insufficient_data. Do not manufacture positive evidence.


Stage 1 — PIT-safe forward-label contract

Implement a versioned label layer using approved qfq market data.

Required research horizons:

  • forward 1 trading-day return;
  • forward 5 trading-day return;
  • forward 20 trading-day return, only when sufficient future observations exist.

For feature date t, labels must use only prices strictly after the feature-information cutoff.

Define explicitly:

  • feature timestamp and information cutoff;
  • entry reference convention;
  • future price reference convention;
  • horizon trading-calendar mapping;
  • suspension/missing-price handling;
  • delisting or unavailable-future-row handling;
  • duplicate-key handling;
  • corporate-action handling under qfq;
  • label nullability;
  • eligibility and exclusion reasons.

Required label metadata:

symbol
feature_date
horizon
label_date
forward_return
label_version
source_snapshot_id
source_data_checksum
calendar_version
code_commit
eligibility_status
exclusion_reason

The label pipeline must be:

  • deterministic;
  • PIT safe;
  • reproducible;
  • checksum governed;
  • independent of candidate factor scores;
  • tested against off-by-one and same-day leakage.

Forbidden:

  • random date reassignment;
  • same-day close used as future information when unavailable at decision time;
  • filling missing future prices with zero return;
  • silently shortening horizons;
  • using labels to alter feature construction.

Stage 2 — Single-factor validation engine

Evaluate each eligible GOAL-11 feature independently before evaluating combined models.

For every feature and horizon, calculate where mathematically defined:

Cross-sectional association

  • Pearson IC;
  • Spearman Rank IC;
  • IC mean;
  • IC median;
  • IC standard deviation;
  • IC information ratio;
  • positive-IC ratio;
  • number of valid dates;
  • effective symbol breadth per date.

Monotonicity and bucket analysis

Use date-wise cross-sectional buckets where breadth permits:

  • quantile mean forward return;
  • quantile median forward return;
  • top-minus-bottom forward-return spread;
  • monotonicity score;
  • minimum observations per quantile;
  • dates skipped because breadth is insufficient.

These are factor-research statistics only. Do not compound them into an equity curve or present them as an investable strategy.

Decay

Where sufficient observations exist, compare signal behavior across 1D, 5D and 20D horizons.

Do not label a factor useful solely because one horizon is positive.

Missingness and implementation risk

Report:

  • missing rate;
  • zero-variance dates;
  • outlier sensitivity;
  • winsorization or clipping policy, if any;
  • rank ties;
  • exposure concentration;
  • symbol and date concentration.

Any preprocessing must be fit on training history only where learning is involved. Do not use full-period statistics to normalize out-of-sample observations.


Stage 3 — Null models, negative controls and false-discovery protection

GOAL-12 must test whether apparent results can arise by chance.

Implement deterministic, seeded research controls such as:

  1. within-date shuffled factor ranks;
  2. date-shifted factors that should have no valid causal timing;
  3. random-score rankings with preserved cross-sectional breadth;
  4. constant or near-constant factor negative control;
  5. naive benchmark scores already available in the repository, where contractually appropriate.

Requirements:

  • deterministic seeds recorded;
  • no contamination of canonical outputs;
  • null distributions stored only as governed local artifacts;
  • confidence intervals reported;
  • observed metrics compared against null distributions;
  • test count recorded.

Apply an explicit multiple-testing policy across factor × horizon comparisons, such as Benjamini–Hochberg false discovery rate, with the method and alpha level versioned before final result inspection.

Do not use uncorrected isolated p-values as proof of alpha.

Where classical assumptions are weak, use date-level resampling or another justified method that respects temporal clustering. Do not independently resample symbol-date rows as though they were IID.


Stage 4 — Chronological out-of-sample and walk-forward validation

Validate the GOAL-11 interpretable alpha and fixed linear-ranking baseline using strict temporal separation.

Required properties:

  • chronological ordering only;
  • expanding-window or rolling-window design documented and versioned;
  • training dates strictly precede validation dates;
  • validation dates strictly precede held-out test dates where the sample permits;
  • embargo or gap when required by overlapping forward horizons;
  • no random train/test split;
  • no tuning on the final held-out period;
  • no target leakage through normalization, feature selection or imputation;
  • deterministic fold definitions.

Purging and overlap handling

For multi-day forward labels, prevent label-window overlap from leaking future observations across fold boundaries.

Implement and test a purged chronological split or equivalent evidence-backed approach.

Model scope

The existing fixed linear-ranking baseline is the principal combined-model baseline.

Do not introduce a large model zoo. A second simple benchmark is allowed only when it materially improves scientific comparison and adds no heavy dependency or model binary.

Do not add deep learning, transformers, RL, DQN or automated hyperparameter sweeps.

Required out-of-sample metrics

Per fold and aggregated:

  • Pearson IC;
  • Rank IC;
  • Precision@K;
  • Recall@K, only with a clearly defined research relevance threshold;
  • NDCG@K or another justified ranking metric;
  • rank turnover;
  • top-K overlap stability;
  • prediction dispersion;
  • coefficient/sign stability for linear models;
  • feature contribution stability;
  • valid dates and effective sample size.

Do not report classification accuracy without a justified classification target.


Stage 5 — Robustness and sensitivity analysis

Test whether conclusions survive reasonable perturbations.

Required analyses, subject to data sufficiency:

Subperiod stability

  • early versus late sample;
  • rolling windows;
  • expansion windows;
  • exclusion of the most recent period as a sensitivity check.

Market-state segmentation

Use only existing governed regime features or deterministic contracts.

Examples:

  • higher versus lower volatility;
  • positive versus negative index trend;
  • broader versus narrower market breadth.

Do not data-mine arbitrary regime thresholds to create a pass.

Universe sensitivity

  • all eligible symbols;
  • minimum-history subset;
  • minimum-observation subset;
  • required-universe concentration analysis.

Do not silently exclude weak-performing symbols.

Preprocessing sensitivity

Compare only a bounded set of predeclared choices, such as:

  • raw cross-sectional ranks;
  • predeclared clipping/winsorization;
  • missing-value exclusion versus permitted training-only imputation.

Horizon sensitivity

Compare 1D, 5D and 20D without selecting only the most favorable horizon.

Transaction-friction warning metrics

Measure rank turnover and signal persistence as implementation-risk diagnostics.

Do not calculate executable orders, holdings, transaction-cost-adjusted PnL or portfolio equity curves in this goal.


Stage 6 — Research decision and approval taxonomy

Build a deterministic research decision engine that consumes the validation evidence and assigns each candidate factor and combined score one status.

Allowed statuses:

research_supported_candidate
research_weak_evidence
research_unstable
research_rejected
research_insufficient_data

The decision contract must consider at minimum:

  • sample sufficiency;
  • out-of-sample Rank IC direction and magnitude;
  • confidence interval;
  • false-discovery-adjusted evidence;
  • sign stability;
  • subperiod stability;
  • null-model comparison;
  • missingness;
  • breadth;
  • turnover/persistence warning;
  • horizon consistency;
  • concentration risk.

The thresholds must be:

  • specified before final decision generation;
  • versioned;
  • documented;
  • tested just inside and outside each boundary;
  • incapable of setting production_ready=true.

A research_supported_candidate means only that the factor warrants future risk/portfolio research. It is not approval for recommendation or production use.

Create a governed summary containing:

candidate_id
feature_or_model_version
horizon
research_status
production_ready=false
evidence_summary
warning_codes
sample_counts
metric_summary
null_comparison
stability_summary
provenance
checksum

ready_factor_count for production must remain 0 unless a separate future governance goal explicitly changes that contract.


Stage 7 — Research reporting and artifacts

Create human-readable documentation and machine-readable local outputs.

Required documentation:

  • docs/quant/GOAL12_ALPHA_VALIDATION_ARCHITECTURE.md
  • docs/quant/GOAL12_LABEL_AND_SPLIT_CONTRACT.md
  • docs/quant/GOAL12_STATISTICAL_VALIDATION_METHOD.md
  • docs/quant/GOAL12_RESEARCH_DECISION_POLICY.md
  • a concise final research findings document that clearly separates evidence, limitations and future work.

Required code components should include clear modules for:

  • label generation;
  • eligibility filtering;
  • single-factor metrics;
  • chronological/purged folds;
  • null controls;
  • multiple-testing correction;
  • robustness analysis;
  • research decision generation;
  • audit/validation runner.

Generated full-size datasets, model outputs and detailed reports must remain under governed local output/data roots and must not be committed.

Only small deterministic fixtures and compact canonical contract evidence may be committed when necessary for tests and audits.

No notebooks are required. Prefer tested Python modules, CLI/runner entry points and Markdown methodology.

Dashboard/API boundary

Do not add dashboard pages or alter the canonical API topology in this goal unless a minimal read-only contract is strictly required and all architecture approvals are updated.

Default expectation:

  • preserve 14 canonical interfaces;
  • preserve 22 GET routes;
  • preserve 0 write routes;
  • defer visualization to a separate goal after research validity is established.

Engineering and GitHub cleanliness requirements

One-goal workflow

Use exactly:

  • one issue: this Issue;
  • one implementation branch: codex/goal12-alpha-validation-robustness;
  • one PR into project-current.

Do not create multiple exploratory PRs, duplicate branches or repository copies in GitHub.

Local temporary worktrees are allowed but must not be committed or pushed as extra branches.

Commit hygiene

Use a small number of coherent commits. Avoid noisy checkpoint commits, generated-file churn and repeated baseline rewrites.

Before final push:

  • squash or reorganize meaningless local checkpoints when safe;
  • ensure every committed file belongs to the goal;
  • remove temporary debug code;
  • remove unused fixtures;
  • verify no local absolute paths are committed.

Never commit

  • runtime snapshots;
  • live provider responses;
  • raw payloads;
  • credentials, tokens or cookies;
  • local logs;
  • .venv;
  • Pytest temp directories;
  • caches;
  • full generated feature/label datasets;
  • null-simulation matrices;
  • model binaries;
  • large charts or screenshots;
  • temporary notebooks;
  • local deployment evidence;
  • macOS launchd files generated for one machine;
  • Windows-specific transformed canonical CSV bytes.

Preserve the GOAL-11 LF and SHA-256 cross-platform protections.

Production baseline protection

Do not modify unless a real regression is proved and the change is tightly scoped:

  • Tencent provider architecture;
  • daily refresh;
  • snapshot governance;
  • deployment scripts;
  • launchd configuration;
  • frontend standalone production deployment;
  • canonical provider semantics;
  • qfq and amount-null contracts;
  • production API behavior.

Do not restart, redeploy or mutate the user's local running services from a remote Max task.


Required tests

Add focused tests covering at minimum:

  • forward-label off-by-one behavior;
  • trading-calendar horizon mapping;
  • missing future-price exclusion;
  • no zero-fill for missing labels;
  • PIT cutoff enforcement;
  • chronological fold ordering;
  • purging/embargo around overlapping horizons;
  • training-only preprocessing;
  • no test-period tuning;
  • deterministic feature-label joins;
  • duplicate-key rejection;
  • IC and Rank IC calculations;
  • quantile bucket eligibility and monotonicity;
  • date-level resampling determinism;
  • null-control determinism;
  • false-discovery correction;
  • insufficient-data decisions;
  • threshold boundary cases;
  • supported/weak/unstable/rejected classifications;
  • production_ready=false invariance;
  • production ready_factor_count=0 invariance;
  • no recommendation, weight, order or action fields;
  • artifact checksums and provenance;
  • fresh-clone reproducibility;
  • LF/CRLF byte stability for existing canonical evidence.

Full validation gate

Before opening or finalizing the PR, run all repository-approved validations, including:

  • Python compileall;
  • full pytest with warnings treated as errors;
  • canonical validation profile;
  • GOAL-11 audit;
  • new GOAL-12 audit;
  • architecture parity;
  • workflow audit;
  • safety audit;
  • adapter audit;
  • provider taxonomy audit;
  • PIT audit;
  • leakage audit;
  • destructive-change audit;
  • recommendation/trading/production lock audit;
  • API topology validation;
  • secret scan;
  • raw-payload scan;
  • large-file and artifact hygiene scan;
  • clean-worktree check;
  • fresh-clone validation;
  • fresh clone under Windows-style core.autocrlf=true or an equivalent repository-approved cross-platform byte-stability check.

Frontend validation is required only if frontend source changes. The default expectation is no frontend change.

Do not update baselines to hide failures. Every legitimate baseline change must be causally explained and reproducible.


Acceptance criteria

GOAL-12 is complete when:

  1. The latest authoritative project-current containing GOAL-11 is used as the base.
  2. A PIT-safe, deterministic multi-horizon label contract exists.
  3. Data sufficiency is measured before scientific conclusions are made.
  4. GOAL-11 features are validated independently across eligible horizons.
  5. Null controls and negative controls exist and are deterministic.
  6. Multiple-testing control is applied.
  7. Chronological, purged walk-forward evaluation is implemented.
  8. The interpretable alpha and linear-ranking baseline are evaluated strictly out of sample.
  9. Robustness is tested across bounded predeclared subperiod, regime, universe and preprocessing conditions.
  10. Every candidate receives a deterministic research status.
  11. production_ready remains false and production ready_factor_count remains zero.
  12. No recommendation, position, order, trading or portfolio action output is created.
  13. Research artifacts are reproducible, checksummed, provenance-backed and kept out of Git where appropriate.
  14. Existing deployment/provider/API behavior remains intact.
  15. Full tests, audits, clean-worktree and fresh-clone checks pass.
  16. One clean reviewable PR is opened into project-current.
  17. The PR is not merged automatically.

A valid completion may conclude that all factors are weak, unstable, rejected or insufficiently supported. Scientific honesty is part of acceptance.


Required final report

Update this Issue and the PR with:

  • authoritative base SHA;
  • branch and final commit SHA;
  • changed-file summary;
  • data sufficiency table;
  • feature and label date ranges;
  • eligible sample counts by horizon;
  • exact label contract;
  • exact chronological/purged fold design;
  • single-factor metric summary;
  • combined-model out-of-sample summary;
  • null and multiple-testing results;
  • robustness findings;
  • research status for every candidate;
  • explicit confirmation that production_ready=false and ready_factor_count=0;
  • tests and audits;
  • fresh-clone and cross-platform byte-stability results;
  • PR URL;
  • remaining limitations and the recommended next goal.

Do not overstate predictive evidence. Report negative and inconclusive findings with the same prominence as positive findings.


Completion boundary

Commit and push the valid implementation, then open one Draft PR into project-current.

Stop before:

  • marking the PR ready without an explicit review decision;
  • merging;
  • deploying;
  • restarting local services;
  • unlocking factors for production;
  • implementing risk-based positions;
  • generating recommendations or trading actions.

This is intended as a large multi-pass research engineering task. Perform architecture review, methodology design, implementation, adversarial validation, documentation and fresh-clone verification before stopping.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestneeds-rebaseBranch conflicts with the authoritative base and must be reconciled before reviewscope: research-onlyNon-actionable research scope; no execution or production unlockstatus: current-reviewCurrent item requiring owner or reviewer attention

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions