Skip to content

Commit d657671

Browse files
emfelthamemfeltham
committed
various changes
1 parent 4c32ada commit d657671

39 files changed

Lines changed: 3114 additions & 16142 deletions

‎.github/workflows/nightly-ci.yml‎

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
name: Nightly CI
2+
3+
on:
4+
schedule:
5+
# Run at 2 AM UTC every day
6+
- cron: '0 2 * * *'
7+
workflow_dispatch: # Allow manual triggering
8+
9+
jobs:
10+
comprehensive-validation:
11+
runs-on: ubuntu-latest
12+
13+
steps:
14+
- uses: actions/checkout@v4
15+
16+
- name: Setup Julia
17+
uses: julia-actions/setup-julia@v1
18+
with:
19+
version: '1.11'
20+
21+
- name: Cache Julia packages
22+
uses: julia-actions/cache@v1
23+
24+
- name: Install dependencies
25+
run: |
26+
julia --project=. -e "using Pkg; Pkg.instantiate()"
27+
28+
- name: Run package tests
29+
run: |
30+
julia --project=. -e "using Pkg; Pkg.test()"
31+
32+
- name: Run comprehensive seed sweep
33+
run: |
34+
julia --project=. scripts/seed_sweep.jl --seeds 30 --N 600 --T 200 --treated-share 0.10 --F 1:8 --L -30:-1 --phi 0.0 --rho 0.5 --sigma-y 0.2 --beta1 0.3 --beta2 0.5 --out test/validation/nightly_seed_sweep.json
35+
36+
- name: Run bias and coverage analysis
37+
run: |
38+
julia --project=. scripts/sim_bias_coverage.jl --seeds 20 --iterations 400 --out test/validation/nightly_bias_coverage.json
39+
40+
- name: Run placebo permutation tests
41+
run: |
42+
julia --project=. scripts/placebo_permutation.jl --permutations 200 --iterations 100 --out test/validation/nightly_placebo.json
43+
44+
- name: Upload comprehensive validation artifacts
45+
uses: actions/upload-artifact@v4
46+
if: always()
47+
with:
48+
name: nightly-validation-results
49+
path: |
50+
test/validation/nightly_*.json
51+
retention-days: 90
52+
53+
- name: Create validation summary
54+
if: always()
55+
run: |
56+
echo "# Nightly Validation Summary" > validation_summary.md
57+
echo "Generated: $(date)" >> validation_summary.md
58+
echo "" >> validation_summary.md
59+
60+
if [ -f test/validation/nightly_seed_sweep.json ]; then
61+
echo "## Seed Sweep Results" >> validation_summary.md
62+
julia --project=. -e "
63+
using JSON
64+
data = JSON.parsefile(\"test/validation/nightly_seed_sweep.json\")
65+
summary = data[\"summary\"]
66+
println(\"- MAE Mean: \", summary[\"mae_mean\"])
67+
println(\"- MAE SD: \", summary[\"mae_sd\"])
68+
println(\"- MxE Max: \", summary[\"mxe_max\"])
69+
" >> validation_summary.md
70+
fi
71+
72+
if [ -f test/validation/nightly_bias_coverage.json ]; then
73+
echo "" >> validation_summary.md
74+
echo "## Bias & Coverage Results" >> validation_summary.md
75+
julia --project=. -e "
76+
using JSON
77+
data = JSON.parsefile(\"test/validation/nightly_bias_coverage.json\")
78+
bias_summary = data[\"bias\"][\"summary\"]
79+
coverage_summary = data[\"coverage\"][\"summary\"]
80+
println(\"- Overall MAE: \", bias_summary[\"overall_mae\"])
81+
println(\"- Overall RMSE: \", bias_summary[\"overall_rmse\"])
82+
println(\"- Coverage Rate: \", coverage_summary[\"overall_coverage\"])
83+
" >> validation_summary.md
84+
fi
85+
86+
if [ -f test/validation/nightly_placebo.json ]; then
87+
echo "" >> validation_summary.md
88+
echo "## Placebo Test Results" >> validation_summary.md
89+
julia --project=. -e "
90+
using JSON
91+
data = JSON.parsefile(\"test/validation/nightly_placebo.json\")
92+
summary = data[\"placebo\"][\"summary\"]
93+
println(\"- Type I Error Rate: \", summary[\"overall_type1\"])
94+
println(\"- Successful Permutations: \", summary[\"n_successful\"], \"/\", summary[\"n_attempted\"])
95+
" >> validation_summary.md
96+
fi
97+
98+
- name: Upload validation summary
99+
uses: actions/upload-artifact@v4
100+
if: always()
101+
with:
102+
name: validation-summary
103+
path: validation_summary.md
104+
retention-days: 90

‎.github/workflows/pr-ci.yml‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
name: PR CI
2+
3+
on:
4+
pull_request:
5+
branches: [ main ]
6+
7+
jobs:
8+
test:
9+
runs-on: ubuntu-latest
10+
11+
steps:
12+
- uses: actions/checkout@v4
13+
14+
- name: Setup Julia
15+
uses: julia-actions/setup-julia@v1
16+
with:
17+
version: '1.11'
18+
19+
- name: Cache Julia packages
20+
uses: julia-actions/cache@v1
21+
22+
- name: Install dependencies
23+
run: |
24+
julia --project=. -e "using Pkg; Pkg.instantiate()"
25+
26+
- name: Run package tests
27+
run: |
28+
julia --project=. -e "using Pkg; Pkg.test()"
29+
30+
- name: Run validation seed sweep
31+
run: |
32+
julia --project=. scripts/seed_sweep.jl --seeds 10 --N 800 --T 250 --treated-share 0.08 --F 1:8 --L -30:-1 --phi 0.0 --rho 0.5 --sigma-y 0.18 --beta1 0.3 --beta2 0.5 --out test/validation/pr_seed_sweep.json
33+
34+
- name: Upload validation artifacts
35+
uses: actions/upload-artifact@v4
36+
if: always()
37+
with:
38+
name: pr-validation-results
39+
path: |
40+
test/validation/pr_seed_sweep.json
41+
retention-days: 30

‎Manifest.toml‎

Lines changed: 41 additions & 3 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Project.toml‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,8 @@ version = "0.1.0"
1111

1212
[deps]
1313
Accessors = "7d9f7c33-5ae7-4f3b-8dc6-eff91059b697"
14+
BenchmarkTools = "6e4b80f9-dd63-53aa-95a3-0cdb28fa8baf"
15+
CSV = "336ed68f-0bac-5ca0-87d4-7b16caf5d00b"
1416
Colors = "5ae59095-9a9b-59fe-a467-6f913c188581"
1517
DataFrames = "a93c6f00-e57d-5684-b7b6-d8193f3e46c0"
1618
DataFramesMeta = "1313f7d8-7da2-5740-9ea0-a2ca25f37964"
@@ -28,8 +30,10 @@ Statistics = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"
2830
StatsBase = "2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91"
2931

3032
[compat]
33+
BenchmarkTools = "1.6.0"
3134
Documenter = "1.14.1"
32-
julia = "1.6"
35+
JLD2 = "0.5.15"
36+
julia = "1.10"
3337

3438
[extras]
3539
Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"

‎README.md‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -54,11 +54,14 @@ Pkg.add("https://github.com/human-nature-lab/TSCSMethods.jl")
5454
- [**Tutorial**](https://human-nature-lab.github.io/TSCSMethods.jl/tutorial/): Step-by-step analysis walkthrough
5555
- [**Methodology**](https://human-nature-lab.github.io/TSCSMethods.jl/methodology/): Statistical methods and assumptions
5656
- [**API Reference**](https://human-nature-lab.github.io/TSCSMethods.jl/api/): Complete function documentation
57+
- [**Validation**](https://human-nature-lab.github.io/TSCSMethods.jl/validation/): Test suite and calibration gates
5758

5859
## Examples
5960

6061
See the [Jupyter notebook vignette](./vignette/vignette.ipynb) for a complete analysis example with simulated data.
6162

63+
For a high-level summary of validation tests, see [VALIDATION_TESTS.md](./VALIDATION_TESTS.md).
64+
6265
## Method Overview
6366

6467
The package implements the extended matching approach developed in Feltham et al. (2023), building on Imai et al. (2021), for time-series cross-sectional data:
@@ -72,8 +75,8 @@ This approach addresses key challenges in panel data analysis: selection bias, t
7275

7376
## System Requirements
7477

75-
- **Julia**: 1.9 or later
76-
- **Memory**: 8GB+ recommended for large datasets
78+
- **Julia**: 1.10 or later
79+
- **Memory**: 8GB+ recommended (larger for larger datasets)
7780
- **OS**: Windows, macOS, or Linux
7881

7982
The package automatically installs all required dependencies.
@@ -119,4 +122,4 @@ Contributions are welcome! Please see the [documentation](https://human-nature-l
119122

120123
- Imai, K., Kim, I. S., & Wang, E. H. (2021). Matching Methods for Causal Inference with Time-Series Cross-Sectional Data. *American Journal of Political Science*.
121124
- Feltham, E., Forastiere, L., Alexander, M., & Christakis, N. A. (2023). Mass gatherings for political expression had no discernible association with the local course of the COVID-19 pandemic in the USA in 2020 and 2021. *Nature Human Behaviour*.
122-
- Kim, I. S., Ruah, A., Wang, E., & Imai, K. (2020). Insongkim/PanelMatch [R, C]. https://github.com/insongkim/PanelMatch (Original work published 2018)
125+
- Kim, I. S., Ruah, A., Wang, E., & Imai, K. (2020). Insongkim/PanelMatch [R, C]. https://github.com/insongkim/PanelMatch (Original work published 2018)

‎RELEASE_CHECKLIST.md‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
# Release Checklist — TSCSMethods.jl
2+
3+
Use this checklist to validate statistical correctness, finalize packaging, and publish a tagged release. Items marked Gate must pass before release.
4+
5+
## Statistical Validation
6+
- Noiseless exact recovery (sanity): Gate
7+
- Run: `julia --project=. test/test_noiseless_exact.jl`
8+
- Expect: oracle ATT equals `delta` exactly; estimator ATT within atol ≤ 1e-6 of `delta`.
9+
- Coverage (null DGP): Gate
10+
- Run: `julia --project=. scripts/sim_bias_coverage.jl --seeds 20 --iterations 400 --out test/validation/bias_coverage.json`
11+
- Expect: overall 95% CI coverage ∈ [0.93, 0.97].
12+
- Placebo/Permutation (real data): Gate
13+
- Run: `julia --project=. scripts/placebo_permutation.jl --permutations 200 --iterations 100 --out test/validation/placebo.json`
14+
- Expect: overall Type I error ∈ [0.03, 0.07].
15+
- Seed sweep (randomized DGP): Report-only
16+
- Run: `julia --project=. scripts/seed_sweep.jl --out test/validation/seed_sweep.json`
17+
- Action: Attach JSON to release or link from docs; do not gate by default.
18+
19+
## Tests & CI
20+
- Unit tests: Gate
21+
- Run: `julia --project=. -e 'using Pkg; Pkg.test()'`
22+
- Ensure new tests pass (noiseless, randomized correctness, confounded DGP, coverage unit).
23+
- CI matrix: Gate
24+
- Green on GitHub Actions for Julia 1.6, 1.10, 1.11 across OSes.
25+
- Validation workflow:
26+
- PR: runs tests + seed sweep (report), uploads JSON artifacts.
27+
- Nightly: runs coverage and placebo (gated), uploads artifacts.
28+
29+
## Packaging & Metadata
30+
- Versioning: Gate
31+
- Bump `Project.toml` version (semantic versioning); add `CHANGELOG.md` entry.
32+
- Julia version consistency: Gate
33+
- Align minimum Julia version across `Project.toml`, README, and docs (CI currently tests 1.6/1.10/1.11).
34+
- Dependencies: Gate
35+
- Ensure optional R/Bayes factor path is disabled by default (`dobayesfactor=false`); no unintended external deps.
36+
37+
## Quality & Docs
38+
- Formatting: Gate
39+
- Run formatter check (GitHub Actions Format Check); fix any diffs.
40+
- Documentation: Gate
41+
- Docs build passes: `julia --project=docs docs/make.jl`.
42+
- Validation docs present: `docs/src/validation.md`; README links included.
43+
- Add brief release notes summarizing validation results and any API changes.
44+
45+
## Tag & Publish
46+
- Create a git tag (e.g., `vX.Y.Z`) and push; verify CI and docs deploy green.
47+
- Attach or link validation artifacts (seed_sweep.json, bias_coverage.json, placebo.json) in the release.
48+
- Announce version, validation status, and highlights in release notes.
49+
50+
Notes
51+
- If any Gate fails, fix or adjust configuration and re-run before tagging.
52+
- Retain validation artifacts for traceability; they demonstrate calibration at release time.

0 commit comments

Comments
 (0)