Skip to content

Commit ac3cd1b

Browse files
authored
Merge pull request #39 from Genentech/estimator-qc
Estimator qc
2 parents c362c1a + 96c1be5 commit ac3cd1b

58 files changed

Lines changed: 4036 additions & 714 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.claude/CLAUDE.md

Lines changed: 176 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -91,14 +91,38 @@ Fix spelling, grammar, and other minor problems without asking the user. Label a
9191

9292
Only report what you have changed.
9393

94+
## Programming rules for LLMs
95+
1. mostly lower-case comments
96+
2. no in-line comments
97+
3. don't number comments
98+
4. comments are followed by '----', such as "# dependencies----". nothing that takes a full line.
99+
5. no unnecessary code changes beyond what is already done
100+
6. unless I ask, don't print too much code at once
101+
7. don't do unnecessary print statements within the code.
102+
8. don't add unnecessary try/catch statements or make unnecessary validation checks. i'm working by myself, no need for these things -- I want to see the errors!
103+
9. NO EMOJIS
104+
10. in R code, 2 spaces per tab and base R pipe
105+
94106
## Refactor rules
95-
We're attemping to refactor many issues with this repositroy. In general, let's:
96-
- make sure all examples are functional and not wrapped in dontrun{}
107+
108+
### Per-function checklist
109+
110+
When refactoring any estimator or internal function, apply all of these:
111+
112+
1. Make runnable examples (no `\dontrun{}`)
113+
2. Remove commented-out code
114+
3. Remove TODOs
115+
4. Add type checks (checkmate)
116+
5. Factor out tidyverse (`filter``df[cond, ]`, `mutate` → direct assignment)
117+
6. `<-` for assignment (not `=`)
118+
7. Drop debug prints (`cat`, `print`, commented `# print(...)`)
119+
8. Consistent variable naming without periods (e.g. `pi_S` not `pi.S`)
120+
9. Drop backtick column access (`temp$\`piA\`` → `temp$piA`)
121+
122+
### General rules
123+
97124
- make sure we have test cases for all functions
98-
- remove @TODO blocks and other ugly code
99125
- remove magrittr pipe and replace with base R pipe
100-
- remove tidyverse dependencies
101-
- replace "=" with "<-"
102126
- make sure every function is documented
103127
- add validation to function inputs
104128

@@ -109,3 +133,150 @@ Run this one-liner to validate the package before committing:
109133
```
110134
Rscript -e "devtools::document()" && Rscript -e "styler::style_pkg()" && Rscript -e "spelling::spell_check_package()" && Rscript -e "lintr::lint_package()" && Rscript -e "devtools::check(vignettes = FALSE)"
111135
```
136+
137+
## Changing the API
138+
139+
### Goals
140+
141+
1. **Better method constructors** — replace `setup_method_weighting(method_name="IPW", ...)` with `ec_ipw()`, etc. Each constructor carries its own estimation logic.
142+
2. **Polymorphic dispatch**`run_analysis()` calls a generic on the method object instead of an if/else tree. Adding a new method = writing one constructor.
143+
3. **Merge bootstrap** — bootstrap is an inference option on the method, not a separate code path.
144+
4. **(Future) Model formula interface**`outcome ~ treatment | covariates` instead of column name args.
145+
146+
### Current workflow (to deprecate)
147+
148+
```r
149+
method <- setup_method_weighting(
150+
method_name = "IPW",
151+
optimal_weight_flag = FALSE,
152+
wt = 0,
153+
model_form_piS = "S ~ x1 + x2 + x3 + x4 + x5"
154+
)
155+
156+
analysis <- setup_analysis_primary(
157+
data = SyntheticData,
158+
trial_status_col_name = "S",
159+
treatment_col_name = "A",
160+
outcome_col_name = c("y1", "y2"),
161+
covariates_col_name = c("x1", "x2", "x3", "x4", "x5"),
162+
method_weighting_obj = method
163+
)
164+
165+
res <- run_analysis(analysis)
166+
```
167+
168+
### Desired workflow
169+
170+
```r
171+
method <- ec_ipw(
172+
ps_formula = "S ~ x1 + x2 + x3 + x4 + x5",
173+
weight = NULL, # NULL = optimal, 0 = no borrowing, 0.3 = fixed
174+
bootstrap = 500, # NULL = sandwich SE only
175+
bootstrap_ci_type = NULL # NULL defaults to "perc" when bootstrap is set
176+
)
177+
178+
analysis <- setup_analysis(
179+
data = SyntheticData,
180+
outcomes = c("y1", "y2"),
181+
treatment = "A",
182+
trial_status = "S",
183+
covariates = c("x1", "x2", "x3", "x4", "x5"),
184+
method = method
185+
)
186+
187+
res <- run_analysis(analysis)
188+
```
189+
190+
### Method constructors
191+
192+
| Constructor | Replaces | Phase |
193+
|---|---|---|
194+
| `ec_ipw()` | `setup_method_weighting(method_name="IPW", ...)` | Primary |
195+
| `ec_aipw()` | `setup_method_weighting(method_name="AIPW", ...)` | Primary |
196+
| `did_ec_ipw()` | `setup_method_DID(method_name="IPW", ...)` | OLE |
197+
| `did_ec_aipw()` | `setup_method_DID(method_name="AIPW", ...)` | OLE |
198+
| `did_ec_or()` | `setup_method_DID(method_name="OR", ...)` | OLE |
199+
| `scm()` | `setup_method_SCM(...)` | OLE |
200+
201+
Each constructor returns an S4 method object. The S4 class defines a generic `estimate()` that `run_analysis()` dispatches on — no if/else.
202+
203+
### How dispatch works
204+
205+
```r
206+
# S4 generic
207+
setGeneric("estimate", function(method, data, ...) standardGeneric("estimate"))
208+
209+
# Each method class implements estimate()
210+
setMethod("estimate", "ec_ipw_method", function(method, data, ...) {
211+
# IPW estimation logic lives here
212+
})
213+
214+
# run_analysis() becomes:
215+
run_analysis <- function(analysis_obj) {
216+
estimate(analysis_obj@method, data = analysis_obj@data, ...)
217+
}
218+
```
219+
220+
### Interim dispatch (during migration)
221+
222+
While new method constructors coexist with the old if/else tree in `run_analysis()`, each new class gets an `else if` block at the end of `run_analysis()` that calls `estimate()`:
223+
224+
```r
225+
# In run_analysis.R, BEFORE the final } else { stop(...) }:
226+
} else if (is(method, "ec_ipw_method")) {
227+
res <- estimate(method,
228+
data = data,
229+
outcomes = outcome_col_name,
230+
treatment = treatment_col_name,
231+
trial_status = trial_status_col_name,
232+
covariates = covariates_col_name,
233+
alpha = alpha,
234+
quiet = quiet
235+
)
236+
}
237+
```
238+
239+
This pattern is repeated for each new method class as it's created. The old if/else branches for the legacy classes remain untouched. Once all 6 methods are migrated, the entire if/else tree is replaced with a single `estimate()` call.
240+
241+
New method objects inherit from `method_primary_obj` or `method_OLE_obj`, so they pass the existing `checkmate::assert_class(method, "method_primary_obj")` validation in `setup_analysis_primary()`.
242+
243+
### Implementation order
244+
245+
1. Create full-pipeline regression tests for all 6 methods (old API, locked numerical values) ✓
246+
2. Create all 6 method constructors (start with `ec_ipw()`)
247+
3. Each constructor returns an S4 object with estimation logic via `estimate()` generic
248+
4. For each new method, add an `else if (is(method, "xxx_method"))` to `run_analysis()`
249+
5. Add new-API tests to each pipeline test file (same expected values)
250+
6. Once all 6 are done: refactor `setup_analysis()` into a single function (merge `_primary`/`_OLE`, add `T_cross = NULL`)
251+
7. Once all 6 are done: replace the entire if/else in `run_analysis()` with one `estimate()` call
252+
8. Deprecate `setup_method_weighting`, `setup_method_DID`, `setup_method_SCM`, `setup_analysis_primary`, `setup_analysis_OLE`
253+
254+
### Design decisions
255+
256+
- **S4 classes for method objects** — keeps rigorous type definitions, consistent with existing package patterns.
257+
- **`T_cross` goes in `setup_analysis()`** — it's a property of the study design, not the method.
258+
- **`bootstrap_ci_type` is nullable** — defaults to `"perc"` when `bootstrap` is non-NULL, ignored otherwise.
259+
260+
### Full-pipeline regression tests
261+
262+
Before refactoring any estimator, we lock in its numerical outputs on `SyntheticData` so any code change that alters results is caught. Tests use the old API (setup_method → setup_analysis → run_analysis) with exact values at `tolerance = 1e-6`. As new API constructors are added, we add parallel assertions against the same expected values.
263+
264+
Rename existing `test-vignette_results_*` files and split by method:
265+
266+
| Test file | Method | Covers |
267+
|---|---|---|
268+
| `test-full_pipeline_ec_ipw.R` | EC-IPW | weight=0, optimal, fixed (0.3), bootstrap point estimates |
269+
| `test-full_pipeline_ec_aipw.R` | EC-AIPW | weight=0, optimal, fixed (0.3), bootstrap point estimates |
270+
| `test-full_pipeline_did_ec_ipw.R` | DID-EC-IPW | bootstrap CIs, point estimates |
271+
| `test-full_pipeline_did_ec_aipw.R` | DID-EC-AIPW | bootstrap CIs, point estimates |
272+
| `test-full_pipeline_did_ec_or.R` | DID-EC-OR | bootstrap CIs, point estimates |
273+
| `test-full_pipeline_scm.R` | SCM | bootstrap CIs, point estimates |
274+
275+
Each test file asserts:
276+
- Point estimates (exact values)
277+
- Standard errors / standard deviations (exact values)
278+
- CI bounds for non-bootstrap (exact values)
279+
- Bootstrap point estimates match non-bootstrap
280+
- Borrow weight (where applicable)
281+
282+
Simulation tests (`test-vignette_results_*_simulation.R`) stay separate — they test Monte Carlo properties, not individual estimator outputs.

.github/workflows/R-CMD-check.yaml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -47,5 +47,5 @@ jobs:
4747
- uses: r-lib/actions/check-r-package@v2
4848
with:
4949
upload-snapshots: true
50-
build_args: '"--no-manual"'
51-
args: 'c("--no-manual", "--as-cran")'
50+
build_args: 'c("--no-manual", "--no-build-vignettes")'
51+
args: 'c("--no-manual", "--ignore-vignettes", "--as-cran")'

DESCRIPTION

Lines changed: 14 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -73,18 +73,10 @@ Suggests:
7373
testthat (>= 3.0.0)
7474
Config/testthat/edition: 3
7575
Collate:
76-
'DID_EC_AIPW_bootstrap.R'
77-
'DID_EC_AIPW.R'
78-
'DID_EC_IPW_bootstrap.R'
79-
'DID_EC_IPW.R'
80-
'DID_EC_OR_bootstrap.R'
81-
'DID_EC_OR.R'
8276
'EC_AIPW_OPT_bootstrap.R'
8377
'EC_AIPW_OPT.R'
8478
'EC_IPW_OPT_bootstrap.R'
8579
'EC_IPW_OPT.R'
86-
'SCMboot.R'
87-
'SCM.R'
8880
'bootstrap_class.R'
8981
'method_class.R'
9082
'analysis_class.R'
@@ -94,10 +86,24 @@ Collate:
9486
'method_weighting_class.R'
9587
'analysis_primary_class.R'
9688
'data.R'
89+
'ec_ipw.R'
90+
'did_ec_ipw.R'
91+
'did_ec_aipw.R'
92+
'did_ec_or.R'
93+
'ec_aipw.R'
94+
'legacy_DID_EC_AIPW_bootstrap.R'
95+
'legacy_DID_EC_AIPW.R'
96+
'legacy_DID_EC_IPW_bootstrap.R'
97+
'legacy_DID_EC_IPW.R'
98+
'legacy_DID_EC_OR_bootstrap.R'
99+
'legacy_DID_EC_OR.R'
100+
'legacy_SCMboot.R'
101+
'legacy_SCM.R'
97102
'package.R'
98103
'rdborrow-package.R'
99104
'run_analysis.R'
100105
'run_simulation.R'
106+
'scm.R'
101107
'simulate_X_copula.R'
102108
'simulate_X_dct_mvnorm.R'
103109
'simulate_X_mixture.R'

NAMESPACE

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,14 @@
11
# Generated by roxygen2: do not edit by hand
22

3+
export(did_ec_aipw)
4+
export(did_ec_ipw)
5+
export(did_ec_or)
6+
export(ec_aipw)
7+
export(ec_ipw)
8+
export(estimate)
39
export(run_analysis)
410
export(run_simulation)
11+
export(scm)
512
export(setup_analysis_OLE)
613
export(setup_analysis_primary)
714
export(setup_bootstrap)
File renamed without changes.

R/EC_AIPW_OPT.R

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -291,12 +291,13 @@ EC_AIPW_OPT <- function(data,
291291

292292
boot.out <- boot(
293293
data = df,
294-
statistic = EC_IPW_OPT_bootstrap,
294+
statistic = EC_AIPW_OPT_bootstrap,
295295
outcome_col_name = outcome_col_name,
296296
trial_status_col_name = trial_status_col_name,
297297
treatment_col_name = treatment_col_name,
298298
covariates_col_name = covariates_col_name,
299299
model_form_piS = model_form_piS,
300+
model_form_mu0_ext = model_form_mu0_ext,
300301
optimal_weight_flag = optimal_weight_flag,
301302
wt = wt,
302303
R = R,

0 commit comments

Comments
 (0)