Better test setup

[edisonywh]

Before users adopt Frankenstein, they would already have a control code path, that is most likely already tested.

When they adopt Frankenstein, they most likely want to also test that if the candidate branch is ran instead, then their existing test should still pass.

Frankenstein right now has no ability to return candidate result; it always return control result, so this becomes impossible.

We can think about an API like

Frankenstein.adopt(fn ->
  do_something()
end)

where do_something/0 will hit a Frankenstein.run/1 that will instead return the candidate result. Potentially look into nimble_ownership for inspiration.

Alternatively, what would also be nice is the ability to automatically let users run two versions of tests. For example, proposed API

@frankenstein true
test "coconut" do
  do_something()
end

this tag, somehow making us compile two version of test on compile time, so the output will be:

# the OG control
test "coconut" do
  do_something()
end

# the candidate branch
test "(frankenstein) coconut" do
  Frankenstein.adopt(fn ->
    do_something() # this is the candidate branch
  end)
end

[peterhorne]

Alternate idea: a test ‘mode’ that can be enabled by putting something like Frankenstein.enable_test_mode! in your test helper. When in test mode:

• The process that invoked Frankenstein.run() now waits for test results
• Test results are compared, when they do not match an exception is raised, failing the test
• If candidate raises an exception, it is re-raised on the original process, also failing the test

Pros:

• All code paths containing experiments will have both variants tested automatically. Users don’t need to manually find all tests that exercise an experiment and update them
• User defined compare functions are tested (sort of)

Cons:

• Users have less flexibility
• Tests aren’t exercised with results from calling candidate fns
• Detecting errors in candidate fns relies on the correctness of the experiment’s compare fn

Imo it’s a question of what’s better: having perfect coverage (automatic) with imperfect tests vs imperfect coverage (manual) with perfect tests.