Support for Bazel

[gergelyfabian]

How could I start using Stryker4s with Scala, when using Bazel as a build system?

I see there is already support for sbt and Maven.

What could be the easiest path to add support for Bazel? What steps would be needed? (I could try contributing a solution)
Is there e.g. a way to use Stryker4s from the command line?

Thank you!

Here is an example Scala project with Bazel: https://github.com/gergelyfabian/bazel-scala-example

[gergelyfabian]

The Bazel plugin I use to run Scala is https://github.com/bazelbuild/rules_scala. In case changes are necessary I could try contributing it over there too.

[gergelyfabian]

Moved the conversation to Slack.

[gergelyfabian]

Tried with such a stryker4s.conf:

# First, install coursier:
#   https://get-coursier.io/docs/cli-installation
# Then run Stryker4s with:
#   cs launch io.stryker-mutator::stryker4s-command-runner:latest.release
# Check for more options:
#   https://stryker-mutator.io/docs/stryker4s/configuration/

stryker4s {
  # Change to the submodule you want to test
  mutate: [ "my-module/**/main/scala/**/*.scala" ]
  scala-dialect: "2.12"
  test-runner: { command: "bazel", args: "test //my-module:all" }
}

Unfortunately this fails with a timeout exception. The issue is, that Bazel has to rebuild a significant part of the code if a new workspace is created (this is done when the code is copied), so it times out. For using Bazel we'd need to enable in-place mutation, because then Bazel caches could be reused.

StrykerJs already has such an option:
https://stryker-mutator.io/docs/stryker-js/configuration/#inplace-boolean

[gergelyfabian]

Another option is to enable overriding the tmpDir in src/main/scala/stryker4s/run/MutantRunner.scala.prepareEnv(), so that we can force the tmpDir to be created in a predefined place, that would enable prebuilding the code with Bazel and be able to use the cache when running mutation tests.

[gergelyfabian]

I've managed to put together a version where Stryker4s can work with Bazel.

Sent some PRs as improvements (merged now):

• Adding local publish support for command runner: sbt: enable local publish for stryker4s-command-runner #1262
• staticTmpDir support, to ensure Bazel's cache works better: Add staticTmpDir static path and cleanTmpDir config options #1263

Other details:

• stryker uses env vars, this needs special care with Bazel, as Bazel cuts out environment vars (they do not affect test results and caching results). Need to use --test_env=ACTIVE_MUTATION to make that env var have effect on test inputs.
• I needed to exclude MethodExpression mutations as they seemed to generate such methods that were non-existent in my codebase (e.g. for scala.util.Try)

Here is the config I used:

# Example Stryker4s configuration.
# https://stryker-mutator.io/docs/stryker4s/getting-started/
#
# Manual steps to do:
#
# First, install coursier:
#   https://get-coursier.io/docs/cli-installation
# Then run official Stryker4s with (doesn't work yet):
#   cs launch io.stryker-mutator::stryker4s-command-runner:latest.release
# Run snapshot version:
#   cs launch io.stryker-mutator:stryker4s-command-runner_2.13:0.14.3+198-b81cc5eb-SNAPSHOT -r sonatype:snapshots -- --debug
# Check for more options:
#   https://stryker-mutator.io/docs/stryker4s/configuration/

stryker4s {
  # Change to the submodule you want to test
  # Ensure you turn off fatal warnings, as stryker4s adds such code that will generate warnings
  mutate: [ "my-submodule/**/main/scala/**/MyFoo.scala" ]
  scala-dialect: "2.12"
  # Use --test_env to enable Stryker4s modify prod's behavior (it augments prod code to be able to change its behavior with env vars).
  # https://bazel.build/reference/command-line-reference#flag--test_env
  # This is better than --action_env, as built artifacts do not have to change.
  test-runner: { command: "bazel", args: "test --test_env=ACTIVE_MUTATION -- //my-submodule:my_test" }
  static-tmp-dir: true
  debug {
    log-test-runner-stdout: true
    debug-test-runner: false
  }
  excluded-mutations: ["MethodExpression"]
}

[gergelyfabian]

Adding some follow up from Bazel Slack channel.

Instead of passing --test_env=ACTIVE_MUTATION to bazel, we can set up the test target with:

    env_inherit = [
        "ACTIVE_MUTATION",
    ],

And Stryker4s still works.
This enables potentially better usage of the Bazel cache.

[gergelyfabian]

Link to related rules_scala feature: bazel-contrib/rules_scala#1401

[hugo-vrijswijk]

Thanks for the extensive research and documentation here! I've been wanting to add a FAQ to the Stryker4s docs for a while, and I think this would be a great fit for it when I get around to it :)

[gergelyfabian]

Of course. I would be happy to also contribute to (at least review) any documentation changes. I guess I found also some bugs related to our usecase (index out of bounds exceptions and such), so maybe will also try to research those. Hugo van Rijswijk ***@***.***> ezt írta (időpont: 2022. dec. 6., K 20:15):

…

Thanks for the extensive research and documentation here! I've been wanting to add a FAQ to the Stryker4s docs for a while, and I think this would be a great fit for it when I get around to it :)

[gergelyfabian]

Updated example in #1261 (comment) after latest PR changes.

[gergelyfabian]

Updated example in #1261 (comment) after PRs got merged.

[YVbakker]

will be easier after #1522

[ittaiz]

Does stryker support bazel? Are there docs about it? (I found a small note about the property but not more then that)

[gergelyfabian]

Does stryker support bazel? Are there docs about it? (I found a small note about the property but not more then that)

I managed to make it work by a wrapper script that checks for Bazel targets and a specific config file, that then sets up stryker's config, and runs stryker with that.

Configuration template that I use is this:

test_target = "{TEST_TARGET}"
exclude_mutations = {EXCLUDE_MUTATIONS}

stryker4s {
  # Change to the submodule you want to test.
  # Ensure you turn off fatal warnings, as stryker4s adds such code that will generate warnings.
  mutate: {MUTATE}
  scala-dialect: "2.12"
  # Use --test_env to enable Stryker4s modify prod's behavior (it augments prod code to be able to change its behavior with env vars).
  # https://bazel.build/reference/command-line-reference#flag--test_env
  # This is better than --action_env, as built artifacts do not have to change.
  test-runner: { command: "bazel", args: "test --test_tag_filters=coverage --test_env=ACTIVE_MUTATION "${test_target} }
  # Set up a static tmp dir (where Stryker4s copies code) to optimize Bazel cache usage (the path counts into the workspace's name - it's a hash of the path).
  static-tmp-dir: true
  # Always leave tmp dir.
  #clean-tmp-dir: false
  # See Bazel output, to be sure it's running optimally.
  debug {
    log-test-runner-stdout: true
    debug-test-runner: false
  }
  excluded-mutations: ${exclude_mutations}
}