sbt-scalafmt for sbt 2.x transitively pulls in scala-collection-compat_2.13

[eed3si9n]

Steps

sbt 2.0.0-RC10

project/plugins.sbt

addSbtPlugin("org.scalameta" % "sbt-scalafmt" % "2.5.6")
addSbtPlugin("ch.epfl.scala" % "sbt-scalafix" % "0.14.6")

Problem

$ sbt --server
[info] welcome to sbt 2.0.0-RC10 (Azul Systems, Inc. Java 17.0.12)
[info] loading project definition from /private/tmp/bbb/project
[error] Modules were resolved with conflicting cross-version suffixes in ProjectRef(uri("file:/private/tmp/bbb/project/"), "bbb-build"):
[error]    org.scala-lang.modules:scala-collection-compat _3, _2.13
[error] java.lang.RuntimeException: Conflicting cross-version suffixes in: org.scala-lang.modules:scala-collection-compat

Expectation

We can use these plugins together.

[kitbellew]

@tgodzik this looks like the sourcecode problem in scalameta. anyone know how to solve it once and for all? perhaps @jchyb does, as he's done the lion's share converting the parser to _3.

[eed3si9n]

I think this is more to do with:

1. sbt-scalafmt using 2.13 version of scalafmt-dynamic, which on its own is okay, and
2. scalafmt-dynamic pulling in 2.13 version of Coursier, which pulls in a bunch of other 2.13 libraries, including scala-collection-compat:

$ cs resolve --tree org.scalameta:sbt-scalafmt_sbt2_3:2.5.6 | rg collection
  Result:
   │  │  │  │  └─ org.scala-lang.modules:scala-collection-compat_2.13:2.12.0
   │  │  │  │  └─ org.scala-lang.modules:scala-collection-compat_2.13:2.12.0

Potential workarounds might be:

1. provide an implementation of scalafmt-dynamic that can use sbt's Coursier and/or lm interface
2. stuff coursier into src/main/resource (the approach sbt-native-image seems to take)
3. make an über JAR of Coursier (the approach sbt takes) or the whole scalafmt-dynamic

[kitbellew]

1. provide an implementation of scalafmt-dynamic that can use sbt's Coursier and/or lm interface

this i don't know how to do. -dynamic determines the version that it needs and loads it, by obtaining some java.net.URLs from coursier (which downloads it first if it's missing) and passing to URLClassLoader.

can sbt download the dependencies first (although that i also don't know how to do, from within the plugin) and provide a list of those local URLs?

2. stuff coursier into src/main/resource (the approach sbt-native-image seems to take)

which means that you're ok that sbt build will have two versions of coursier, judging by this comment?

3. make an über JAR of Coursier (the approach sbt takes) or the whole scalafmt-dynamic

how do i do that?

[eed3si9n]

using update?

can sbt download the dependencies first (although that i also don't know how to do, from within the plugin) and provide a list of those local URLs?

sbt 1.x and 2.x codebase have a shaded version of Coursier and lm interface. If the plugin can do the config file parsing, you can inject libraryDependencies scoped to a sandbox configuration, and grab scalafmt via update like it's a normal dependency.

shelling out to Coursier CLI

which means that you're ok that sbt build will have two versions of coursier, judging by sbt/sbt#9010 (comment)?

By stuffed coursier I mean Coursier CLI. That would be executed as a forked process, so I guess it's yet another version of Couriser, but they aren't going to be in the same JVM and cause classpath conflict on scala-collection-compat etc.

shading

how do i do that?

Here's what we do to shade Coursier (https://github.com/sbt/sbt/blob/v2.0.0-RC10/build.sbt#L1353-L1360)

lazy val lmCoursierShaded = project
  .settings(
    libraryDependencies ++= Seq(coursier etc),
    assemblyShadeRules := ....
  )

lazy val lmCoursierShadedPublishing = project
  .in(file("lm-coursier/target/shaded-publishing-module"))
  .settings(
    scalaVersion := scala3,
    name := "librarymanagement-coursier",
    Compile / packageBin := (lmCoursierShaded / assembly).value,
    Compile / exportedProducts := Seq(Attributed.blank((Compile / packageBin).value))
  )

[kitbellew]

using update?

can sbt download the dependencies first (although that i also don't know how to do, from within the plugin) and provide a list of those local URLs?

sbt 1.x and 2.x codebase have a shaded version of Coursier and lm interface. If the plugin can do the config file parsing, you can inject libraryDependencies scoped to a sandbox configuration, and grab scalafmt via update like it's a normal dependency.

i can't use sbt syntax but can use the sbt Scala code accessible from the plugin. the plugin uses scalafmt via an interface declared in scalafmt-interfaces (this part is in Java) and implemented in scalafmt-dynamic.

i am planning to inject a downloader interface so that coursier can be completely extracted but i can't figure out how to implement in the plugin; llm suggested accessing dependencyResolution.value and invoking update on it.

however, i can't figure out:

• how to get a module descriptor with all the dependencies i want to download (or do i do them one at a time?) but without a project that goes with them
• how to take advantage of caching within sbt
• and, most importantly, where do i get the jars of the dependencies that were resolved? i need a class loader using them, i was going to use URLClassLoader as usual; UpdateReport is provided but I don't see how to extract from it this information.

[eed3si9n]

See https://github.com/coursier/sbt-coursier/blob/main/modules/lm-coursier/src/main/scala/lmcoursier/CoursierDependencyResolution.scala#L69-L95 for example.

most importantly, where do i get the jars of the dependencies that were resolved?

You can either use managedJars function in https://github.com/sbt/sbt/blob/1.12.x/main/src/main/scala/sbt/Defaults.scala#L4200-L4212 and adjust it to work on both sbt 1.x and 2.x. Or use the CoursierDependencyResolution example code:

    val jars =
      for {
        reportConfiguration <- report.configurations.filter(_.configuration.name == "runtime")
        module <- reportConfiguration.modules
        (_, jar) <- module.artifacts
      } yield jar

how to take advantage of caching within sbt

To make it work on sbt 1.x and 2.x, you can use sbt 1.x style caching. See around https://www.scala-sbt.org/1.x/docs/Caching.html#Tracking+file+attributes in the docs.

how to get a module descriptor with all the dependencies i want to download

I think you can make something up like com.example:sandbox:0.1.0.

    val fakeModule =
      ModuleDescriptorConfiguration(
        ModuleID("com.example", "sandbox", "0.1.0"),
        ModuleInfo("sandbox")
      )
      .withDependencies(...) // your dependencies here

[kitbellew]

@eed3si9n thank you for that. perhaps you can help me with another problem.

when running scripted tests in this repo with scala 3.7.4, requesting scalafmt for scala 2.13 (specifically: org.scalameta:scalafmt-core_2.13:2.2.1 and org.scala-lang:scala-reflect:2.13.18), sometimes among the jars returned in UpdateReport are scala library and scala reflect for 2.12.21 or 2.10.7 (depending on the test, I am guessing, but not 2.13.18 as requested), and when i wrap all these jar files in URLClassLoader, it fails with ReflectiveOperationException. llm thinks that sbt might be injecting these scala deps, but if that's the case, i don't know how to avoid that.

when i run the same scripted tets with scala 2.12.21, they all seem to pass.

for reference, here's the code so far:

import java.lang as jl
import sbt.internal.librarymanagement.InternalDefaults.{modulePrefixTemp, sbtOrgTemp}
import sbt.{librarymanagement as lm, richUpdateReport}

...

  override def download(
      version: String,
      reporter: ScalafmtReporter,
      dependencies: jl.Iterable[RepositoryPackage],
  ): jl.Iterable[File] = {
    val name = s"${modulePrefixTemp}scalafmt-${version}_$scalaBinaryVersion"
    val moduleDependencies = dependencies.asScala
      .map(d => lm.ModuleID(d.group, d.artifact, d.version)).toVector
    val module = dependencyResolution.moduleDescriptor(
      lm.ModuleDescriptorConfiguration(
        lm.ModuleID(sbtOrgTemp, name, "1.0"),
        lm.ModuleInfo("sandbox"),
      ).withDependencies(moduleDependencies),
    )

    dependencyResolution.update(
      module,
      updateConfiguration,
      lm.UnresolvedWarningConfiguration(),
      taskStreams.log,
    ).fold(
      x => throw x.resolveException,
      x => x.allFiles.asJava,
    )
  }

[eed3si9n]

I'm not 100% sure, but you might need to call .withScalaModuleInfo(...) on your sandbox module, if there's some Scala mixup that's happening?

https://github.com/sbt/sbt/blob/v1.12.8/main/src/main/scala/sbt/Defaults.scala#L3514

    ModuleDescriptorConfiguration(projectID.value, projectInfo.value)
      .withValidate(ivyValidate.value)
      .withScalaModuleInfo(scalaModuleInfo.value)
      .withDependencies(deps)

sbt does inject standard libraries when autoScalaLibrary is true etc, but not at this layer.

[kitbellew]

I'm not 100% sure, but you might need to call .withScalaModuleInfo(...) on your sandbox module, if there's some Scala mixup that's happening?

https://github.com/sbt/sbt/blob/v1.12.8/main/src/main/scala/sbt/Defaults.scala#L3514

ModuleDescriptorConfiguration(projectID.value, projectInfo.value)
  .withValidate(ivyValidate.value)
  .withScalaModuleInfo(scalaModuleInfo.value)
  .withDependencies(deps)

sbt does inject standard libraries when autoScalaLibrary is true etc, but not at this layer.

alas... if I use dependencyResolution.value (which is the only thing I can access), it comes preconfigured, and both autoScalaLibrary and, equally importantly, the scala version that is hardcoded into the object (ignoring scalaModuleInfo) apply:

https://github.com/coursier/sbt-coursier/blob/main/modules/lm-coursier/src/main/scala/lmcoursier/CoursierDependencyResolution.scala#L142

https://github.com/coursier/sbt-coursier/blob/main/modules/lm-coursier/src/main/scala/lmcoursier/CoursierDependencyResolution.scala#L238

if my guess is correct in that sbt is using sbt-coursier unchanged, then that's the library that needs to be changed.

[eed3si9n]

Interesting. Could you open a draft PR with what you have so I can reference it from an issue plz?