Rewrite fails to load `classpath.tsv.gz` resources from additional GAVs of recipes when used as a library

[cmoulliard]

Issue

When we execute openrewrite part of a java quarkus application where the openrewrite GAVs have been loaded part of the AppClassLoader, then we hit a problem as rewrite during the execution of some recipes cannot load the classpath.tsv.gz of by example org.openrewrite.recipe:rewrite-java-dependencies:1.51.1 if we use an URLClassloader having as parent the AppClassloader.

The problem takes place when OpenRewrite runs a recipe like org.openrewrite.java.ReplaceAnnotation where they access on the JavaParser class the method classpathFromResources(Context,String ) able to search about such a resource.

As you can see hereafter the current code uses as ClassLoader the one having loaded the JavaParser class and not at all the URLClassloader created using the Environmen Builder and Environment.Collection<? extends ResourceLoader> resourceLoaders

// org.openrewrite.java.ReplaceAnnotation
a = JavaTemplate.builder(annotationTemplateToInsert)
                        .javaParser(classpathResourceName == null ?
        JavaParser.fromJavaVersion().classpath(JavaParser.runtimeClasspath()) :
        JavaParser.fromJavaVersion().classpathFromResources(ctx, classpathResourceName)) // HERE
...
// JavaParser class
public B classpathFromResources(ExecutionContext ctx, String... classpath) {
    this.artifactNames = emptyList();
    this.classpath = dependenciesFromResources(ctx, classpath);
    return (B) this;
}
...
static List<Path> dependenciesFromResources(ExecutionContext ctx, String... artifactNamesWithVersions) {
    if (artifactNamesWithVersions.length == 0) {
        return emptyList();
    }
    List<Path> artifacts = new ArrayList<>(artifactNamesWithVersions.length);
    Set<String> missingArtifactNames = new LinkedHashSet<>(Arrays.asList(artifactNamesWithVersions));
    List<String> availableArtifacts = new ArrayList<>();

    TypeTable typeTable = TypeTable.fromClasspath(ctx, missingArtifactNames);
    ...

// TypeTable class    
    public static @Nullable TypeTable fromClasspath(ExecutionContext ctx, Collection<String> artifactNames) {
        try {
            ClassLoader classLoader = findCaller().getClassLoader();
            // The ClassLoader we got it here is the AppClassLoader !!

FYI: I created a project showing the issue here: https://github.com/ch007m/tsv-classloading-issue

[cmoulliard]

As discussed on slack, the problem is not fixed at allow even if the change improves the situation !

This is not because by example the java class recipe ReplaceAnnotation, packaged part of the module rewrite-java, has been loaded part of the AppClassLoader that automatically we can assume that the classpath.tsv.gz that we are looking for  is available from this classloader as the classpathResourceName that we are searching (example: jakarta @inject is coming from some additional JARs provided by the user that we should load using a new URLClassLoader !

Here is what I suggest to do:

• Create a new ResourceLoader able to be used to scan the AppClassLoader + additionalJars loaded using an URLClassloader
• Add to the Environment class a new collection to define this TypesResourceLoader => private final Collection<? extends ResourceLoader> tsvResourceLoaders;
• Have a TsvResourcesClassPath scanning and loader

So, when openrewrite starts and when an Environment is created from the Fluent builder and build, the runtime Classpath and/or additionalJars() loaded should be scanned with the help of new the URLClassLoader to find ALL the classpath.tsv.gz resources, decompress them and create the classes locally under ~/.rewrite/classpath/.tt

When a recipe calls JavaParser.fromJavaVersion().classpathFromResources(ctx, classpathResourceName)), then access a new method able to find locally if the FQName of the class exists (= match) to got it. There is certainly a better option which is also to use a cache of the classes, etc

@timtebeek @knutwannheden

[timtebeek]

And to be clear: this is only an issue when using OpenRewrite as a library right? No such issues with the Maven/Gradle plugins?

[cmoulliard]

And to be clear: this is only an issue when using OpenRewrite as a library right? No such issues with the Maven/Gradle plugins?

Yes. When we use the maven plugin, the process enrich the URLClassLoader of the maven class - org.codehaus.plexus.classworlds.realm.ClassRealm during the merge step. As this is an URLClassLoader and not a ClassLoader, this is why we don't observe such an issue

public abstract class AbstractRewriteBaseRunMojo
...
            URLClassLoader recipeArtifactCoordinatesClassloader = getRecipeArtifactCoordinatesClassloader();
            if (recipeArtifactCoordinatesClassloader != null) {
                merge(getClass().getClassLoader(), recipeArtifactCoordinatesClassloader);
            }
            Environment env = environment(recipeArtifactCoordinatesClassloader);

    private void merge(ClassLoader targetClassLoader, URLClassLoader sourceClassLoader) {
        ClassRealm targetClassRealm;
        try {
            targetClassRealm = (ClassRealm) targetClassLoader;
        } catch (ClassCastException e) {
            getLog().warn("Could not merge ClassLoaders due to unexpected targetClassLoader type", e);
            return;
        }
        Set<String> existingVersionlessJars = new HashSet<>();
        for (URL existingUrl : targetClassRealm.getURLs()) {
            existingVersionlessJars.add(stripVersion(existingUrl));
        }
        for (URL newUrl : sourceClassLoader.getURLs()) {
            if (!existingVersionlessJars.contains(stripVersion(newUrl))) {
                targetClassRealm.addURL(newUrl);
            }
        }
    }

[cmoulliard]

The same issue also exists when we try to load a recipe which is packaged from an additional jar. The error will occur here when the classloader of the RecipeLoader class corresponds to the AppClassLoader and not the URLClassLoader having loaded the jar of the recipes !

/// https://github.com/openrewrite/rewrite/blob/main/rewrite-core/src/main/java/org/openrewrite/internal/RecipeLoader.java#L39-L41

public class RecipeLoader {

    private final @Nullable ClassLoader classLoader;
    private @Nullable ObjectMapper mapper;

    public RecipeLoader(@Nullable ClassLoader classLoader) {
        this.classLoader = classLoader;
    }

    public Recipe load(String recipeName, @Nullable Map<String, Object> recipeArgs) {
        try {
            Class<?> recipeClass = Class.forName(recipeName, true,
                    classLoader == null ? this.getClass().getClassLoader() : classLoader);
            return RecipeIntrospectionUtils.constructRecipe(recipeClass, recipeArgs == null ? emptyMap() : recipeArgs);
        } catch (ReflectiveOperationException | RecipeIntrospectionException | IllegalArgumentException e) {
            return instantiateRecipeUsingJackson(recipeName, recipeArgs == null ? emptyMap() : recipeArgs);
        }
    }

Error:

Caused by: org.openrewrite.RecipeException: Recipe(s) not found: io.moderne.prethink.UpdatePrethinkContextNoAiStarter                                                                                    
Did you mean: io.moderne.prethink.ComprehendCode                                                                                                                                                         
        at org.openrewrite.config.Environment.activateRecipes(Environment.java:161)                                                                                                                      
        at org.openrewrite.config.Environment.activateRecipes(Environment.java:173)                                                                                                                      
        at dev.snowdrop.rewrite.service.RewriteService.processRecipes(RewriteService.java:272) 

@knutwannheden

[timtebeek]

The recipe ComprehendCode is proprietary:

Moderne Proprietary. Only for use by Moderne customers under the terms of a commercial contract.

[cmoulliard]

The recipe ComprehendCode is proprietary:

I agree but my issue is not specific to this recipe but is a generic issue that we will have when the application starting openrewrite don't package in its pom.xml the rewrite module