[KYUUBI #7172] Shade kyuubi spark lineage module

[wForget]

Why are the changes needed?

To avoid the issues of kyuubi util classes not being found when placing lineage plugin jar in $SPARK_HOME/jars .

How was this patch tested?

run mvn clean -DskipTests package -am -pl extensions/spark/kyuubi-spark-lineage

logs:

[INFO] --- shade:3.5.2:shade (default) @ kyuubi-spark-lineage_2.12 ---
[INFO] Including org.apache.kyuubi:kyuubi-util-scala_2.12:jar:1.11.0-SNAPSHOT in the shaded jar.
[INFO] Including org.apache.kyuubi:kyuubi-util:jar:1.11.0-SNAPSHOT in the shaded jar.
[INFO] Excluding org.scala-lang:scala-library:jar:2.12.19 from the shaded jar.
[INFO] Excluding org.slf4j:slf4j-api:jar:1.7.36 from the shaded jar.
[INFO] Excluding org.apache.atlas:atlas-client-v2:jar:2.3.0 from the shaded jar.
[INFO] Excluding org.apache.atlas:atlas-intg:jar:2.3.0 from the shaded jar.
[INFO] Excluding commons-validator:commons-validator:jar:1.6 from the shaded jar.
[INFO] Excluding commons-beanutils:commons-beanutils:jar:1.9.2 from the shaded jar.
[INFO] Excluding commons-digester:commons-digester:jar:1.8.1 from the shaded jar.
[INFO] Excluding com.fasterxml.jackson.jaxrs:jackson-jaxrs-base:jar:2.15.4 from the shaded jar.
[INFO] Excluding com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:jar:2.15.4 from the shaded jar.
[INFO] Excluding com.fasterxml.jackson.module:jackson-module-jaxb-annotations:jar:2.15.4 from the shaded jar.
[INFO] Excluding jakarta.xml.bind:jakarta.xml.bind-api:jar:2.3.2 from the shaded jar.
[INFO] Excluding jakarta.activation:jakarta.activation-api:jar:1.2.2 from the shaded jar.
[INFO] Excluding javax.inject:javax.inject:jar:1 from the shaded jar.
[INFO] Excluding commons-configuration:commons-configuration:jar:1.10 from the shaded jar.
[INFO] Excluding org.apache.commons:commons-configuration2:jar:2.8.0 from the shaded jar.
[INFO] Excluding org.apache.atlas:atlas-client-common:jar:2.3.0 from the shaded jar.
[INFO] Excluding com.sun.jersey:jersey-client:jar:1.19 from the shaded jar.
[INFO] Excluding com.sun.jersey.contribs:jersey-multipart:jar:1.19 from the shaded jar.
[INFO] Excluding org.jvnet.mimepull:mimepull:jar:1.9.3 from the shaded jar.
[INFO] Excluding com.sun.jersey:jersey-core:jar:1.19 from the shaded jar.
[INFO] Excluding javax.ws.rs:jsr311-api:jar:1.1.1 from the shaded jar.
[INFO] Excluding cglib:cglib:jar:2.2.2 from the shaded jar.
[INFO] Excluding asm:asm:jar:3.3.1 from the shaded jar.
[INFO] Excluding commons-lang:commons-lang:jar:2.6 from the shaded jar.
[INFO] Dependency-reduced POM written at: /Users/wforget/work/git/kyuubi/extensions/spark/kyuubi-spark-lineage/dependency-reduced-pom.xml
[WARNING] kyuubi-spark-lineage_2.12-1.11.0-SNAPSHOT.jar, kyuubi-util-1.11.0-SNAPSHOT.jar, kyuubi-util-scala_2.12-1.11.0-SNAPSHOT.jar define 4 overlapping resources: 
[WARNING]   - META-INF/DEPENDENCIES
[WARNING]   - META-INF/LICENSE
[WARNING]   - META-INF/MANIFEST.MF
[WARNING]   - META-INF/NOTICE
[WARNING] maven-shade-plugin has detected that some files are
[WARNING] present in two or more JARs. When this happens, only one
[WARNING] single version of the file is copied to the uber jar.
[WARNING] Usually this is not harmful and you can skip these warnings,
[WARNING] otherwise try to manually exclude artifacts based on
[WARNING] mvn dependency:tree -Ddetail=true and the above output.
[WARNING] See https://maven.apache.org/plugins/maven-shade-plugin/
[INFO] Replacing original artifact with shaded artifact.
[INFO] Replacing /Users/wforget/work/git/kyuubi/extensions/spark/kyuubi-spark-lineage/target/kyuubi-spark-lineage_2.12-1.11.0-SNAPSHOT.jar with /Users/wforget/work/git/kyuubi/extensions/spark/kyuubi-spark-lineage/target/kyuubi-spark-lineage_2.12-1.11.0-SNAPSHOT-shaded.jar

Was this patch authored or co-authored using generative AI tooling?

No

[pan3793]

how about creating a new module with -shaded suffix, like what we did for authz?

[wForget]

how about creating a new module with -shaded suffix, like what we did for authz?

This is more like

kyuubi/extensions/spark/kyuubi-extension-spark-3-5/pom.xml

Lines 171 to 190 in 4c1412b

	<plugin>
	<groupId>org.apache.maven.plugins</groupId>
	<artifactId>maven-shade-plugin</artifactId>
	<configuration>
	<shadedArtifactAttached>false</shadedArtifactAttached>
	<artifactSet>
	<includes>
	<include>org.apache.kyuubi:*</include>
	</includes>
	</artifactSet>
	</configuration>
	<executions>
	<execution>
	<goals>
	<goal>shade</goal>
	</goals>
	<phase>package</phase>
	</execution>
	</executions>
	</plugin>

. We are only shading kyuubi-util into lineage plugin jar, and will not shade other dependencies (such as some services client).

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 0.00%. Comparing base (4c1412b) to head (7a5c453).
⚠️ Report is 97 commits behind head on master.

Additional details and impacted files

@@          Coverage Diff           @@
##           master   #7172   +/-   ##
======================================
  Coverage    0.00%   0.00%           
======================================
  Files         701     701           
  Lines       43525   43565   +40     
  Branches     5900    5911   +11     
======================================
- Misses      43525   43565   +40     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[pan3793]

[INFO] Excluding org.apache.atlas:atlas-client-v2:jar:2.3.0 from the shaded jar.
[INFO] Excluding org.apache.atlas:atlas-intg:jar:2.3.0 from the shaded jar.

just make sure, it will not cause runtime class not found issue when atlas is disabled?

[wForget]

@pan3793 Could you please take another look?"

[pan3793]

Thanks, merged to master