Supply-chain hardening: Maven artifact provenance and dependency verification

[nzb-tuxxx]

Summary

This is not intended as a vulnerability report or an accusation. While packaging latest Arch AUR builds and working on sparrowwallet/lark#5 , I noticed a supply-chain hardening question around Maven artifacts containing native code.

I am opening this as a discussion/feedback issue to better understand the current guarantees and whether additional verification would be useful.

Observation

If I understand the build correctly, Sparrow/Lark currently depends on:

com.sparrowwallet:hid4java:0.8.1

which is resolved from the Sparrow Maven registry:

https://code.sparrowwallet.com/api/packages/sparrowwallet/maven/com/sparrowwallet/hid4java/0.8.1/hid4java-0.8.1.jar

This artifact contains prebuilt native libraries, for example:

linux-x86-64/libhidapi-libusb.so

As far as I can tell, the dependency is pinned by Maven version, but not by a committed artifact hash. I could not find Gradle dependency verification metadata, dependency lockfiles, or any equivalent hash pinning in the public Sparrow/Lark/Drongo source tree.

Why I think this matters

A fresh build appears to trust whatever the Maven repository serves for com.sparrowwallet:hid4java:0.8.1.

If an attacker ever managed to compromise code.sparrowwallet.com or the package registry and replace an existing JAR with a malicious one under the same version, a fresh source build would not detect this cryptographically. The native code from that JAR would then run inside the Sparrow process.

This reminded me, at a high level, of the broader lesson from the XZ Utils backdoor incident: native build artifacts deserve extra scrutiny, especially when they are consumed as prebuilt binaries.

Reproducibility angle

One thing I am unsure about is whether reproducible release checks fully cover this scenario.

For an already released version, replacing an existing Maven artifact would likely break reproducibility against the original release artifacts. That is good.

However, if I think like an attacker, the more attractive timing would be shortly before a new Sparrow release is built. If the Maven registry served a malicious hid4java-0.8.1.jar at that moment, and the official release was built from that input, then the new release could still be reproducible against the now-poisoned dependency set. The previous release might no longer be checked, and the new release would appear internally consistent.

So reproducibility helps detect after-the-fact changes to released inputs, but it may not by itself establish that the inputs used for a new release were the intended ones unless those inputs are also pinned or independently verified.

Possible hardening ideas

Some ideas that might reduce this risk:

• Commit Gradle dependency verification metadata, e.g. gradle/verification-metadata.xml, with SHA256 hashes for all resolved Maven artifacts.
• Ensure Maven artifacts are immutable once published, or document that immutability guarantee if it already exists.
• Publish the exact build scripts, source revisions, and build environment used to produce the com.sparrowwallet:* Maven artifacts, especially those containing native code.
• Consider building these artifacts in public CI, for example GitHub Actions, and publishing build logs/provenance alongside the artifacts.
• If older glibc compatibility is the reason for custom native builds, the public workflow could still build in an older Linux container or cross-build environment.
• Publish checksums or signatures for the Maven artifacts independently from the Maven repository itself.
• Prefer upstream artifacts such as org.hid4java:hid4java where possible, or document why the Sparrow-republished artifact intentionally differs.

I may be overthinking this, and I would be interested in your perspective.

From a packaging and downstream review point of view, this looks like an area where additional supply-chain hardening could be useful, especially because the affected artifacts contain native code that is loaded into the Sparrow process.

Or do you think the existing release and reproducibility process already provides enough practical protection for this threat model?

[craigraw]

I think the first thing I have to say is that provenance is a deep rabbit hole, and open source maintainers must balance their effort there with actual maintenance and development work. All sources are potentially corruptible, as we saw just last week with the Github hack.

What I already do is maintain a clean machine with the dependencies in cache. I then check that every Sparrow release built in CI is an exact binary replica of the same built on that machine. This is not perfect of course - nothing is - but it would certainly help in the "attractive timing" scenario you describe.

Commit Gradle dependency verification metadata, e.g. gradle/verification-metadata.xml, with SHA256 hashes for all resolved Maven artifacts.

This seems sensible and worth looking at as an additional "defense-in-depth" measure.

Publish the exact build scripts, source revisions, and build environment used to produce the com.sparrowwallet:* Maven artifacts, especially those containing native code.

I am not sure why you have highlighted com.sparrowwallet:* in terms of provenance, but ideally all the code is reproducible. That's the goal, and the Guix team are currently working on it. You can read about some of their progress here: #1835

If older glibc compatibility is the reason for custom native builds, the public workflow could still build in an older Linux container or cross-build environment.

Github CI no longer supports older Linux containers. Again, possible to solve, but resources are finite.

Prefer upstream artifacts such as org.hid4java:hid4java where possible, or document why the Sparrow-republished artifact intentionally differs.

There is no reason particularly why a native binary in a JAR published to Maven Central would be reproducible, or somehow of better provenance than one in a JAR published on code.sparrowwallet.com. In many cases the provenance is actually worse because the maintainer is no longer around help reproduce it.

Consider building these artifacts in public CI, for example GitHub Actions, and publishing build logs/provenance alongside the artifacts.

I agree, this is the long term goal - balanced of course against all the other needs of the project.

[nzb-tuxxx]

Thanks for the detailed feedback and for explaining how you currently approach release builds.

First, I want to make clear again that this was not intended as an attack on Sparrow, or on your work. My intention here is only to help improve areas where I see obvious room for additional hardening, and this seemed like one of those areas.

It is good to know that you maintain a clean machine with the relevant dependencies cached, and that every Sparrow release built in CI is checked to be an exact binary replica of the build produced on that machine. That makes the specific "attractive timing" scenario I described much less likely to go unnoticed than I had assumed.

That said, my broader concern is still that Sparrow is ultimately only as strong as the weakest link in the supply chain. In my view, that weak point is currently the dependency/build artifact supply chain — and by that I do not only mean code.sparrowwallet.com. It also includes other upstream sources, especially those distributing unauditable, non-reproducible binary code.

I highlighted code.sparrowwallet.com because that was the artifact source I noticed while looking into this specific case, but there are almost certainly others. Strictly speaking, all of them would need to be reviewed to get a complete picture. I also understand that this is difficult to practice consistently as a solo maintainer, especially when balanced against actual maintenance and development work.

There is no reason particularly why a native binary in a JAR published to Maven Central would be reproducible, or somehow of better provenance than one in a JAR published on code.sparrowwallet.com. In many cases the provenance is actually worse because the maintainer is no longer around help reproduce it.

I agree with that. I did not verify how those upstream artifacts are built. My hope was that some of them might be built reproducibly through public CI, with logs or provenance available, but of course that does not have to be the case. Simply being published to Maven Central does not automatically make an artifact more trustworthy.

So I think the most realistic framing here is probably not "Sparrow’s Maven registry is uniquely risky", but rather that native binary artifacts in the dependency graph are generally an area where additional defense-in-depth would be valuable.

From that perspective, committing Gradle dependency verification metadata still seems like a useful and relatively targeted improvement. It would not solve provenance by itself, but it would at least make unexpected artifact changes explicit and reviewable in the source tree.

I also agree that fully reproducible builds for all code and dependencies are the long-term ideal. Until that is practical, hash pinning / dependency verification seems like a pragmatic intermediate step that could improve the current situation without requiring the whole supply chain problem to be solved at once.

[nzb-tuxxx]

I did some local investigation for the verifying dependencies part using gradle/verification-metadata.xml

The gradle default is to skip Hash Pinning if a valid pgp signature was found to keep it simple. But I think we would benefit from having both enabled, as long as it's not to disturbing for your development workflow. In a later step you could also start pgp signing your com.sparrowwallet artifacts.

Maybe you can give it a try:

---

Reproducing the Gradle dependency verification files

The two files needed for hash pinning and pgp verification can be regenerated with Gradle only:

• gradle/verification-metadata.xml
• gradle/verification-keyring.keys

This follows Gradle’s official dependency verification workflow:
https://docs.gradle.org/current/userguide/dependency_verification.html

This setup intentionally verifies both provenance and integrity:

• PGP signatures are verified where upstream artifacts publish signatures.
• SHA512 checksums are pinned for all resolved artifacts, including unsigned artifacts and artifacts whose signing keys cannot be fetched.

Gradle documents that signature verification alone is optimistic during bootstrapping. Therefore, the metadata is generated in two passes: first SHA512 for all artifacts, then PGP plus SHA512.

Key servers

The bootstrap metadata explicitly configures the key servers instead of relying on Gradle’s built-in/default keyserver lookup. This is done because Gradle’s default keyserver lookup was unable to retrieve many of the required signing keys during bootstrapping. Using the two explicit HTTPS keyservers below made the bootstrap result much cleaner, reproducible, and documents exactly which public key sources were used:

• https://keyserver.ubuntu.com
• https://keys.openpgp.org

During bootstrapping, 2 signing keys still could not be downloaded by Gradle’s keyserver lookup:

• 94483BA5F4740C42
• E460DD8433123201

Gradle records those as ignored keys. The affected artifacts remain covered by SHA512 checksums.

Files to commit

Both generated files should be committed to the repository:

• gradle/verification-metadata.xml
• gradle/verification-keyring.keys

Committing the ASCII-armored keyring makes CI and developer builds less dependent on public keyserver availability. Gradle can verify signatures using the checked-in public keys instead of fetching them repeatedly from keyservers.

Only the ASCII-armored keyring is committed because keyring-format is set to armored. The binary gradle/verification-keyring.gpg file is intentionally not used.

Full regeneration from scratch

rm -f gradle/verification-metadata.xml \
      gradle/verification-keyring.keys \
      gradle/verification-keyring.gpg \
      gradle/verification-metadata.dryrun.xml

Create the initial gradle/verification-metadata.xml:

<?xml version="1.0" encoding="UTF-8"?>
<verification-metadata xmlns="https://schema.gradle.org/dependency-verification" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://schema.gradle.org/dependency-verification https://schema.gradle.org/dependency-verification/dependency-verification-1.3.xsd">
   <configuration>
      <verify-metadata>true</verify-metadata>
      <verify-signatures>true</verify-signatures>
      <keyring-format>armored</keyring-format>
      <key-servers>
         <key-server uri="https://keyserver.ubuntu.com"/>
         <key-server uri="https://keys.openpgp.org"/>
      </key-servers>
   </configuration>
   <components>
   </components>
</verification-metadata>

Generate checksums first:

./gradlew --write-verification-metadata sha512 --refresh-keys dependencies :drongo:dependencies :lark:dependencies

Then add PGP verification metadata while keeping SHA512 pinning:

./gradlew --write-verification-metadata pgp,sha512 --refresh-keys dependencies :drongo:dependencies :lark:dependencies

Export the public keys used during verification:

./gradlew --export-keys help

The help task is only used as a harmless Gradle task so that Gradle executes. The actual key export is triggered by --export-keys.

Validate the final result in strict mode:

./gradlew help dependencies :drongo:dependencies :lark:dependencies --dependency-verification strict --refresh-keys

Updating after dependency changes

After adding, removing, or updating dependencies, do not delete the files. Re-run the same two-pass update flow so Gradle incrementally adds missing metadata:

./gradlew --write-verification-metadata sha512 --refresh-keys dependencies :drongo:dependencies :lark:dependencies

./gradlew --write-verification-metadata pgp,sha512 --refresh-keys dependencies :drongo:dependencies :lark:dependencies

./gradlew --export-keys help

./gradlew help dependencies :drongo:dependencies :lark:dependencies --dependency-verification strict --refresh-keys

Review the resulting diff before committing. New entries generated by Gradle should be inspected, especially entries with reasons such as Artifact is not signed or A key couldn't be downloaded.

Build behavior

Once gradle/verification-metadata.xml is committed, Gradle dependency verification is active automatically for builds that resolve dependencies. The default verification mode is strict, so a normal build fails if a resolved artifact is missing verification metadata, has a wrong checksum, has an invalid signature, or is signed by an untrusted key.

Therefore, builds do not need to pass --dependency-verification strict explicitly. The flag is useful in validation commands or CI to make the intended mode obvious.

A normal build can be run as usual:

./gradlew build

[nzb-tuxxx]

As a first step I created a daily pipeline on Codeberg which uses a modified sparrow-wallet-git PKGBUILD which injects predefined verification files (verification-keyring.keys verification-metadata.xml) into the build to verify both (sha512 and pgp signature, where available).

I had to slightly modify the write-verification-metadata command to match on the actual jlink jpackageImage targets.

Lets see how this works over time, if we get errors for wrong, expired or invalid pgp keys or incorrect hash sums.

[craigraw]

Looks good. Two things:

1. I've published my key for toucan/hummingbird, so the number of ignored keys is down to 1. The remaining key will hopefully be published soon, see Publish PGP key mabe02/lanterna#666. It's not essential though, since we have the SHA512 hash pinning.
2. The two --write-verification-metadata passes will eventually need to be run on all 5 platforms to create a merged file, since the JavaFX libraries are platform specific.

As you suggest let's see how your daily pipeline works over time before adopting this here.

[nzb-tuxxx]

Thanks, I have added your missing key.

Good hint for the missing other platforms. If this ever gets adopted to this repo, do you think it is practical to maintain this file manually for all platforms? I was thinking about a github pipeline, which:

• generates a fresh verification-metadata.xml for all platforms one by one using a matrix build
• merges all entries to one single .xml
• compares this new file with the existing
• opens a pr with the new .xml in case of any changes

This way you not only save time for no need of doing this manually, but also get notified once a change is detected. Would that something you would consider using or do you prefer to update this file manually at the same time you bump the artifacts?

[craigraw]

I think a Github pipeline would be useful - ideally Gradle would do the merging since it supports that already, but I'm not sure how practical that is.