Skip to content

chore: add reproducible build verification script - #1684

Closed
basantagoswami wants to merge 16 commits into
developfrom
reproducibility
Closed

chore: add reproducible build verification script#1684
basantagoswami wants to merge 16 commits into
developfrom
reproducibility

Conversation

@basantagoswami

Copy link
Copy Markdown
Contributor

Adds a reproducibility testing script to the reproducibility directory. I took the script developed by xrviv for testing v6.2.3 from walletscrutiny.com/android/com.bullbitcoin.mobile, and did some modifications

  • Use fvm instead of Flutter directly
  • Used updated version of NDK
    etc

#1390 mentions an issue related to obfuscation, which has been fixed in a previous PR, and Rust binaries were already built deterministically. This script should give you complete reproducibility of the app found on Google Play

@basantagoswami
basantagoswami marked this pull request as ready for review December 19, 2025 01:52
@BullishNode

Copy link
Copy Markdown
Contributor

@claude review this PR

@claude

claude Bot commented Dec 19, 2025

Copy link
Copy Markdown

Claude finished @BullishNode's task in 53s —— View job


Reviewing PR #1684

  • Gather context and read changed files
  • Review README.md
  • Review verify_build.sh script
  • Provide detailed review feedback

Review Summary

This PR adds a comprehensive reproducible build verification script. The implementation is well-structured and follows WalletScrutiny's standards. Below are my findings:

✅ Strengths

  1. Robust error handling - Uses set -euo pipefail and a comprehensive on_error trap (lines 19, 22-42)
  2. Flexible container support - Supports both Docker and Podman (lines 65-74)
  3. Two verification modes - GitHub universal APK and device split APK comparison
  4. Memory checks - Warns users about insufficient RAM before resource-intensive builds (lines 101-137)
  5. Good documentation - The README and inline help are clear

⚠️ Issues and Suggestions

1. Typo in README.md (line 45)

- This script losely follows
+ This script loosely follows

2. Hardcoded image tag in docker build (line 965)

$CONTAINER_CMD build --memory=4g --no-cache --squash --ulimit nofile=65536:65536 -t bullbitcoin-verifier:v6.1.0 .

The tag v6.1.0 is hardcoded instead of using $appVersion. Should be:

-t bullbitcoin-verifier:v${appVersion} .

And the corresponding run commands (lines 987, 1039) should also use bullbitcoin-verifier:v${appVersion}.

3. Podman compatibility issue (line 965)

The --squash flag is a Docker-specific option. For Podman, consider:

if [[ "$CONTAINER_CMD" == "docker" ]]; then
    SQUASH_FLAG="--squash"
else
    SQUASH_FLAG=""
fi

4. Potential command injection risk (line 181)

The getSigner function uses $BASE directly in shell expansion:

apksigner verify --print-certs "$BASE"

While $BASE comes from basename, filenames with special characters could cause issues. The current usage is safe since APK files from known sources are used, but defensive coding would quote variables consistently.

5. Missing shellcheck directive

Consider adding # shellcheck shell=bash at the top for static analysis compatibility.

6. Apktool version inconsistency (lines 151-152 vs 932-933)

The containerApktool function downloads apktool v2.10.0, but the Dockerfile installs v2.9.3:

# containerApktool (line 152)
wget https://github.com/iBotPeaches/Apktool/releases/download/v2.10.0/apktool_2.10.0.jar

# Dockerfile (line 933)
sudo wget -q https://bitbucket.org/iBotPeaches/apktool/downloads/apktool_2.9.3.jar

This version mismatch could affect reproducibility verification.

7. README command mismatch (lines 14-24)

README shows ./verify_build but the script is verify_build.sh:

./verify_build --version <version>

Should be:

./verify_build.sh --version <version>

8. Interactive prompt in automated context (lines 127-132)

The read -p for memory warning could hang in CI/automated environments. Consider adding a --non-interactive flag.

9. Unused variable warning (line 362-363)

After containerApktool, the return code is checked with $?, but this should be captured immediately as other commands in between could overwrite it:

containerApktool "$tempExtractDir" "$apkDir/base.apk"
ret=$?
# ... (no commands in between)
if [ $ret -ne 0 ]; then

📝 Minor Suggestions

  1. Consider adding a --dry-run mode to preview what the script would do
  2. The script version v0.5.1 (line 283) could be extracted as a constant
  3. Consider adding SHA256 verification for downloaded tools (bundletool, apktool)

Overall Assessment

This is a well-written verification script that should serve its purpose well. The issues identified are mostly minor improvements rather than blocking problems. The core functionality for reproducible build verification appears sound.

Recommendation: Address items #2 (hardcoded version), #6 (apktool version mismatch), and #7 (README command name) before merging, as these could cause user confusion or incorrect behavior.


@basantagoswami

Copy link
Copy Markdown
Contributor Author

@i5hi

Getting the actual .env file used in production from the published APK is quite trivial. User just needs to download the APK from GitHub releases, unzip it, and the .env is available at unzipped_apk/assets/flutter_assets. I think it'll be better if we publish the actual values as part of .env.template itself to make it easier for people to reproduce the app

@ethicnology

Copy link
Copy Markdown
Member

Getting the actual .env file used in production from the published APK is quite trivial. User just needs to download the APK from GitHub releases, unzip it, and the .env is available at unzipped_apk/assets/flutter_assets. I think it'll be better if we publish the actual values as part of .env.template itself to make it easier for people to reproduce the app

@i5hi Rather than publishing the dotenv, because we may need to have some hidden values in the future anyway, we could get rid of the dotenv and replace it by migrating the values to the secure storage or even to an encrypted sqlite

Comment thread reproducibility/verify_build.sh Outdated
# Create Dockerfile
# =================

create_dockerfile() {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Creating the Dockerfile from a concatenated string is hard to maintain; we should update the existing Dockerfile in the repository instead.

@basantagoswami
basantagoswami marked this pull request as draft January 28, 2026 17:22
@basantagoswami

Copy link
Copy Markdown
Contributor Author

Converting it to draft, so that I can check the latest changes I am about to push before marking it ready for review

@basantagoswami
basantagoswami force-pushed the reproducibility branch 3 times, most recently from 08db1d0 to fcf43fd Compare January 30, 2026 15:14
@basantagoswami
basantagoswami marked this pull request as ready for review February 20, 2026 15:16
@ethicnology

Copy link
Copy Markdown
Member

@claude any suggestions on this PR ?

Comment thread reproducibility/verify_build.sh
@basantagoswami

Copy link
Copy Markdown
Contributor Author

@claude review once now, some things have been updated

@claude

claude Bot commented Mar 5, 2026

Copy link
Copy Markdown

Claude finished @basantagoswami's task in 1m 18s —— View job


Reviewing PR #1684 (Updated)

  • Gather context and read changed files
  • Check changes since last review
  • Review verify_build.sh script
  • Review Dockerfile changes
  • Provide detailed review feedback

Review Summary

The PR has improved significantly since the last review. Key improvements include:

  1. Separate verification tools Dockerfile - Addresses @ethicnology's concern about the embedded Dockerfile. The verification tools are now in reproducibility/Dockerfile
  2. Using root Dockerfile for builds - The script now uses the main Dockerfile for building, which is the right approach
  3. Consistent apktool version - Now using v2.10.0 consistently in the verification container
  4. Better structure - Script is much cleaner at ~450 lines vs the previous ~1000+ lines

✅ Addressed Issues from Previous Review

Previous Issue Status
Hardcoded image tag v6.1.0 ✅ Fixed - now uses v${appVersion}
Embedded Dockerfile as string ✅ Fixed - separate reproducibility/Dockerfile
apktool version mismatch ✅ Fixed - both use v2.10.0
Podman --squash compatibility ✅ Fixed - removed --squash flag

⚠️ Remaining Issues

1. Missing README

The reproducibility directory has no README. Users won't know:

  • Prerequisites (Docker/Podman, RAM, disk space)
  • How to extract split APKs from a device
  • Expected output format
  • Troubleshooting tips

Consider adding reproducibility/README.md.

2. Interactive prompt blocks CI/automation (verify_build.sh:96-98)

read -p "Continue? (y/N): " -n 1 -r
echo
[[ ! $REPLY =~ ^[Yy]$ ]] && exit 1

This will hang in automated/CI environments. Consider adding a --non-interactive or --yes flag:

if [[ "$interactive" == true ]]; then
    read -p "Continue? (y/N): " -n 1 -r
    echo
    [[ ! $REPLY =~ ^[Yy]$ ]] && exit 1
fi

3. Exit code check after container build is unreachable (verify_build.sh:312-315)

$CONTAINER_CMD build \
    --network=host \
    ...
    .

if [[ $? -ne 0 ]]; then
    echo -e "${RED}Build failed${NC}"
    exit 1
fi

With set -e at the top, if the build fails, the script will exit immediately before reaching this check. The conditional is unreachable dead code. Either remove it or use || true on the build command.

4. No SHA256 verification for downloaded tools (reproducibility/Dockerfile:18,25,34)

Downloaded binaries (apktool, SDK tools, bundletool) aren't verified:

RUN wget -q https://github.com/iBotPeaches/Apktool/releases/download/v2.10.0/apktool_2.10.0.jar -O /usr/local/bin/apktool.jar

For reproducibility tooling, consider adding checksum verification:

RUN wget -q https://... -O /tmp/apktool.jar && \
    echo "EXPECTED_SHA256  /tmp/apktool.jar" | sha256sum -c && \
    mv /tmp/apktool.jar /usr/local/bin/apktool.jar

5. Empty .env file may cause issues (root Dockerfile:107-108)

RUN cp .env.template .env

The discussion mentions .env values are needed for reproducibility. If .env.template has empty values, the build may differ from production. This relates to the earlier discussion about publishing actual values in .env.template.


📝 Minor Suggestions

  1. Add --version flag to script - Print script version for debugging
  2. Consider progress indicators - The 30-60 minute build could use echo progress updates
  3. Add cleanup for failed builds - Container images from failed verifications could accumulate

Architecture Notes

The approach is sound:

  • GitHub path: Downloads universal APK → builds APK → compares decoded contents
  • Device path: Uses split APKs → builds AAB → extracts matching splits → compares
  • Properly excludes META-INF (signatures) from comparison
  • Uses apktool for decoding which handles DEX/resources correctly

Overall Assessment

The restructuring addressed the major concerns. The remaining issues are mostly about polish and edge cases. The core verification logic is solid.

Recommendation: Consider addressing items #1 (README) and #2 (interactive prompt) before merging. The other items are nice-to-haves.


@basantagoswami

Copy link
Copy Markdown
Contributor Author

@claude please review again. things have changed again

@claude

claude Bot commented Mar 11, 2026

Copy link
Copy Markdown

Claude finished @basantagoswami's task in 1m 23s —— View job


Reviewing PR #1684 (Third Review)

  • Gather context and read changed files
  • Compare changes since last review
  • Review verify_build.sh script
  • Review Dockerfile changes
  • Review README.md
  • Provide detailed review feedback

Review Summary

Excellent progress! The PR has been substantially improved since the last review. The script is now much cleaner (~464 lines vs 1000+), and all major issues from previous reviews have been addressed.

Addressed Issues from Previous Reviews

Previous Issue Status
Missing README Fixed - comprehensive reproducibility/README.md added
Interactive prompt blocks CI Fixed - --yes flag added (line 56, 75, 277-281)
Hardcoded image tag Fixed - uses v${appVersion} dynamically
Embedded Dockerfile as string Fixed - separate reproducibility/Dockerfile
Apktool version mismatch Fixed - both use v2.10.0
Podman --squash compatibility Fixed - removed --squash flag
Unreachable exit code check Fixed - removed dead code

New Improvements

  1. Version extraction from APK (verify_build.sh:159-170) - When --apk is provided without --version, the script extracts the version from the APK itself

  2. Smart tag verification (verify_build.sh:95-107) - Tag check is correctly skipped when --apk is provided, only enforced for GitHub release verification

  3. cargokit_options.yaml - use_precompiled_binaries: false ensures Rust is built from source for reproducibility

  4. Root Dockerfile comments (Dockerfile:1-6) - Clear documentation of what is/isn't pinned

  5. Comprehensive README (reproducibility/README.md) - Explains SOURCE_DATE_EPOCH, CARGO_ENCODED_RUSTFLAGS, and includes device APK extraction instructions


Remaining Minor Items

1. TODO for SHA256 checksums (reproducibility/Dockerfile:14)

# TODO: verify SHA256 checksums of downloaded jars and SDK zip for supply-chain integrity

Not blocking, but worth implementing for defense-in-depth. The bundletool and apktool binaries are downloaded without verification.

2. Missing newline at end of cargokit_options.yaml (cargokit_options.yaml:8)

-use_precompiled_binaries: false
\ No newline at end of file
+use_precompiled_binaries: false

Minor style issue - POSIX files should end with a newline.

3. Potential edge case: grep with empty results (verify_build.sh:252)

apkUrl=$(echo "$releaseJson" | grep -o "https://github.com/..." | head -n1)

If the GitHub release JSON is malformed or the APK URL pattern changes, this grep returns empty. The script handles this (lines 254-257), but consider using jq for more robust JSON parsing if available in the tools image.

4. .env discussion from earlier comments

The discussion about .env.template vs publishing actual values remains open. The current approach (ENV_SOURCE=template) works for reproducibility testing but may diverge from production if .env.template has different values. This is a design decision for the team.


Architecture Assessment

The three-component design is well-structured:

  1. Root Dockerfile - Builds the app with reproducibility flags (SOURCE_DATE_EPOCH, CARGO_ENCODED_RUSTFLAGS)
  2. reproducibility/Dockerfile - Lightweight verification tools image (apktool, bundletool)
  3. verify_build.sh - Orchestrates everything, supports both GitHub and device/Play Store verification paths

The split APK handling (device-spec.json generation, ABI detection from filenames, split name normalization) is thorough and follows the bundletool conventions correctly.


Overall Assessment

This PR is ready for merge. The remaining items are minor polish rather than blocking issues. The reproducibility tooling is comprehensive, well-documented, and follows WalletScrutiny patterns appropriately.

Recommendation: Approve and merge. Consider addressing the cargokit newline and SHA256 checksums in a follow-up PR.


- Remove sudo entirely; all privileged setup done as root before USER switch
- Consolidate all apt installs into a single RUN block including openjdk-21-jdk
- Remove unused packages: zip, software-properties-common
- Replace wget with curl throughout
- Rename USER to APP_USER to avoid shadowing the standard Linux variable
- Move all ENV declarations to the top; derive HOME from APP_USER
- Fix RUSTFLAGS hardcoded /home/docker path to use $HOME
- Merge Rust install, verification, and rustup target adds into one RUN
- Merge Android SDK setup and sdkmanager calls into one RUN
- Copy directly into /app instead of staging directory
- Remove redundant fvm flutter clean on fresh clone
- Add ENV_SOURCE arg to control .env origin (template or local)
- Add FAKE_KEYSTORE arg with secret mount support for production signing
- Update makefile docker-build to expose all build args as overridable variables
…ld timestamp

OpenSSL embeds a wall-clock build timestamp in compiled binaries, causing .so differences between Docker builds run at different times. Setting SOURCE_DATE_EPOCH to the latest git commit timestamp fixes this in both the Dockerfile and the makefile build target.
… Dockerfile

Root Dockerfile
- Remove SOURCE/VERSION args; always build from local source
- Fix ENV_SOURCE=local branch which was a no-op (cp file to itself)

reproducibility/Dockerfile
- Add --platform=linux/amd64; upgrade Java 17 -> 21
- Replace wget with curl; drop wget from apt installs
- Remove apksigner and Android SDK setup — only used for informational output that never affected the verdict
- Merge apktool and bundletool RUN layers

reproducibility/verify_build.sh
- Move all validation (args, git tag check, required tools) before any work
- Add --yes flag to skip interactive prompts for CI/automation
- Replace wget with curl; replace grep -c "^" with wc -l
- Use container_name consistently; get commit hash from local git instead of spinning up a container
- Write RESULTS.md to workspace with verdict, metadata, and full diff
@ethicnology

Copy link
Copy Markdown
Member
  • Test on Linux (debian 13 with podman) fails at step 27
  • Test on MacOs (15.6.1 with podman) fails at installing rust at step 15

I'm trying to upgrade Mac to 26+ in order to see if it fixes the podman bug. Switching to arm64 will fail because flutter does not embed some binary for arm64

Linux logs:

STEP 27/27: RUN SOURCE_DATE_EPOCH=$(git -C /app log -1 --format=%ct) &&     CARGO_ENCODED_RUSTFLAGS=$(printf '%s\037%s\037%s'         "--remap-path-prefix=$HOME/.cargo=/cargo"         "--remap-path-prefix=$HOME/.rustup=/rustup"         "--remap-path-prefix=/app=/build") &&     export SOURCE_DATE_EPOCH CARGO_ENCODED_RUSTFLAGS &&     if [ "$FORMAT" = "aab" ]; then         fvm flutter build appbundle --${MODE};     else         fvm flutter build apk --${MODE};     fi
Downloading android-arm-profile/linux-x64 tools...               1,130ms
Downloading android-arm-release/linux-x64 tools...               1,008ms
Downloading android-arm64-profile/linux-x64 tools...               807ms
Downloading android-arm64-release/linux-x64 tools...               955ms
Downloading android-x64-profile/linux-x64 tools...                 981ms
Downloading android-x64-release/linux-x64 tools...                 989ms
Running Gradle task 'assembleDebug'...
Checking the license for package NDK (Side by side) 29.0.13113456 in /opt/android-sdk/licenses
License for package NDK (Side by side) 29.0.13113456 accepted.
Preparing "Install NDK (Side by side) 29.0.13113456 v.29.0.13113456 rc1".
"Install NDK (Side by side) 29.0.13113456 v.29.0.13113456 rc1" ready.
Installing NDK (Side by side) 29.0.13113456 in /opt/android-sdk/ndk/29.0.13113456
"Install NDK (Side by side) 29.0.13113456 v.29.0.13113456 rc1" complete.
"Install NDK (Side by side) 29.0.13113456 v.29.0.13113456 rc1" finished.
Checking the license for package NDK (Side by side) 27.0.12077973 in /opt/android-sdk/licenses
License for package NDK (Side by side) 27.0.12077973 accepted.
Preparing "Install NDK (Side by side) 27.0.12077973 v.27.0.12077973".
"Install NDK (Side by side) 27.0.12077973 v.27.0.12077973" ready.
Installing NDK (Side by side) 27.0.12077973 in /opt/android-sdk/ndk/27.0.12077973
"Install NDK (Side by side) 27.0.12077973 v.27.0.12077973" complete.
"Install NDK (Side by side) 27.0.12077973 v.27.0.12077973" finished.
Checking the license for package NDK (Side by side) 28.2.13676358 in /opt/android-sdk/licenses
License for package NDK (Side by side) 28.2.13676358 accepted.
Preparing "Install NDK (Side by side) 28.2.13676358 v.28.2.13676358".
"Install NDK (Side by side) 28.2.13676358 v.28.2.13676358" ready.
Installing NDK (Side by side) 28.2.13676358 in /opt/android-sdk/ndk/28.2.13676358
"Install NDK (Side by side) 28.2.13676358 v.28.2.13676358" complete.
"Install NDK (Side by side) 28.2.13676358 v.28.2.13676358" finished.
Checking the license for package Android SDK Platform 36 in /opt/android-sdk/licenses
License for package Android SDK Platform 36 accepted.
Preparing "Install Android SDK Platform 36 (revision 2)".
"Install Android SDK Platform 36 (revision 2)" ready.
Installing Android SDK Platform 36 in /opt/android-sdk/platforms/android-36
"Install Android SDK Platform 36 (revision 2)" complete.
"Install Android SDK Platform 36 (revision 2)" finished.

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':app:compileFlutterBuildDebug'.
> Failed to create parent directory '/workspaces' when creating directory '/workspaces/BULL/build/app/intermediates/flutter/debug/flutter_assets'

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
> Get more help at https://help.gradle.org.

BUILD FAILED in 5m
[=========                              ] 25%
Running Gradle task 'assembleDebug'...                            301.2s
Gradle task assembleDebug failed with exit code 1
Error: building at STEP "RUN SOURCE_DATE_EPOCH=$(git -C /app log -1 --format=%ct) &&     CARGO_ENCODED_RUSTFLAGS=$(printf '%s\037%s\037%s'         "--remap-path-prefix=$HOME/.cargo=/cargo"         "--remap-path-prefix=$HOME/.rustup=/rustup"         "--remap-path-prefix=/app=/build") &&     export SOURCE_DATE_EPOCH CARGO_ENCODED_RUSTFLAGS &&     if [ "$FORMAT" = "aab" ]; then         fvm flutter build appbundle --${MODE};     else         fvm flutter build apk --${MODE};     fi": while running runtime: exit status 1
make: *** [makefile:62: docker-build] Error 1

Mac logs:

Segmentation fault (core dumped)
Error: building at STEP "RUN curl -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain 1.77.2 &&     rustc --version && cargo --version &&     rustup target add aarch64-linux-android armv7-linux-androideabi x86_64-linux-android i686-linux-android": while running runtime: exit status 139

make: *** [docker-build] Error 139

@basantagoswami

Copy link
Copy Markdown
Contributor Author

Closing this PR because we have #2017 by Azad with my commits from this PR, and also #2018 (draft) that contains more changes by me on top of his branch.

@thibistaken
thibistaken deleted the reproducibility branch May 13, 2026 13:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants