Skip to content

Commit e9fa9a3

Browse files
bryan-minimalbryan-minimal
authored
capstone, upx, fq: complete the reverse-engineering workbench (#549)
* capstone: package the disassembly framework 5.0.9 The disassembler engine underneath most open reverse-engineering tooling, packaged as a shared library so consumers can link it rather than vendor it. BUILD_SHARED_LIBS defaults to OFF upstream (and BUILD_STATIC_LIBS to ON), so a stock build ships libcapstone.a only. Anything linking that absorbs capstone statically, which puts a capstone CVE beyond pkgscan's reach entirely — nothing in the consumer's tree would name capstone at all. Shared on, static off, and the `ships_shared_lib` test pins it rather than trusting the flag. Tests disassemble real bytes across five architectures, including two fringe ones (riscv64, ppc64be), because CAPSTONE_ARCHITECTURE_DEFAULT gates every per-arch support option at once: turning it off yields a cstool that builds, runs, and reports the same version while silently having lost whole architectures. A deliberate no-such-arch case is included as a control, so the greps cannot pass on an error message. 5.0.9 rather than 6.0.0-Alpha10: the 6.0.0 alphas are tagged prerelease=false on GitHub but are alphas by name. Note rizin does NOT consume this — it pins capstone "next" (the unreleased v6) and vendors it, as recorded in that package. * upx: package the executable packer 5.2.0 The standard tool for unpacking UPX-compressed binaries, which is most of what an RE workbench wants it for. Three things worth a reviewer's attention: Vendored compressors are STRUCTURAL here, not a config choice. vendor/ carries ucl, lzma-sdk, zlib, zstd and bzip2, and the CMake build offers no find_package() or system-lib option at all — unlike rizin, where seventeen use_sys_* flags existed and flipping them was the whole job. It is not an oversight either: upx's output is read back by a decompression stub welded onto the packed executable, so compressor and stub must agree bit-for-bit. The consequence is stated rather than left to be discovered — those five libraries are invisible to pkgscan on this package, permanently. The source is not fetched with extract = true. Upstream compresses the -src tarball with an xz SHA-256 integrity check (xz --list reports Check: SHA-256, not the usual CRC64) and the fetcher's decoder rejects it: compression error: Unsupported SHA-256 checksum (not yet implemented) build.sh unpacks it with the system tar instead. runtime_deps carries libstdc++/libgcc beyond glibc — found by the checker, not guessed: with glibc alone even `upx --version-short` died with "error while loading shared libraries: libstdc++.so.6". The test packs a real binary, EXECUTES the packed copy and compares its output to the original, then unpacks and cmp's byte-for-byte. Nothing cheaper works: `upx --version` runs without touching a compressor, and `upx --best` will happily produce a file that is smaller and completely unrunnable if the per-architecture stub is wrong. * fq: package the binary-format query tool 0.17.0 jq for binaries: ~120 format decoders behind a jq expression language, which makes "what is in this file, structurally" scriptable across a corpus rather than a thing you do one file at a time in a hex editor. Built CGO_ENABLED=0 (upstream's own setting), so the result is a pure-Go static binary with no DT_NEEDED — hence the empty runtime_deps, same shape as helm. If CGO ever creeps back in, the missing-runtime_deps checker is what notices. No -X version stamping: fq keeps its version as a plain const in fq.go, and the Go linker only rewrites vars — a -ldflags -X aimed at a const is silently ignored, which is the kind of thing that looks maintained and does nothing. The test decodes rather than smoke-tests. `fq --version` and even `fq -n '1+1'` pass on a build whose format decoders are all broken, since the jq engine and the decoders are separate packages and only the decoders are the reason to ship this. So it asserts on a gzip's INFLATED payload — proving the DEFLATE decoder ran and returned the original bytes — not merely on the header field, and adds a second decoder (ELF) over a real file. The expected ELF machine is derived from uname rather than hardcoded, after an earlier package in this loadout pinned an architecture literal and only failed once it reached amd64 CI. * capstone, upx: record the aggregate licence, not the headline one Both packages declared a single licence where the shipped artifact carries more than one. The repo already models this — 54 packages use compound SPDX expressions — so these were simply wrong, not stylistically different. capstone: the tree ships LICENSE.TXT (BSD-3-Clause) AND LICENSE_LLVM.TXT for the decoders generated from LLVM's tables. My comment claimed that second file was "Apache-2.0 WITH LLVM-exception". It is not: capstone 5.x predates LLVM's relicensing and the file is the old LLVM Release License, i.e. University of Illinois/NCSA. Now (BSD-3-Clause AND NCSA), with the comment corrected. upx: was GPL-2.0-or-later alone, which omits the vendored code compiled into the binary. Established from the build rather than from the vendor/ listing, which turns out to overstate it in one direction and understate it in another: ucl GPL-2.0-or-later linked (own CMake target) zlib Zlib linked (own CMake target) lzma-sdk public domain linked INVISIBLY — compress_lzma.cpp #includes its .cpp files directly, so it never appears as its own objects in the ninja log doctest MIT linked the same way, via src/check/dt_impl.cpp; this release binary carries its test framework bzip2 — NOT linked; CMakeLists.txt:269 hard-disables it zstd — NOT linked; CMakeLists.txt:270 hard-disables it So (GPL-2.0-or-later AND Zlib AND MIT); the LZMA SDK is public domain and adds no conditions and no SPDX identifier. The earlier claim that five vendored libraries ride invisibly in this package was wrong in both directions — it is three, and two of the five named are not in the binary at all.
1 parent 3ccfc4f commit e9fa9a3

6 files changed

Lines changed: 481 additions & 0 deletions

File tree

‎packages/capstone/build.ncl‎

Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
let { standaloneTest, Attrs, BuildSpec, Local, OutputBin, OutputData, OutputLib, Source, Test, .. } = import "minimal.ncl" in
2+
let base = import "../base/build.ncl" in
3+
let cmake = import "../cmake/build.ncl" in
4+
let ninja = import "../ninja/build.ncl" in
5+
let pkgconf = import "../pkgconf/build.ncl" in
6+
let toolchain = import "../toolchain/build.ncl" in
7+
let glibc = import "../glibc/build.ncl" in
8+
let version = "5.0.9" in
9+
{
10+
name = "capstone",
11+
build_deps = [
12+
{ file = "build.sh" } | Local,
13+
base,
14+
cmake,
15+
ninja,
16+
pkgconf,
17+
toolchain,
18+
glibc,
19+
{
20+
url = "https://github.com/capstone-engine/capstone/archive/refs/tags/%{version}.tar.gz",
21+
sha256 = "0619da31af08152600af95c481527ef6d756c0a8404fca7544a4fdf6dfc2c0f9",
22+
extract = true,
23+
strip_prefix = "capstone-%{version}",
24+
} | Source,
25+
],
26+
cmd = "./build.sh",
27+
outputs = {
28+
cstool = { glob = "usr/bin/cstool" } | OutputBin,
29+
libs = { glob = "usr/lib/libcapstone.so*" } | OutputLib,
30+
includes = { glob = "usr/include/**" } | OutputData,
31+
pkgconfig = { glob = "usr/lib/pkgconfig/*.pc" } | OutputData,
32+
cmake_files = { glob = "usr/lib/cmake/**" } | OutputData,
33+
},
34+
runtime_deps = [glibc],
35+
attrs =
36+
{
37+
upstream_version = version,
38+
source_provenance = { category = 'GithubRepo, owner = "capstone-engine", repo = "capstone" },
39+
# Two licences, both permissive, and the tree ships both files:
40+
# LICENSE.TXT BSD-3-Clause (COSEINC)
41+
# LICENSE_LLVM.TXT University of Illinois/NCSA Open Source License,
42+
# covering the arch decoders generated from LLVM's
43+
# tables. This is the OLD "LLVM Release License" —
44+
# capstone 5.x predates LLVM's relicensing, so it is
45+
# NCSA here, NOT Apache-2.0 WITH LLVM-exception.
46+
# Recorded as the aggregate rather than the headline licence, since
47+
# NCSA's attribution clause travels with the binary too.
48+
license_spdx = "(BSD-3-Clause AND NCSA)",
49+
} | Attrs,
50+
tests = {
51+
smoketest = standaloneTest "/bin/cstool -v",
52+
53+
# DISASSEMBLE, don't just run --version.
54+
#
55+
# The failure this guards is a build that works and is quietly less
56+
# capable. `CAPSTONE_ARCHITECTURE_DEFAULT` gates every per-arch
57+
# `CAPSTONE_<ARCH>_SUPPORT` option, so changing one value drops whole
58+
# architectures out of the library while cstool still builds, still runs,
59+
# and still reports the same version. Nothing else here would notice:
60+
# `enumerate bins` sees a binary, `output types valid` sees a .so, a
61+
# smoke test sees a version string.
62+
#
63+
# So the assertions are real instructions on real bytes, including two
64+
# fringe architectures — the ones a "slim it down" change removes first,
65+
# and the ones an RE workbench misses last.
66+
disassembles =
67+
{
68+
class = 'Standalone,
69+
test_deps = [base],
70+
cmds = [
71+
# x86-64: push rbp / mov rbp, rsp — the universal function prologue.
72+
["/bin/bash", "-c", "cstool x64 '554889e5' | grep -qi push && cstool x64 '554889e5' | grep -qi mov"],
73+
# arm64: mul x1, x1, x2 — capstone's own test vector.
74+
["/bin/bash", "-c", "cstool arm64 '217c029b' | grep -qi mul"],
75+
# 32-bit ARM: a branch-with-link.
76+
["/bin/bash", "-c", "cstool arm 'edffffeb' | grep -qi bl"],
77+
# Fringe #1 — riscv64 (0x00000013 = nop).
78+
["/bin/bash", "-c", "cstool riscv64 '13000000' | grep -qiE 'nop|addi'"],
79+
# Fringe #2 — ppc64, BIG-endian. `ppc64` alone is CS_MODE_LITTLE_ENDIAN
80+
# in capstone 5 (cstool.c: "ppc64" => CS_MODE_64|CS_MODE_LITTLE_ENDIAN),
81+
# and 7c0802a6 is the big-endian encoding of `mflr r0` — the mode
82+
# name has to match the bytes or this decodes to garbage. Caught by
83+
# running it: the first version used `ppc64` and failed.
84+
["/bin/bash", "-c", "cstool ppc64be '7c0802a6' | grep -qi mflr"],
85+
# CONTROL: an architecture that does not exist must FAIL. Without
86+
# this, every grep above could be matching an error message rather
87+
# than disassembly, and the whole test would prove nothing.
88+
["/bin/bash", "-c", "! cstool no-such-arch 00 >/dev/null 2>&1"],
89+
],
90+
} | Test,
91+
92+
# The shared library is the reason to package this rather than let each
93+
# consumer vendor it. `BUILD_SHARED_LIBS` defaults to **OFF** upstream, so
94+
# a stock build produces libcapstone.a only — and anything linking that
95+
# absorbs capstone statically, making a capstone CVE invisible to pkgscan.
96+
# That default is one line in build.sh away from silently coming back.
97+
ships_shared_lib =
98+
{
99+
class = 'Standalone,
100+
test_deps = [base, pkgconf],
101+
cmds = [
102+
["/bin/bash", "-c", "test -e /usr/lib/libcapstone.so"],
103+
# A real versioned SONAME, not just the dev symlink.
104+
["/bin/bash", "-c", "ls /usr/lib/libcapstone.so.* >/dev/null 2>&1"],
105+
# ...and no static archive, so nobody links it by accident.
106+
["/bin/bash", "-c", "! test -e /usr/lib/libcapstone.a"],
107+
# pkg-config must resolve, or a downstream build quietly falls back
108+
# to its own vendored capstone.
109+
["/bin/bash", "-c", "PKG_CONFIG_PATH=/usr/lib/pkgconfig pkg-config --exists capstone"],
110+
],
111+
} | Test,
112+
},
113+
} | BuildSpec

‎packages/capstone/build.sh‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
#!/bin/sh
2+
set -eu
3+
4+
# Reproducibility flags (see AGENTS.md).
5+
export CFLAGS="${CFLAGS:-} -ffile-prefix-map=$(pwd)=/builddir -gno-record-gcc-switches"
6+
export CXXFLAGS="$CFLAGS"
7+
export LDFLAGS="${LDFLAGS:-} -Wl,--build-id=none"
8+
export ARFLAGS=Drc
9+
10+
# BUILD_SHARED_LIBS defaults to OFF and BUILD_STATIC_LIBS to ON upstream — the
11+
# opposite of what a distro wants. Left alone, this package would ship
12+
# libcapstone.a and every consumer would absorb capstone statically, which puts
13+
# a capstone CVE beyond pkgscan's reach: nothing in the consumer's tree would
14+
# name capstone at all. So shared on, static off, deliberately.
15+
#
16+
# CMAKE_INSTALL_LIBDIR=lib keeps the installed .pc and cmake files pointing at
17+
# usr/lib rather than the GNUInstallDirs 64-bit default lib64.
18+
#
19+
# CAPSTONE_ARCHITECTURE_DEFAULT=ON is the upstream default and is set here
20+
# explicitly because it is load-bearing: it gates every per-architecture
21+
# CAPSTONE_<ARCH>_SUPPORT option at once, and turning it off yields a working
22+
# cstool that silently cannot disassemble whole architectures. The
23+
# `disassembles` test pins the consequence rather than trusting this line.
24+
#
25+
# Tests off: upstream's suite needs its own fixtures and adds build time; the
26+
# standalone tests in build.ncl assert the properties we actually care about.
27+
cmake -S . -B build -G Ninja \
28+
-DCMAKE_INSTALL_PREFIX=/usr \
29+
-DCMAKE_INSTALL_LIBDIR=lib \
30+
-DCMAKE_BUILD_TYPE=Release \
31+
-DBUILD_SHARED_LIBS=ON \
32+
-DBUILD_STATIC_LIBS=OFF \
33+
-DCAPSTONE_BUILD_CSTOOL=ON \
34+
-DCAPSTONE_BUILD_TESTS=OFF \
35+
-DCAPSTONE_BUILD_CSTEST=OFF \
36+
-DCAPSTONE_ARCHITECTURE_DEFAULT=ON \
37+
-DCAPSTONE_X86_REDUCE=OFF
38+
39+
cmake --build build -j"$(nproc)"
40+
DESTDIR="$OUTPUT_DIR" cmake --install build

‎packages/fq/build.ncl‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
let { standaloneTest, Attrs, BuildSpec, Local, Needs, OutputBin, Source, Test, .. } = import "minimal.ncl" in
2+
let base = import "../base/build.ncl" in
3+
let go = import "../go/build.ncl" in
4+
let toolchain = import "../toolchain/build.ncl" in
5+
let version = "0.17.0" in
6+
{
7+
name = "fq",
8+
build_deps = [
9+
{ file = "build.sh" } | Local,
10+
base,
11+
go,
12+
toolchain,
13+
{
14+
url = "https://github.com/wader/fq/archive/refs/tags/v%{version}.tar.gz",
15+
sha256 = "c5658b2bc635a1d344c64e37d7311157f0fc4b20cc3cfa4d09bdd2f023692d57",
16+
extract = true,
17+
strip_prefix = "fq-%{version}",
18+
} | Source,
19+
],
20+
needs =
21+
{
22+
dns = {},
23+
internet = {},
24+
} | Needs,
25+
cmd = "./build.sh",
26+
outputs = {
27+
fq = { glob = "usr/bin/fq" } | OutputBin,
28+
},
29+
# Empty by design: built CGO_ENABLED=0, so the binary is pure-Go static with
30+
# no DT_NEEDED at all. Same shape as `helm`. If CGO ever creeps back in, the
31+
# `missing runtime_deps` checker is what notices.
32+
runtime_deps = [],
33+
attrs =
34+
{
35+
upstream_version = version,
36+
source_provenance = { category = 'GithubRepo, owner = "wader", repo = "fq" },
37+
license_spdx = "MIT",
38+
} | Attrs,
39+
tests = {
40+
smoketest = standaloneTest "/bin/fq --version",
41+
42+
# DECODE SOMETHING. `fq --version` and even `fq -n '1+1'` pass on a build
43+
# with every format decoder broken or absent: the jq engine is one Go
44+
# package and the ~120 format decoders are another, and only the decoders
45+
# are the reason to ship this tool.
46+
#
47+
# The gzip case is the sharp one — asserting on `compression_method`
48+
# alone would only prove fq read two header bytes. Asserting on
49+
# `uncompressed` proves it ran the DEFLATE decoder over the payload and
50+
# got the original bytes back, which is the actual capability.
51+
decodes =
52+
{
53+
class = 'Standalone,
54+
test_deps = [base],
55+
cmds = [
56+
[
57+
"/bin/bash",
58+
"-c",
59+
m%"
60+
set -eu
61+
msg="hello world hello world"
62+
printf '%s' "$msg" | gzip -n > /build/t.gz
63+
64+
# 1. The header field decodes symbolically, not as a raw number.
65+
m=$(fq -d gzip -r '.members[0].compression_method | tovalue' /build/t.gz)
66+
[ "$m" = "deflate" ] || { echo "compression_method=$m" >&2; exit 1; }
67+
68+
# 2. THE REAL ONE: fq inflated the payload and got our bytes back.
69+
got=$(fq -d gzip -r '.members[0].uncompressed | tostring' /build/t.gz)
70+
[ "$got" = "$msg" ] || {
71+
echo "inflate mismatch: '$got' != '$msg'" >&2; exit 1; }
72+
"%
73+
],
74+
[
75+
"/bin/bash",
76+
"-c",
77+
m%"
78+
set -eu
79+
# 3. A second, unrelated decoder over a real file: fq's own ELF.
80+
#
81+
# Derived from uname rather than hardcoded — an earlier package
82+
# in this loadout pinned an architecture literal and only failed
83+
# once it reached amd64 CI. fq renders EM_ARM64 as "arm64" and
84+
# EM_X86_64 as "x86_64" (format/elf/elf.go machineNames).
85+
case "$(uname -m)" in
86+
aarch64 | arm64) want=arm64 ;;
87+
x86_64) want=x86_64 ;;
88+
*) echo "unsupported arch $(uname -m)" >&2; exit 1 ;;
89+
esac
90+
got=$(fq -d elf -r '.header.machine | tovalue' /usr/bin/fq)
91+
[ "$got" = "$want" ] || {
92+
echo "elf machine: got '$got', want '$want'" >&2; exit 1; }
93+
"%
94+
],
95+
# 4. The jq engine itself still works. Last, because on its own it
96+
# proves the least.
97+
["/bin/bash", "-c", "test \"$(fq -n '1+1')\" = 2"],
98+
],
99+
} | Test,
100+
},
101+
} | BuildSpec

‎packages/fq/build.sh‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
#!/bin/sh
2+
set -eu
3+
export GOROOT=/usr/go
4+
5+
# CGO_ENABLED=0 is upstream's own build setting (see fq's Makefile) and is what
6+
# makes the result a pure-Go static binary with no DT_NEEDED — hence the empty
7+
# runtime_deps in build.ncl. Flipping this back on would silently add a libc
8+
# dependency that nothing here declares.
9+
export CGO_ENABLED=0
10+
11+
# Reproducibility (see AGENTS.md): -trimpath strips the build directory out of
12+
# recorded paths and -buildid= clears the non-deterministic build ID. -s -w
13+
# drop the symbol and DWARF tables, matching the other Go packages here.
14+
#
15+
# No -X version stamping: fq keeps its version as a plain const in fq.go, so
16+
# `fq --version` already reports 0.17.0 from the source tree. A -ldflags -X
17+
# aimed at a const would be silently ignored — the linker only rewrites vars.
18+
go build -trimpath -ldflags "-buildid= -s -w" -o fq .
19+
20+
mkdir -p "$OUTPUT_DIR/usr/bin"
21+
install -m 755 fq "$OUTPUT_DIR/usr/bin/fq"

0 commit comments

Comments
 (0)