Skip to content

AGENTS.md

Latest commit

 

History

History
316 lines (256 loc) · 13.3 KB

AGENTS.md

File metadata and controls

316 lines (256 loc) · 13.3 KB
Raw

Agents Guide for ffi-converter

This document helps AI coding agents work effectively with the ffi-converter project. For the user-facing description of the tool, see README.md.

Project Overview

ffi-converter generates Go bindings for C libraries using libffi (no CGo). It parses C headers and outputs Go code.

Prime directive. The Go side is what matters: the correctness of this repo's source and the correctness and compilability of the code it emits. Exotic C parsing is not the goal. A construct that cannot be modelled faithfully is skipped whole and reported in Header.Diagnostics (parser) or Generator.Diagnostics() (generator), which main.go prints to stderr as warning: lines. A mis-parsed construct is a bug; a diagnosed skip is correct behaviour. Never "emit something plausible".

Do not repeat code. One source of truth per concept, table-driven tests.

Project Structure

.
├── main.go                 # CLI entry point, flag parsing, diagnostics
├── parser/
│   ├── types.go            # AST types + typedef resolution
│   ├── lex.go              # Tokenizer
│   ├── decl.go             # Shared declarator parser
│   └── parser.go           # Top-level declaration dispatch
├── generator/
│   ├── ctypes.go           # THE C type table (Go type, FFI type, size)
│   ├── names.go            # Identifier mangling and the name allocator
│   ├── enums.go            # C enum value semantics
│   ├── resolve.go          # CType -> Go/FFI mapping, per use context
│   └── generator.go        # Planning and emission
├── integration_test.go     # Compiles the generated bindings for every header
├── runtime_test.go         # Builds real C shared libs and calls through libffi
├── libname_test.go         # -lib derivation from the header path
├── testdata/
│   ├── *.h                 # Header corpus, one per class of defect
│   ├── csrc/               # C implementations for the runtime test
│   └── out/                # Committed output for calculator.h
└── go.mod                  # Tool has no non-stdlib dependencies

Build and Test Commands

# Build the tool (binary is ./ffi-converter)
go build .

# Run with a test header
./ffi-converter -header testdata/calculator.h -output testdata/out -package calculator -lib calculator

# Full end-to-end verification
go test ./...

testdata/out/ has no go.mod, so cd testdata/out && go build . cannot work — it fails with "no required module provides package github.com/jupiterrider/ffi". Do not add one. go test ./... is the real check:

  • integration_test.go runs the CLI over every header in testdata/, writes each result into a temp module with a pinned go.mod/go.sum, and really compiles it against the real github.com/jupiterrider/ffi and golang.org/x/sys. It also asserts every emitted file parses with go/parser and is gofmt-stable.
  • runtime_test.go goes further for the headers that have a C implementation in testdata/csrc/: it compiles a real shared library with the platform C compiler, generates the bindings, and runs a driver that calls every wrapper through libffi into that C code — plain and under -race. It skips when no C compiler is installed.

Type Mapping Reference

When modifying the generator, use these mappings:

C Type Go Type FFI Type
void (none) &ffi.TypeVoid
bool bool &ffi.TypeUint8
char / signed char int8 &ffi.TypeSint8
unsigned char / uint8_t uint8 &ffi.TypeUint8
short / int16_t int16 &ffi.TypeSint16
int / int32_t int32 &ffi.TypeSint32
unsigned int / uint32_t uint32 &ffi.TypeUint32
long int64 (+ portability diagnostic) &ffi.TypeSint64
unsigned long uint64 (+ portability diagnostic) &ffi.TypeUint64
long long / int64_t int64 &ffi.TypeSint64
unsigned long long / uint64_t uint64 &ffi.TypeUint64
size_t / uintptr_t uint / uintptr ffiTypeSizeT (chosen at init)
ssize_t / intptr_t / ptrdiff_t int ffiTypeSSizeT (chosen at init)
float float32 &ffi.TypeFloat
double float64 &ffi.TypeDouble
long double (unsupported, diagnosed) -
const char* string &ffi.TypePointer
char* / unsigned char* (param) []byte &ffi.TypePointer
char* (return) string &ffi.TypePointer
char** / const char** **byte &ffi.TypePointer
T** and deeper (not char**) *unsafe.Pointer &ffi.TypePointer
void* unsafe.Pointer &ffi.TypePointer
struct T* (param/return) *T &ffi.TypePointer
struct by value StructName &FFITypeStructName
enum named int type, width per member range matching &ffi.TypeSint32 / &ffi.TypeUint32 / &ffi.TypeSint64
opaque handle (typedef struct X_s* X) X (uintptr) &ffi.TypePointer
T[n] as a struct member [n]T element type repeated n times via ffiRepeat
any pointer in a struct member, any depth uintptr &ffi.TypePointer
T[n] as a parameter (incl. typedef'd) decays to the pointer mapping &ffi.TypePointer

The builtin part of this table lives in exactly one place in the code: cTypeTable in generator/ctypes.go. Position-dependent decisions (param vs return vs struct field, pointer flavour) live in generator/resolve.go. Do not add a second switch anywhere.

long/unsigned long cannot be represented portably by one static Go type, so the mapping above is kept and a diagnostic is emitted once per header rather than pretending it is portable. size_t/ssize_t/uintptr_t/intptr_t/ ptrdiff_t avoid the problem: their libffi type is a generated variable initialised at init from unsafe.Sizeof.

Enum width is chosen by enumUnderlying in generator/enums.go from the actual member value range: int64 when the range escapes [MinInt32, MaxUint32], uint32 when it exceeds MaxInt32 but is non-negative, int32 otherwise. It is not hardcoded to int32.

long double has no faithful Go representation and is kindUnsupported: any declaration mentioning it is skipped with a diagnostic.

Out-parameters must stay GC-visible. T** and deeper used to map to uintptr, which made the generated API impossible to call soundly: the only way to hand it Go memory was uintptr(unsafe.Pointer(&out)), and that is not a live reference, so a goroutine stack move during the call left C writing to a stale address. runtime.KeepAlive does not prevent stack copying. This lost writes reproducibly, about one run in three. A real Go pointer (**byte, *unsafe.Pointer) is tracked by the collector and updated by a stack move, so the pattern is now sound. Do not reintroduce a uintptr out-parameter.

The struct-field exception runs the other way: a Go struct that mirrors C memory must not hold GC-visible pointers, so every pointer field is uintptr at any depth. ctx == ctxField is checked first in resolvePointer for exactly that reason.

Critical FFI Patterns

Return values for integers < 8 bytes

Always use ffi.Arg and cast afterward:

var result ffi.Arg  // NOT int32
myFunc.Call(&result)
return int32(result)

The decision is made by the resolved size of the type, so enums and typedef'd scalars are covered too.

String parameters

Convert through the single generated mustCString helper — it panics on an embedded NUL rather than quietly handing C a nil pointer — and keep the result alive across the call:

namePtr := mustCString(name)
myFunc.Call(..., &namePtr)
runtime.KeepAlive(namePtr)

[]byte parameters use the companion bytePtr helper (nil for an empty slice) with the same runtime.KeepAlive. Both helpers are emitted at most once per package, only when something actually needs them; the import set is computed from what was emitted, so there are never unused imports.

String return values

var resultPtr *byte
myFunc.Call(&resultPtr)
if resultPtr == nil {
	return ""
}
return unix.BytePtrToString(resultPtr)

Struct by value

Pass struct pointer directly (not wrapped in unsafe.Pointer):

myFunc.Call(..., &config)  // config is a struct

Return buffers smaller than a register

libffi requires the rvalue buffer be no smaller than the system register size. ffi.Arg covers integers, bools and enums; struct-by-value and float32 returns cannot use it, so they are received into a padded wrapper instead of a bare variable:

var result struct {
	Val T
	_   uint64
}
myFunc.Call(&result.Val)

return result.Val

The address handed to Call is still &result.Val, so libffi writes at the start of a buffer that is guaranteed large enough. Do not "simplify" this back to var result T.

Verification Checklist

After making changes:

  1. go build ./...
  2. go vet ./...
  3. gofmt -l . (must print nothing)
  4. go test ./... (integration test compiles the generated bindings for every header; runtime test calls real C through libffi)
  5. Regenerate the committed sample and check it in: go run . -header testdata/calculator.h -output testdata/out -package calculator -lib calculator
  6. Read the warning: lines the tool prints for the corpus. A new warning means something stopped being supported; a missing warning next to missing output is a bug, because every skip must be reported.

Adding New C Type Support

  1. If the parser cannot see it, extend parser/decl.go; a construct that cannot be modelled faithfully must be skipped whole with a diagnostic, never half-parsed.
  2. Add ONE row to cTypeTable in generator/ctypes.go. There is no second switch to update.
  3. Add a header to testdata/ and a named regression case to the table tests in generator/generator_test.go.
  4. Run go test ./...: the integration test really compiles the generated bindings for every header in testdata/.

Common Issues

Parser misses a construct

The parser is a tokenizer (parser/lex.go) plus a declarator parser (parser/decl.go), not a C compiler, and there is no preprocessor and no #include following. It does descend into extern "C" { ... } and extern "C++" { ... } — the block form, the #ifdef __cplusplus idiom, the single-declaration form and nesting — so headers that wrap their whole API that way are read normally, not skipped.

Anything it cannot represent is skipped whole and reported in Header.Diagnostics, which main.go prints as warning: lines:

  • unions
  • bitfields
  • function-pointer members, and functions returning function pointers
  • anonymous nested structs/unions
  • multi-dimensional and non-constant-sized arrays
  • function definitions (static inline ...); only declarations translate
  • file-scope variable declarations
  • declarations whose shape comes from a macro, such as zlib's int foo OF((int a)); — resolving that needs a preprocessor

The generator adds its own skips for what the parser saw but cannot be emitted: variadic functions (libffi needs one CIF per call site), long double, and any declaration referring to a type that was itself skipped.

Diagnose and skip; never emit something that merely looks plausible. Warnings do not fail the run — main.go still exits 0 and writes everything it could translate.

Generated code doesn't compile

Check that:

  • FFI types match Go struct field order exactly
  • Pointer vs value semantics are correct
  • ffi.Arg is used for small integer returns

Wrong function signature

Check the emitted names: every identifier is allocated from one package-wide nameSet, so a clash shows up as a numeric suffix rather than as broken code. Names that would be illegal Go are repaired rather than emitted: a C name whose Go form would start with a digit is prefixed, so _0 becomes the type/function name X0 and the parameter name x0. Keyword collisions get a trailing underscore. Nothing should ever reach format.Source as a syntax error.

Duplicate declarations

Legal C, and what a broken include guard produces. Identical redeclarations of the same function collapse to one wrapper silently. A conflicting redeclaration keeps the first and emits warning: skipped a second, conflicting declaration of function "bar": the first one is used. Neither case may emit a phantom second wrapper.

Dependencies

The tool itself has no external dependencies beyond the standard library, so go.mod has no require block.

Generated code requires:

  • github.com/jupiterrider/ffi v0.7.0
  • golang.org/x/sys v0.28.0

These versions are pinned in integration_test.go (ffiModule, sysModule, plus a go.sum literal so the compile checks stay hermetic and offline), and that is the only place to change them.

Never downgrade ffi below v0.5.1. Earlier releases mirrored ffi_cif as one 32-byte struct on every architecture, but the aarch64 ABI adds FFI_EXTRA_CIF_FIELDS (aarch64_flags, aarch64_nfixedargs), making the real ffi_cif 40 bytes. On arm64 ffi_prep_cif wrote 8 bytes past the end of the Go allocation; once the neighbouring object was reused, ffi_call marshalled from corrupted state. Symptoms: float/double arguments arriving as zero (a bogus nfixedargs makes every argument variadic under Apple's ABI) and intermittent SIGSEGVs. v0.5.1 added cif_arm64.go. If runtime_test.go starts failing with zeroed floats on darwin/arm64, check this pin first.