feat(marshal,pass-style): admit immutable ArrayBuffer through codecs#3226
Closed
kriskowal wants to merge 2 commits into
Closed
feat(marshal,pass-style): admit immutable ArrayBuffer through codecs#3226kriskowal wants to merge 2 commits into
kriskowal wants to merge 2 commits into
Conversation
🦋 Changeset detectedLatest commit: abc1010 The changes in this PR will be included in the next version bump. This PR includes changesets to release 16 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Member
Author
|
Submitted in a draft state as it will need to be refactored atop @erights work to transition from immutable ArrayBuffer to Uint8Array backed by ArrayBuffer to the extent that can be emulated and usage patterns exist that can transparently straddle implementation with and without support immutable array buffers. |
|
Refreshed in
Salvaged from #2871: The @erights — would appreciate your take on whether you want this to drive forward as the codec-admission landing point, or whether you'd rather rebase #2871 on top of |
This was referenced Apr 30, 2026
erights
reviewed
May 1, 2026
Comment on lines
+17
to
+18
| - **smallcaps**: byteArray encodes as `"*<hex>"`. The `*` prefix is now | ||
| reserved. |
Contributor
There was a problem hiding this comment.
We already reserved *. This PR no longer reserves it but instead uses it. I think this is just a terminology difference. This PR overall is certainly clear that it was reserved.
| * * `$` - remotable | ||
| * * `&` - promise | ||
| * | ||
| * All other special characters (`"'()*,`) are reserved for future use. |
Contributor
There was a problem hiding this comment.
This means * was already reserved.
| }); | ||
| const { body } = serialize(mkByteArray([0xde, 0xad, 0xbe, 0xef])); | ||
| // smallcaps body has a leading `#` sentinel before the JSON text. | ||
| t.true(body.includes('"*deadbeef"'), `got ${body}`); |
Contributor
There was a problem hiding this comment.
prefer test for exact text using slice(1) or whatever.
Contributor
|
From the PR description.
|
|
Refreshed Applied:
Deliberately not propagated:
Tests: |
erights
reviewed
May 2, 2026
| @@ -10,6 +10,8 @@ import { | |||
| isErrorLike, | |||
Contributor
There was a problem hiding this comment.
For this encodePassable.js file, I defer to @gibson042 . My review should not be taken to include this file.
erights
reviewed
May 2, 2026
| @@ -1,7 +1,9 @@ | |||
| import harden from '@endo/harden'; | |||
Contributor
There was a problem hiding this comment.
Just noting: This is the key file to change when rebasing on top of #3164 . I will postpone reviewing this file until then.
erights
reviewed
May 2, 2026
| @@ -0,0 +1,49 @@ | |||
| import test from '@endo/ses-ava/test.js'; | |||
Contributor
There was a problem hiding this comment.
Likewise, I will postpone reviewing this file until rebased on #3164
Immutable `ArrayBuffer` (passStyle `'byteArray'`) is now serializable
through the capdata, smallcaps, and encode-passable codecs:
- `@endo/pass-style` gains `byteArrayToHex`, `hexToByteArray`,
`byteArrayToUint8Array`, and `uint8ArrayToByteArray` helpers, all
routed through `@endo/hex`.
- `@endo/marshal`:
- capdata: `byteArray` encodes as
`{"@qclass":"byteArray","data":"<hex>"}`.
- smallcaps: `byteArray` encodes as `"*<hex>"`; the `*` prefix is
now reserved.
- encode-passable: `byteArray` encodes as
`a<encodeBigInt(byteLength)>:<hex>`. The Elias-delta length
prefix gives shortlex ordering matching `compareRank` with no
arbitrary size cap.
- marshal-justin: renders byteArray as `hexToByteArray("<hex>")`.
Syrup already supported this value through its bytestring primitive;
CBOR remains out of scope until `packages/ocapn/src/cbor` lands.
…codecs Apply the relevant feedback from #2871 to subsume that PR: - @gibson042 #2871:byteArray.js:127 — `uint8ArrayToByteArray` now honors the source `Uint8Array`'s `byteOffset` and `length`, so a view over a slice of a larger buffer round-trips correctly: `uint8ArrayToByteArray(new Uint8Array(buf, byteOffset, length))` represents exactly the bytes in `[byteOffset, byteOffset + length)`. Documented the zero-copy / copy tradeoff with `sliceToImmutable`. - @gibson042 #2871:byteArray.js:130 — comment on `byteArrayToHex` / `hexToByteArray` is now phrased as "Like `encodeHex` but…" / "Like `decodeHex` but…", matching gibson's suggested wording for the byteArray-flavored wrappers. - @erights #2871:encodeToCapData.js:453 / encodeToSmallcaps.js — the factory functions `makeDecodeFromCapData` and `makeDecodeFromSmallcaps` are now `harden`-ed at module scope, matching the existing harden of their encoder counterparts (`makeEncodeToCapData`, `makeEncodeToSmallcaps`). - @erights #2871:marshal-test-data.js — propagate Mark's full byteArray fixture set: `roundTripPairs`, `jsonJustinPairs`, `unsortedSample`, and `sortedSample`. The Justin test compartment exposes `hexToByteArray` so the rendered Justin expression evaluates. Not propagated, deliberately: - `packages/marshal/src/rankOrder.js` — Mark's `compareRankRemotablesTied` refactor is orthogonal to codec admission and lands as its own PR (bots #73 today, awaiting an upstream PR). - `packages/pass-style/src/hex.js` — Mark vendored a private hex implementation in pass-style; we route through the merged `@endo/hex` package (#3208) instead. As a result, `packages/pass-style/index.js` exports `byteArrayToHex` / `hexToByteArray` but not `uint8ArrayToHex` / `hexToUint8Array`; consumers needing raw hex helpers should `import { encodeHex, decodeHex } from '@endo/hex'`. Tests: `@endo/marshal` 79 + 1 skip, `@endo/pass-style` 29. Lint clean.
kriskowal
force-pushed
the
kriskowal-marshal-binary
branch
from
2d5dd2a to
abc1010
Compare
kriskowal
added a commit
that referenced
this pull request
May 15, 2026
Refs: #2871 Refs: #2883 Refs: #3226 ## Description Splits the rank-comparison refactor out from #2871 so that the codec-admission core of that PR can land in #3226 without depending on this orthogonal change. Introduces `compareRankRemotablesTied` (and its anti-rank twin), which behaves like `compareRank` but treats all remotables as tied for the same rank rather than short-circuiting on the first remotable encountered. Makes this the default `compare` for the rank-cover operations: - `isRankSorted`, `assertRankSorted`, `sortByRank`, `getIndexCover` (commit 1) - `unionRankCovers`, `intersectRankCovers` (commit 2) `compare` becomes optional and is moved to the trailing position on each. `getIndexCover`'s argument order shifts from `(sorted, compare, rankCover)` to `(sorted, rankCover, compare?)`; the only callers are within the Endo monorepo and are updated. `unionRankCovers` and `intersectRankCovers` likewise shift from `(compare, covers)` to `(covers, compare?)`, and `@endo/patterns`'s `patternMatchers.js` (the only caller) is updated. `compareRankRemotablesTied` and `compareAntiRankRemotablesTied` are re-exported from `@endo/marshal`'s `index.js`. The first commit is salvaged from #2871 with original authorship preserved (Mark S. Miller). ### Security Considerations No change to trust boundaries, no new authorities. The comparator change is internal to marshal's rank ordering; callers that pass an explicit `compare` keep their existing behavior. ### Scaling Considerations The tied comparator visits every remotable rather than short-circuiting on the first, so rank-cover operations over inputs containing many remotables do more work per call than the prior `compareRank` default. This matches the intent of the operations (they want a stable cover, not an early exit), and the prior behavior is still available by passing `compareRank` explicitly. ### Documentation Considerations The argument-order changes on `getIndexCover`, `unionRankCovers`, and `intersectRankCovers` are caller-visible. No internal callers remain on the old signatures. External callers, if any, would need to swap argument order; this is captured in the changeset. ### Testing Considerations `packages/marshal/test/rankOrder.test.js` is extended to: - Exercise the new optional/default-arg behavior of all six rank-cover operations. - Resolve two pre-existing TODOs that asked for the `totally orders ranks` and `is transitive` property tests to run against both `compareRank` and `compareRankRemotablesTied`. Both are total preorders, so both properties hold for both, and we now parameterize the tests to exercise each. ### Compatibility Considerations Argument-order shifts on three exported functions (`getIndexCover`, `unionRankCovers`, `intersectRankCovers`) and one new default comparator on six. Captured in a changeset. The `compareRankRemotablesTied` and `compareAntiRankRemotablesTied` exports are additions; nothing is removed. ### Upgrade Considerations No live-system upgrade concerns. The changeset covers the package-level migration for external consumers.
Merged
boneskull
pushed a commit
that referenced
this pull request
May 27, 2026
This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and the packages will be published to npm automatically. If you're not ready to do a release yet, that's fine, whenever you add more changesets to master, this PR will be updated. # Releases ## @endo/bytes@1.0.0 ### Major Changes - [#3257](#3257) [`dd45f4a`](dd45f4a) Thanks [@kriskowal](https://github.com/kriskowal)! - Add `@endo/bytes` package providing `concatBytes`, `bytesEqual`, `bytesFromText`, `bytesToText`, `bytesToImmutable`, and `bytesFromImmutable` for platform-neutral byte handling. Built on `Uint8Array` with `TextEncoder`/`TextDecoder` captured once at module load. `bytesToImmutable` and `bytesFromImmutable` cross between mutable views and the immutable `ArrayBuffer` shape produced by the proposal-immutable-arraybuffer shim, so passable bytes can be carried across vat boundaries. Hardened, SES-safe; usable across Node, XS, and browser realms. The release is the first publish, going out as `1.0.0` from the `0.1.0` workspace floor (major bump from a `0.x.y` baseline lands at `1.0.0`). ## @endo/base64@1.1.0 ### Minor Changes - [#3216](#3216) [`7325bbe`](7325bbe) Thanks [@kriskowal](https://github.com/kriskowal)! - `@endo/base64`'s named exports (`encodeBase64`, `decodeBase64`, `atob`, `btoa`) are now frozen. Consumers that previously assigned to or extended these exports will see a `TypeError` under SES; read-only consumers are unaffected. The shim entry point `@endo/base64/shim.js` (which `@endo/init/pre.js` uses to install `globalThis.atob` / `globalThis.btoa` before `lockdown()`) is unchanged and continues to be safe to load pre-lockdown. ### Patch Changes - [#3216](#3216) [`7325bbe`](7325bbe) Thanks [@kriskowal](https://github.com/kriskowal)! - `encodeBase64` and `decodeBase64` now dispatch to the native `Uint8Array.prototype.toBase64` and `Uint8Array.fromBase64` intrinsics (TC39 proposal-arraybuffer-base64, Stage 4) on platforms that ship them, falling back to the existing pure-JavaScript implementation otherwise. Existing call sites pick up the speedup with no API change, and `decodeBase64`'s error messages (including any caller-supplied `name` and the failing offset) are unchanged. ## @endo/compartment-mapper@2.2.0 ### Minor Changes - [#3150](#3150) [`75253ad`](75253ad) Thanks [@boneskull](https://github.com/boneskull)! - This adds a new option, `additionalLocations`, to `mapNodeModules`. This option enables addition of arbitrary packages to the resulting Compartment Map. It enables execution of packages which need to dynamically load files from Compartments which would otherwise not appear in the map. Specific use-cases include tooling like Webpack and ESLint. ### Patch Changes - [#2901](#2901) [`69ca27c`](69ca27c) Thanks [@kriskowal](https://github.com/kriskowal)! - Sweep `makeFunctorFromMap` in `bundle.js` and `bundle-lite.js` to use optional chaining and nullish coalescing for the alias lookup now that [#1514](#1514) has completed. The "unable to locate module" diagnostic now distinguishes the alias-resolved case from the direct-key case, naming both the alias's resolved key and the original import key when the miss occurs via an alias. - [#3201](#3201) [`67ed1ce`](67ed1ce) Thanks [@boneskull](https://github.com/boneskull)! - Compatibility fix for passthrough-style wildcards in subpath exports. - Updated dependencies \[[`ad7a177`](ad7a177), [`f2aa55a`](f2aa55a), [`fa0b6a9`](fa0b6a9), [`459347b`](459347b)]: - @endo/hex@1.1.0 - ses@2.1.0 ## @endo/eslint-plugin@2.5.0 ### Minor Changes - [#3255](#3255) [`638306e`](638306e) Thanks [@kriskowal](https://github.com/kriskowal)! - Migrate the bundled `@endo/imports` ESLint config off the unmaintained `eslint-plugin-import` and onto the actively-maintained `eslint-plugin-import-x` soft fork. This is done via a Yarn package alias (`eslint-plugin-import: 'npm:eslint-plugin-import-x@4.16.2'` in the `dev` catalog), so the package on disk is still named `eslint-plugin-import` and ESLint continues to register its rules under the existing `import/*` namespace. The import-x implementation ships its own `unrs-resolver`, which natively honours the `package.json` `exports` field, so the explicit `import/resolver` settings block is no longer required and has been removed. Downstream consumers do not need to rename any `import/*` rule references; existing `eslintrc` snippets continue to work. ### Patch Changes - [#3274](#3274) [`e153a5a`](e153a5a) Thanks [@kriskowal](https://github.com/kriskowal)! - `harden-exports` now collects export names from all binding pattern shapes that may appear on the left-hand side of `export const ... = ...`: aliased object destructuring (`{ propName: aliasName }`), object and array rest, nested patterns, sparse holes, and default-value assignment patterns. A new `unknownBindingPattern` report surfaces any pattern type the helper does not recognize, in place of silent passthrough. ## @endo/evasive-transform@2.2.0 ### Minor Changes - [#3131](#3131) [`5c098c4`](5c098c4) Thanks [@naugtur](https://github.com/naugtur)! - - Add `customVisitor` option - a visitor function to be called on each node of the Babel traverse, in addition to the standard transforms in evasive-transform. Receives the same path argument as a normal Babel visitor. - Add an evasion for a class or object method being named `import` or `eval` ## @endo/hex@1.1.0 ### Minor Changes - [#3208](#3208) [`ad7a177`](ad7a177) Thanks [@kriskowal](https://github.com/kriskowal)! - Add `@endo/hex` package providing `encodeHex` and `decodeHex` as a ponyfill for the TC39 `Uint8Array.prototype.toHex` and `Uint8Array.fromHex` intrinsics (proposal-arraybuffer-base64, Stage 4). Dispatches to the native intrinsics at module load when available, falling through to a portable pure-JavaScript implementation elsewhere. ## @endo/marshal@1.10.0 ### Minor Changes - [#3265](#3265) [`45d06cd`](45d06cd) Thanks [@kriskowal](https://github.com/kriskowal)! - Add `compareRankRemotablesTied` and `compareAntiRankRemotablesTied`, which behave like `compareRank` and `compareAntiRank` except that they treat all remotables as tied for the same rank instead of short-circuiting the comparison on encountering a remotable. Make the `compare` parameter optional on `isRankSorted`, `assertRankSorted`, `sortByRank`, `getIndexCover`, `unionRankCovers`, and `intersectRankCovers`, defaulting to `compareRankRemotablesTied`. The following parameter orders change so the optional comparator lands at the end: - `getIndexCover(sorted, compare, rankCover)` → `getIndexCover(sorted, rankCover, compare?)` - `unionRankCovers(compare, covers)` → `unionRankCovers(covers, compare?)` - `intersectRankCovers(compare, covers)` → `intersectRankCovers(covers, compare?)` These are breaking changes to helpers that have no known external callers within the Endo monorepo. Within the monorepo, the only callers of `unionRankCovers` and `intersectRankCovers` are in `@endo/patterns`'s `patternMatchers.js`, which has been updated to match the new parameter order. Salvaged from <#2871> so that the codec-admission core of that PR (now in [#3226](#3226)) can land independently of this rank-comparison refactor. ## @endo/ocapn@1.1.0 ### Minor Changes - [#3256](#3256) [`bdb9ddc`](bdb9ddc) Thanks [@kriskowal](https://github.com/kriskowal)! - - Add a `framing` option to `makeTcpNetLayer` (`@endo/ocapn/netlayer/tcp-testing`). The default is `'syrup'`, which wraps each message in the `<length>:<payload>` framing implemented by `@endo/syrup-frame`. This is the framing the OCapN TCP-for-testing netlayer is moving toward (cf. the 2025-12-09 OCapN plenary, <https://github.com/ocapn/ocapn/blob/main/meeting-minutes/2025-12-09.md>), and is robust to TCP chunk boundaries that split a single OCapN message. Pass `framing: 'none'` to interoperate with the existing `ocapn/ocapn-test-suite` Python `testing_only_tcp` netlayer, which writes a syrup-encoded record with `sendall` and reads one back with `syrup.syrup_read` (no length prefix on the wire). The `'none'` option exists only for that suite's sake and goes away once the suite either adopts syrup framing or is retired. - [#3192](#3192) [`08b077d`](08b077d) Thanks [@kumavis](https://github.com/kumavis)! - Sync `@endo/ocapn` with [ocapn-test-suite](https://github.com/ocapn/ocapn-test-suite) at commit [74db78f08a40efba1e2b975d809374ff0e7acf60](ocapn/ocapn-test-suite@74db78f) (2026-02-25). - GC operations use list payloads (`exportPositions` / `wireDeltas`, `answerPositions`); wire labels `op:gc-exports` and `op:gc-answers`. - Remove `op:deliver-only`; fire-and-forget delivery uses `op:deliver` with `answerPosition` and `resolveMeDesc` set to `false`. - Codec refactors: `makeOcapnFalseForOptionalCodec` for optional `false` branches, homogeneous Syrup lists via `makeListCodecFromEntryCodec`, and related cleanup in operations and peer location hints. CI integration for the Python test suite is pinned to the same commit. - [#3209](#3209) [`20f9e21`](20f9e21) Thanks [@kumavis](https://github.com/kumavis)! - - Add a WebSocket netlayer exported as `@endo/ocapn/netlayer/ws` (`makeWebSocketNetLayer`). Used for interop with Guile-Goblins peers and for any other transport that prefers a framed WebSocket over the raw TCP test netlayer. - Add `@endo/ocapn/netlayer/tcp-testing` to the package's `exports` map so consumers can import the existing test netlayer without reaching into `src/`. - The main entry (`@endo/ocapn`) now re-exports `makeClient` and the swissnum helpers `swissnumFromBytes` / `swissnumToBytes` so consumers don't need a deep `src/client/...` import for the common case. - `makeClient` accepts a new `logger` option; when omitted the existing console-based logger is used, so this is backwards-compatible. - The CapTP version-mismatch log on `start-session` now includes both the received and expected version strings. ### Patch Changes - Updated dependencies \[[`ad7a177`](ad7a177), [`dd45f4a`](dd45f4a), [`45d06cd`](45d06cd), [`38fe678`](38fe678)]: - @endo/hex@1.1.0 - @endo/bytes@1.0.0 - @endo/marshal@1.10.0 - @endo/syrup-frame@0.1.1 ## ses@2.1.0 ### Minor Changes - [#3284](#3284) [`f2aa55a`](f2aa55a) Thanks [@boneskull](https://github.com/boneskull)! - Allows `Map.prototype.getOrInsert`, `Map.prototype.getOrInsertComputed`, `WeakMap.prototype.getOrInsert` and `WeakMap.prototype.getOrInsertComputed`. ### Patch Changes - [#3223](#3223) [`fa0b6a9`](fa0b6a9) Thanks [@gibson042](https://github.com/gibson042)! - Addresses infidelities and inefficiencies of [#3214](#3214) - [#3214](#3214) [`459347b`](459347b) Thanks [@erights](https://github.com/erights)! - Fixes bug [#3202](#3202) repairing momentary breakage of setFloat16 and setFloat32 ## @endo/bundle-source@4.3.1 ### Patch Changes - [#3261](#3261) [`7309d69`](7309d69) Thanks [@turadg](https://github.com/turadg)! - Replace `ts-blank-space` with Amaro for TypeScript type erasure, matching the parser Node.js uses for type stripping. - [#3216](#3216) [`7325bbe`](7325bbe) Thanks [@kriskowal](https://github.com/kriskowal)! - `@endo/base64`'s named exports (`encodeBase64`, `decodeBase64`, `atob`, `btoa`) are now frozen. Consumers that previously assigned to or extended these exports will see a `TypeError` under SES; read-only consumers are unaffected. The shim entry point `@endo/base64/shim.js` (which `@endo/init/pre.js` uses to install `globalThis.atob` / `globalThis.btoa` before `lockdown()`) is unchanged and continues to be safe to load pre-lockdown. - [#3237](#3237) [`b845f85`](b845f85) Thanks [@kriskowal](https://github.com/kriskowal)! - Update `BundleOptions` and the `powers` parameter types to reflect what `bundleSource` already accepts at runtime: `cacheSourceMaps` is now documented on `BundleOptions`, `commonDependencies` and `importHook` (only for the `endoZipBase64` format) are surfaced, and the optional `powers` parameter accepts the wider `BundlePowers` shape (including `pathResolve`, `userInfo`, `env`, `platform`, and `computeSha512`). This is a JSDoc/typings-only change; no runtime behavior changes. - Updated dependencies \[[`7325bbe`](7325bbe), [`7325bbe`](7325bbe), [`69ca27c`](69ca27c), [`5c098c4`](5c098c4), [`67ed1ce`](67ed1ce), [`75253ad`](75253ad)]: - @endo/base64@1.1.0 - @endo/compartment-mapper@2.2.0 - @endo/evasive-transform@2.2.0 ## @endo/captp@4.5.1 ### Patch Changes - [#2901](#2901) [`69ca27c`](69ca27c) Thanks [@kriskowal](https://github.com/kriskowal)! - Sweep `makeFinalizingMap`'s `get` operator to use optional chaining (`keyToRef.get(key)?.deref()`) now that [#1514](#1514) has completed, replacing the prior explicit conditional. Behavior is unchanged. - Updated dependencies \[[`45d06cd`](45d06cd)]: - @endo/marshal@1.10.0 ## @endo/patterns@1.9.1 ### Patch Changes - [#3265](#3265) [`45d06cd`](45d06cd) Thanks [@kriskowal](https://github.com/kriskowal)! - Add `compareRankRemotablesTied` and `compareAntiRankRemotablesTied`, which behave like `compareRank` and `compareAntiRank` except that they treat all remotables as tied for the same rank instead of short-circuiting the comparison on encountering a remotable. Make the `compare` parameter optional on `isRankSorted`, `assertRankSorted`, `sortByRank`, `getIndexCover`, `unionRankCovers`, and `intersectRankCovers`, defaulting to `compareRankRemotablesTied`. The following parameter orders change so the optional comparator lands at the end: - `getIndexCover(sorted, compare, rankCover)` → `getIndexCover(sorted, rankCover, compare?)` - `unionRankCovers(compare, covers)` → `unionRankCovers(covers, compare?)` - `intersectRankCovers(compare, covers)` → `intersectRankCovers(covers, compare?)` These are breaking changes to helpers that have no known external callers within the Endo monorepo. Within the monorepo, the only callers of `unionRankCovers` and `intersectRankCovers` are in `@endo/patterns`'s `patternMatchers.js`, which has been updated to match the new parameter order. Salvaged from <#2871> so that the codec-admission core of that PR (now in [#3226](#3226)) can land independently of this rank-comparison refactor. - Updated dependencies \[[`45d06cd`](45d06cd)]: - @endo/marshal@1.10.0 ## @endo/syrup-frame@0.1.1 ### Patch Changes - [#3256](#3256) [`38fe678`](38fe678) Thanks [@kriskowal](https://github.com/kriskowal)! - - New package `@endo/syrup-frame`: a sibling of `@endo/netstring` that drops the trailing `,` separator, so each framed payload on the wire is literally a Syrup byte-string record (`<length>:<payload>`). - Provides `makeSyrupReader` and `makeSyrupWriter` with the same shape as the netstring equivalents, including the `chunked` zero-copy writer mode. - Intended for testing interoperability with the OCapN TCP-for-testing protocol, which has moved to adopt this framing. The framing is among the protocols under consideration in the OCapN pre-standards group at time of writing; the 2025-12-09 OCapN plenary recorded the consensus that the TCP-for-testing netlayer should carry messages as length-prefixed Syrup byte strings (<https://github.com/ocapn/ocapn/blob/main/meeting-minutes/2025-12-09.md> and the parallel discussion on ocapn/ocapn#104). ## @endo/daemon@3.0.0 ### Major Changes - [#3204](#3204) [`291e224`](291e224) Thanks [@gibson042](https://github.com/gibson042)! - Detect truncation of Unix domain socket paths ### Patch Changes - [#3171](#3171) [`c372670`](c372670) Thanks [@turadg](https://github.com/turadg)! - TypeScript 6 conformance: public types in `types.d.ts` are now more precise. - `Context.thisDiesIfThatDies` and `thatDiesIfThisDies` parameters tightened from `string` to `FormulaIdentifier` (a branded string type). - `RemoteControl.accept`/`connect` and `RemoteControlState.accept`/`connect` take `remoteGateway: ERef<EndoGateway>` (was `Promise<EndoGateway>`); `RemoteControl.connect` now returns `ERef<EndoGateway>` (was `Promise<EndoGateway>`); `RemoteControlState.connect` returns `{ state; remoteGateway: ERef<EndoGateway> }`. - `EndoInspector` generic type parameter renamed from `Record` to `RecordT` to avoid shadowing the built-in `Record` utility type; `lookup` and `list` now use method syntax so that `EndoInspector<'some literal'>` remains assignable to `EndoInspector<string>` under `strictFunctionTypes`. TypeScript consumers implementing or calling these interfaces may need to update their types accordingly. - Updated dependencies \[[`ad7a177`](ad7a177), [`7325bbe`](7325bbe), [`7325bbe`](7325bbe), [`f2aa55a`](f2aa55a), [`69ca27c`](69ca27c), [`fa0b6a9`](fa0b6a9), [`69ca27c`](69ca27c), [`dd45f4a`](dd45f4a), [`45d06cd`](45d06cd), [`67ed1ce`](67ed1ce), [`459347b`](459347b), [`75253ad`](75253ad)]: - @endo/hex@1.1.0 - @endo/base64@1.1.0 - ses@2.1.0 - @endo/captp@4.5.1 - @endo/compartment-mapper@2.2.0 - @endo/bytes@1.0.0 - @endo/marshal@1.10.0 - @endo/patterns@1.9.1 ## @endo/cli@2.3.13 ### Patch Changes - Updated dependencies \[[`ad7a177`](ad7a177), [`7309d69`](7309d69), [`7325bbe`](7325bbe), [`b845f85`](b845f85), [`c372670`](c372670), [`291e224`](291e224), [`dd45f4a`](dd45f4a), [`45d06cd`](45d06cd)]: - @endo/hex@1.1.0 - @endo/bundle-source@4.3.1 - @endo/daemon@3.0.0 - @endo/bytes@1.0.0 - @endo/patterns@1.9.1 Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
gibson042
pushed a commit
to gibson042/endo
that referenced
this pull request
Jun 2, 2026
Salvaged from endojs#2871 so that the codec-admission core of that PR (now in endojs#3226) can land without depending on this orthogonal refactor. - Add `compareRankRemotablesTied` and `compareAntiRankRemotablesTied`, which behave like `compareRank` / `compareAntiRank` but consider all remotables tied for the same rank instead of short-circuiting on encountering one. - Make the `compare` parameter optional on `isRankSorted`, `assertRankSorted`, `sortByRank`, and `getIndexCover`, defaulting to `compareRankRemotablesTied`. Reorder `getIndexCover`'s parameters from `(sorted, compare, rankCover)` to `(sorted, rankCover, compare?)` so the optional parameter lands at the end. No known external callers within the Endo monorepo. - Update `rankOrder.test.js` to exercise the new defaults and to reference `compareRankRemotablesTied` where intent is preserve-order rather than short-circuit. Refs: endojs#2883
kriscendobot
pushed a commit
to endojs/endo-but-for-bots
that referenced
this pull request
Jun 30, 2026
…n fresh erights directed on PR #572 (review 4597598287): for all three "admit immutable ArrayBuffer through codecs" changes, withdraw and open a fresh view-based implementation PR rather than retarget the branches. Records the decision as Design Decision 6 and removes the disposition open question; closing upstream endojs/endo#3226 is left as a maintainer-coordinated action.
Contributor
|
Closing per endojs/endo-but-for-bots#572 (comment) |
kriscendobot
pushed a commit
to endojs/endo-but-for-bots
that referenced
this pull request
Jul 1, 2026
erights closed #429, #57, and endojs/endo#3226 on 2026-06-30 (PR #572 review), executing the withdrawal disposition the design already recorded. Convert the future-framed 'closing ... is a maintainer-coordinated action' language into the accomplished fact, leaving 'open the fresh view-based PR' as the sole remaining step. Bump the Updated metadata and README row to 2026-07-01.
kriscendobot
pushed a commit
to endojs/endo-but-for-bots
that referenced
this pull request
Jul 1, 2026
…mutable ArrayBuffer Pivots the OCapN byteArray passStyle data model per erights' directive on PR #429: passStyleOf attaches byteArray to a plain frozen Uint8Array backed by a plain frozen immutable ArrayBuffer (the view), and a bare immutable ArrayBuffer is no longer byteArray (not passable). The capdata / smallcaps / encode-passable / marshal-justin wire forms are unchanged; the JS boundary becomes frozenBytes / thawnBytes with the hex helpers re-cast to Uint8Array. Supersedes the bare-buffer premise of PRs #429, #57, and upstream endojs/endo#3226; prototyped on feat/narrow-bytearray-to-uint8.
kriscendobot
pushed a commit
to endojs/endo-but-for-bots
that referenced
this pull request
Jul 1, 2026
…n fresh erights directed on PR #572 (review 4597598287): for all three "admit immutable ArrayBuffer through codecs" changes, withdraw and open a fresh view-based implementation PR rather than retarget the branches. Records the decision as Design Decision 6 and removes the disposition open question; closing upstream endojs/endo#3226 is left as a maintainer-coordinated action.
kriscendobot
pushed a commit
to endojs/endo-but-for-bots
that referenced
this pull request
Jul 1, 2026
erights closed #429, #57, and endojs/endo#3226 on 2026-06-30 (PR #572 review), executing the withdrawal disposition the design already recorded. Convert the future-framed 'closing ... is a maintainer-coordinated action' language into the accomplished fact, leaving 'open the fresh view-based PR' as the sole remaining step. Bump the Updated metadata and README row to 2026-07-01.
9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Admits the immutable
ArrayBuffer(passStyle'byteArray') through thecapdata, smallcaps, and encode-passable codecs in
@endo/marshal, andadds the supporting hex/byteArray helpers in
@endo/pass-style.@endo/pass-stylegainsbyteArrayToHex,hexToByteArray,byteArrayToUint8Array, anduint8ArrayToByteArray, all routedthrough
@endo/hex.@endo/marshal:byteArrayencodes as{"@qclass":"byteArray","data":"<hex>"}.byteArrayencodes as"*<hex>"; the*prefix isnow reserved for byteArray.
byteArrayencodes asa<encodeBigInt(byteLength)>:<hex>. The Elias-delta length prefixgives shortlex ordering matching
compareRankwith no arbitrarysize cap, and every body character is safe inside both
legacyOrderedandcompactOrderedarray framings.byteArrayashexToByteArray("<hex>").Syrup already supported this value through its bytestring primitive; CBOR
remains out of scope until
packages/ocapn/src/cborlands.Most critical files to review:
packages/marshal/src/encodePassable.js(ordering + framing safety)packages/marshal/src/encodeToSmallcaps.js(new reserved prefix)packages/marshal/src/encodeToCapData.js(new@qclass)packages/pass-style/src/byteArray.js(helper surface and validation)Security Considerations
The new encodings introduce new vocabulary that decoders must validate:
the smallcaps
*prefix, the capdata@qclass: "byteArray", and theencode-passable
aprefix. Decoders reject malformed hex andlength-mismatched payloads, and the resulting
ArrayBufferis hardened(immutable) before exposure, preserving the pass-style invariant that
peers cannot mutate shared bytes. No new authority is introduced; this
admits existing immutable-
ArrayBuffervalues through codecs thatpreviously rejected them.
Scaling Considerations
Hex encoding doubles payload size relative to raw bytes for capdata and
smallcaps. The encode-passable form uses an Elias-delta-encoded
byteLengthprefix, so there is no arbitrary size cap and the cost islogarithmic in
byteLength. Callers that need compact wire formats forlarge byte payloads should continue to prefer Syrup (or, eventually,
CBOR).
Documentation Considerations
packages/marshal/docs/smallcaps-cheatsheet.mdis updated to list thenewly reserved
*prefix..changeset/byte-array-hex-codecs.md) describes theuser-visible additions across
@endo/hex,@endo/pass-style, and@endo/marshal.byteArrayvalues are nowround-trippable through all three codecs and that the
*smallcapsprefix is reserved.
Testing Considerations
packages/marshal/test/byteArray.test.jscovers round-trippingthrough capdata, smallcaps, encode-passable, and marshal-justin, plus
decoder rejection of malformed payloads.
packages/pass-style/test/byte-array.test.jscovers the newhex/Uint8Array helpers.
packages/marshal/test/encodePassable.test.jsis updated to reflectthe new
a-prefix encoding and shortlex ordering.No CI or testnet impact beyond the new unit tests.
Compatibility Considerations
This is purely additive at the producer side: codecs only emit the new
forms when given an immutable
ArrayBuffer, which older versions of@endo/pass-stylecannot produce. However, older decoders will rejectmessages containing byteArray-encoded values they do not recognize
(unknown
@qclass, unknown smallcaps prefix, unknown encode-passableprefix). This is fail-closed behavior, but it means upgrades must
sequence consumers ahead of producers.
The smallcaps
*prefix was previously unassigned but unreserved; anyad-hoc consumer that happened to accept
*-prefixed strings will nowsee them interpreted as byteArray.
Upgrade Considerations
producer that emits immutable
ArrayBuffervalues into a capdata,smallcaps, or encode-passable channel.
change cannot contain byteArray values, so existing stores remain
decodable.
@endo/hex,@endo/pass-style, and@endo/marshal).