Skip to content

Commit 2d5dd2a

Browse files
kriskowalkriskowal
committed
feat(marshal,pass-style): admit immutable ArrayBuffer through codecs
Immutable `ArrayBuffer` (passStyle `'byteArray'`) is now serializable through the capdata, smallcaps, and encode-passable codecs: - `@endo/pass-style` gains `byteArrayToHex`, `hexToByteArray`, `byteArrayToUint8Array`, and `uint8ArrayToByteArray` helpers, all routed through `@endo/hex`. - `@endo/marshal`: - capdata: `byteArray` encodes as `{"@qclass":"byteArray","data":"<hex>"}`. - smallcaps: `byteArray` encodes as `"*<hex>"`; the `*` prefix is now reserved. - encode-passable: `byteArray` encodes as `a<encodeBigInt(byteLength)>:<hex>`. The Elias-delta length prefix gives shortlex ordering matching `compareRank` with no arbitrary size cap. - marshal-justin: renders byteArray as `hexToByteArray("<hex>")`. Syrup already supported this value through its bytestring primitive; CBOR remains out of scope until `packages/ocapn/src/cbor` lands.
1 parent 459347b commit 2d5dd2a

14 files changed

Lines changed: 397 additions & 12 deletions

‎.changeset/byte-array-hex-codecs.md‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
---
2+
'@endo/hex': minor
3+
'@endo/pass-style': minor
4+
'@endo/marshal': minor
5+
---
6+
7+
Immutable `ArrayBuffer` (passStyle `'byteArray'`) is now serializable through
8+
the capdata, smallcaps, and encode-passable codecs.
9+
10+
- New `@endo/hex` package providing `encodeHex`/`decodeHex` (with
11+
`toHex`/`fromHex` aliases for the TC39 spelling). Short-circuits to native
12+
`Uint8Array.prototype.toHex` and `Uint8Array.fromHex` when available.
13+
- `@endo/pass-style` gains `byteArrayToHex`, `hexToByteArray`,
14+
`byteArrayToUint8Array`, and `uint8ArrayToByteArray` helpers.
15+
- `@endo/marshal`:
16+
- **capdata**: byteArray encodes as `{"@qclass":"byteArray","data":"<hex>"}`.
17+
- **smallcaps**: byteArray encodes as `"*<hex>"`. The `*` prefix is now
18+
reserved.
19+
- **encode-passable**: byteArray encodes as
20+
`a<encodeBigInt(byteLength)>:<hex>`. The Elias-delta length prefix gives
21+
shortlex ordering (matching `compareRank`) with no arbitrary size cap, and
22+
every character in the body is safe inside both `legacyOrdered` and
23+
`compactOrdered` array framings.
24+
- **marshal-justin**: renders byteArray as `hexToByteArray("<hex>")`.
25+
26+
Syrup already supported this value; no change required there.

‎packages/marshal/docs/smallcaps-cheatsheet.md‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ An example-based summary of the Smallcaps encoding of the OCapN [Abstract Syntax
1010
| bigint | Integer | `7n`<br>`-7n` | `"+7"`<br>`"-7"` |
1111
| number | Float64 | `Infinity`<br>`-Infinity`<br>`NaN`<br>`-0`<br>`7.1` | `"#Infinity"`<br>`"#-Infinity"`<br>`"#NaN"`<br>`"#-0"` // unimplemented<br>`7.1` |
1212
| string | String | `'#foo'`<br>`'foo'` | `"!#foo"` // special strings<br>`"foo"` // other strings |
13-
| byteArray | ByteArray | `buf.toImmutable()` | // undecided & unimplemented |
13+
| byteArray | ByteArray | `buf.toImmutable()` | `"*deadbeef"` // after `*`, hex encoding |
1414
| passable symbols | Symbol | `passableSymbolForName('foo')` | `"%foo"` // in transition |
1515
| copyArray | List | `[a,b]` | `[<a>,<b>]` |
1616
| copyRecord | Struct | `{foo:a,'#foo':b}` | `{"!#foo":<b>,"foo":<a>}` // keys sorted |
@@ -28,7 +28,6 @@ An example-based summary of the Smallcaps encoding of the OCapN [Abstract Syntax
2828
* Structs [can only have string-named properties](https://github.com/endojs/endo/blob/master/packages/pass-style/doc/copyRecord-guarantees.md).
2929
* Errors can also carry an optional `errorId` string property.
3030
* We expect to expand the optional error properties over time.
31-
* The ByteArray encoding is not yet designed or implemented.
3231

3332
## Readability Invariants
3433

‎packages/marshal/src/encodePassable.js‎

Lines changed: 41 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,8 @@ import {
1010
isErrorLike,
1111
nameForPassableSymbol,
1212
passableSymbolForName,
13+
byteArrayToHex,
14+
hexToByteArray,
1315
} from '@endo/pass-style';
1416

1517
/**
@@ -473,14 +475,47 @@ const decodeLegacyArray = (encoded, decodePassable, skip = 0) => {
473475
};
474476

475477
/**
478+
* Encodes a ByteArray as `a<length><:><hex>`, where `<length>` is the
479+
* `encodeBigInt` of `byteLength` (a non-negative bigint, which already
480+
* encodes in shortlex-numerical order), followed by the explicit
481+
* separator `:` and the lowercase hex of the bytes.
482+
*
483+
* Shortlex is inherited in two stages:
484+
* 1. `encodeBigInt` on the length preserves numerical order, so shorter
485+
* byteArrays sort before longer ones.
486+
* 2. At equal length, the length encodings are identical and ordering
487+
* falls through to the byte-lex-preserving hex body.
488+
*
489+
* Every character used here — `a`, `p`/`n`/`~`/`#`, decimal digits, `:`,
490+
* and hex digits `[0-9a-f]` — is outside the escape-reserved sets of
491+
* both the `legacyOrdered` and `compactOrdered` array framings, so no
492+
* escaping is needed.
493+
*
476494
* @param {ByteArray} byteArray
477495
* @param {(byteArray: ByteArray) => string} _encodePassable
478496
* @returns {string}
479497
*/
480498
const encodeByteArray = (byteArray, _encodePassable) => {
481-
// TODO implement
482-
Fail`encodePassable(byteArray) not yet implemented: ${byteArray}`;
483-
return ''; // Just for the type
499+
const lenEnc = encodeBigInt(BigInt(byteArray.byteLength));
500+
return `a${lenEnc}:${byteArrayToHex(byteArray)}`;
501+
};
502+
503+
const rByteArrayPayload = /^(p[~]*[0-9]+:[0-9]+):([0-9a-f]*)$/;
504+
505+
/**
506+
* Inverse of {@link encodeByteArray}.
507+
*
508+
* @param {string} encoded The body after the leading `'a'` prefix char.
509+
* @returns {import('@endo/pass-style').ByteArray}
510+
*/
511+
const decodeByteArray = encoded => {
512+
const match = encoded.match(rByteArrayPayload);
513+
match || Fail`Encoded byteArray expected: ${encoded}`;
514+
const [, lenEnc, hex] = /** @type {RegExpMatchArray} */ (match);
515+
const byteLength = Number(decodeBigInt(lenEnc));
516+
hex.length === byteLength * 2 ||
517+
Fail`byteArray length mismatch: header ${q(byteLength)} vs body ${q(hex.length / 2)}`;
518+
return hexToByteArray(hex, 'encodePassable byteArray');
484519
};
485520

486521
const encodeRecord = (record, encodeArray, encodePassable) => {
@@ -749,6 +784,9 @@ const makeInnerDecode = (decodeStringSuffix, decodeArray, options) => {
749784
case ':': {
750785
return decodeTagged(encoded, decodeArray, innerDecode, skip);
751786
}
787+
case 'a': {
788+
return decodeByteArray(getSuffix(encoded, skip + 1));
789+
}
752790
default: {
753791
throw Fail`invalid database key: ${getSuffix(encoded, skip)}`;
754792
}

‎packages/marshal/src/encodeToCapData.js‎

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,8 @@ import {
1414
assertPassableSymbol,
1515
nameForPassableSymbol,
1616
passableSymbolForName,
17+
byteArrayToHex,
18+
hexToByteArray,
1719
} from '@endo/pass-style';
1820
import { X, Fail, q } from '@endo/errors';
1921

@@ -194,8 +196,10 @@ export const makeEncodeToCapData = (encodeOptions = {}) => {
194196
return passable.map(encodeToCapDataRecur);
195197
}
196198
case 'byteArray': {
197-
// TODO implement
198-
throw Fail`marsal of byteArray not yet implemented: ${passable}`;
199+
return {
200+
[QCLASS]: 'byteArray',
201+
data: byteArrayToHex(passable),
202+
};
199203
}
200204
case 'tagged': {
201205
return {
@@ -367,6 +371,12 @@ export const makeDecodeFromCapData = (decodeOptions = {}) => {
367371
const { tag, payload } = jsonEncoded;
368372
return makeTagged(tag, decodeFromCapData(payload));
369373
}
374+
case 'byteArray': {
375+
const { data } = jsonEncoded;
376+
typeof data === 'string' ||
377+
Fail`invalid byteArray data typeof ${q(typeof data)}`;
378+
return hexToByteArray(data, 'capData byteArray');
379+
}
370380
case 'slot': {
371381
// See note above about how the current encoding cannot reliably
372382
// distinguish which we should call, so in the non-default case

‎packages/marshal/src/encodeToSmallcaps.js‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,8 @@ import {
1717
assertPassableSymbol,
1818
nameForPassableSymbol,
1919
passableSymbolForName,
20+
byteArrayToHex,
21+
hexToByteArray,
2022
} from '@endo/pass-style';
2123

2224
/** @import {Passable, RemotableObject} from '@endo/pass-style' */
@@ -50,14 +52,15 @@ const DASH = '-'.charCodeAt(0);
5052
* Of these, smallcaps currently uses the following:
5153
*
5254
* * `!` - escaped string
55+
* * `*` - byteArray (hardened Immutable ArrayBuffer), hex-encoded
5356
* * `+` - non-negative bigint
5457
* * `-` - negative bigint
5558
* * `#` - manifest constant
5659
* * `%` - symbol
5760
* * `$` - remotable
5861
* * `&` - promise
5962
*
60-
* All other special characters (`"'()*,`) are reserved for future use.
63+
* All other special characters (`"'(),`) are reserved for future use.
6164
*
6265
* The manifest constants that smallcaps currently uses for values:
6366
* * `#undefined`
@@ -230,8 +233,7 @@ export const makeEncodeToSmallcaps = (encodeOptions = {}) => {
230233
return passable.map(encodeToSmallcapsRecur);
231234
}
232235
case 'byteArray': {
233-
// TODO implement
234-
throw Fail`marsal of byteArray not yet implemented: ${passable}`;
236+
return `*${byteArrayToHex(passable)}`;
235237
}
236238
case 'tagged': {
237239
return {
@@ -407,6 +409,9 @@ export const makeDecodeFromSmallcaps = (decodeOptions = {}) => {
407409
}
408410
return result;
409411
}
412+
case '*': {
413+
return hexToByteArray(encoding.slice(1), 'smallcaps byteArray');
414+
}
410415
default: {
411416
throw Fail`Special char ${q(
412417
c,

‎packages/marshal/src/marshal-justin.js‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -177,6 +177,11 @@ const decodeToJustin = (encoding, shouldIndent = false, slots = []) => {
177177
assert.typeof(sym, 'symbol');
178178
return;
179179
}
180+
case 'byteArray': {
181+
const { data } = rawTree;
182+
assert.typeof(data, 'string');
183+
return;
184+
}
180185
case 'tagged': {
181186
const { tag, payload } = rawTree;
182187
assert.typeof(tag, 'string');
@@ -339,6 +344,11 @@ const decodeToJustin = (encoding, shouldIndent = false, slots = []) => {
339344
}
340345
return out.next(`passableSymbolForName(${quote(registeredName)})`);
341346
}
347+
case 'byteArray': {
348+
const { data } = rawTree;
349+
assert.typeof(data, 'string');
350+
return out.next(`hexToByteArray(${quote(data)})`);
351+
}
342352
case 'tagged': {
343353
const { tag, payload } = rawTree;
344354
out.next(`makeTagged(${quote(tag)}`);

‎packages/marshal/src/types.js‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,8 @@ export {};
4747
* } |
4848
* EncodingClass<'tagged'> & { tag: string,
4949
* payload: Encoding
50-
* }
50+
* } |
51+
* EncodingClass<'byteArray'> & { data: string }
5152
* } EncodingUnion
5253
*
5354
* Note that the '@@asyncIterator' encoding is deprecated. Use 'symbol' instead.

0 commit comments

Comments
 (0)