RangeError: start offset of Int32Array should be a multiple of 4

[tbrockman]

Stacktrace:

RangeError: start offset of Int32Array should be a multiple of 4
    at new Int32Array (<anonymous>)
    at new MetadataBlock (file:///home/theo/dev/ezcodelol/node_modules/.pnpm/@zenfs+core@2.2.2/node_modules/@zenfs/core/dist/backends/single_buffer.js:303:13)
    at new MetadataBlock (eval at __decorateStruct (file:///home/theo/dev/ezcodelol/node_modules/.pnpm/memium@0.2.0/node_modules/memium/dist/struct.js:52:24), <anonymous>:6:6)
    at SuperBlock.rotateMetadata (file:///home/theo/dev/ezcodelol/node_modules/.pnpm/@zenfs+core@2.2.2/node_modules/@zenfs/core/dist/backends/single_buffer.js:489:30)
    at SingleBufferStore.set (file:///home/theo/dev/ezcodelol/node_modules/.pnpm/@zenfs+core@2.2.2/node_modules/@zenfs/core/dist/backends/single_buffer.js:674:33)
    at SyncMapTransaction.setSync (file:///home/theo/dev/ezcodelol/node_modules/.pnpm/@zenfs+core@2.2.2/node_modules/@zenfs/core/dist/backends/store/map.js:20:20)
    at SyncMapTransaction.set (file:///home/theo/dev/ezcodelol/node_modules/.pnpm/@zenfs+core@2.2.2/node_modules/@zenfs/core/dist/backends/store/store.js:24:21)
    at WrappedTransaction.set (file:///home/theo/dev/ezcodelol/node_modules/.pnpm/@zenfs+core@2.2.2/node_modules/@zenfs/core/dist/backends/store/store.js:135:24)
    at async StoreFS.commitNew (file:///home/theo/dev/ezcodelol/node_modules/.pnpm/@zenfs+core@2.2.2/node_modules/@zenfs/core/dist/backends/store/fs.js:905:13)
    at async StoreFS.createFile (file:///home/theo/dev/ezcodelol/node_modules/.pnpm/@zenfs+core@2.2.2/node_modules/@zenfs/core/dist/backends/store/fs.js:452:16)

Context:
Ran into this when some code like the following (which seems to fail while copying my node_modules folder -- can work on a reproduction if the issue doesn't seem obvious):

Note

"Why are you not using fs.cp here?", you might ask. It seems to result in an empty filesystem when copying Passthrough -> SingleBuffer, so I just wrote my own dumb version.

export const copyDir = async (fs: typeof _fs, source: string, dest: string, filter: (path: string) => boolean = () => true) => {
    const symlinkQueue: { src: string; dest: string }[] = [];

    async function copyRecursive(src: string, dst: string) {
        try {
            const entries = await fs.readdir(src, { withFileTypes: true });
            await fs.mkdir(dst, { recursive: true });

            for (const entry of entries) {
                const srcPath = path.join(src, entry.name);
                const srcRelPath = path.relative(source, srcPath);
                const dstPath = path.join(dst, entry.name);

                if (!filter(srcRelPath)) {
                    continue;
                }

                if (entry.isDirectory()) {
                    await copyRecursive(srcPath, dstPath);
                } else if (entry.isFile()) {
                    const data = nodeFs.readFileSync(srcRelPath);
                    const stats = nodeFs.statSync(srcRelPath);

                    await fs.writeFile(dstPath, data, { encoding: 'utf-8' });
                    // Copy file metadata
                    // console.log('stats', stats)
                    await fs.chown(dstPath, stats.uid, stats.gid);
                    await fs.chmod(dstPath, stats.mode);
                    await fs.utimes(dstPath, stats.atime, stats.mtime);
                } else if (entry.isSymbolicLink()) {
                    symlinkQueue.push({ src: srcPath, dest: dstPath });
                }
            }
        } catch (e) {
            console.error(`Failed to copy ${src} to ${dest}:`, e);
            throw e;
        }
    }

    async function resolveSymlinks() {
        for (const { src, dest } of symlinkQueue) {
            try {
                const target = await fs.readlink(src);
                const absoluteTarget = path.resolve(path.dirname(src), target);

                try {
                    await fs.stat(absoluteTarget);
                    await fs.symlink(target, dest);
                } catch {
                    await fs.copyFile(absoluteTarget, dest);
                }
            } catch (err) {
                console.error(`Failed to copy symlink ${src}:`, err);
            }
        }
    }

    await copyRecursive(source, dest);
    await resolveSymlinks();
}

export const takeSnapshot = async (props: Partial<TakeSnapshotProps> = {}) => {
    let { root, filter } = { ...snapshotDefaults, ...props };

    const estimateUsed = async (folderPath: string) => {
        let total = 0;
    
        const walk = async (dir: string) => {
            const entries = await nodePromises.readdir(dir, { withFileTypes: true })
            for (const entry of entries) {
                const fullPath = path.join(dir, entry.name);
                const stats = await nodePromises.lstat(fullPath);
    
                if (entry.isDirectory()) {
                    await walk(fullPath);
                } else {
                    total += stats.blocks * 512;
                }
            }
        };
    
        await walk(folderPath);
        return total;
    };
    

    const buffer = new ArrayBuffer((await estimateUsed(root))); // ???
    console.debug('buffer', { size: buffer.byteLength, root });

    const readable = await resolveMountConfig({ backend: Passthrough, fs: nodeFs, prefix: root });
    const writable = await resolveMountConfig({ backend: SingleBuffer, buffer });
    // TODO: there's probably a better way to do this
    const hostPath = '/mnt/host';
    const snapshotPath = '/mnt/snapshot';
    mount(hostPath, readable);
    mount(snapshotPath, writable);
    await readable.ready()
    await writable.ready()
    await copyDir(_fs, hostPath, snapshotPath, filter)

    return buffer;
};

// ...
// await takeSnapshot()

Expectation:

This seems like it should work (and it did somewhere around ~2.0.0 -- or perhaps I broke this), not exactly sure what's going wrong here.

Very brief debugging attempt:

Looks like somehow this.used_bytes for the MetadataBlock isn't divisible by 4 (see second argument).

Not sure if related, but is field alignment correct here?

Example stats of file causing this issue:

failed to write {
  stats: Stats {
    dev: 64513,
    mode: 33204,
    nlink: 2,
    uid: 1000,
    gid: 1000,
    rdev: 0,
    blksize: 4096,
    ino: 86000425,
    size: 47,
    blocks: 8,
    atimeMs: 1746813668894.515,
    mtimeMs: 1746805215458.8567,
    ctimeMs: 1746805217720.872,
    birthtimeMs: 1746805215458.8567
  },
  data: 'import { assoc } from "../fp";\nexport = assoc;\n'
}

[james-pre]

This is due to MetadataBlock extending Int32Array. The fix is pretty easy, I'll try to have a patch out sometime tomorrow.

[tbrockman]

This is due to MetadataBlock extending Int32Array. The fix is pretty easy, I'll try to have a patch out sometime tomorrow.

Awesome, thanks for the quick turnaround! Glad I reported it rather than digging into it too much more myself then 😬

[james-pre]

Hey @tbrockman,

A new test to help prevent a regression would be really helpful. Please let me know if you'd like to open a PR for this.

[tbrockman]

Sure. Know why something like the following might be failing?

test('should recursively copy files with with same stats', async () => {
	const buffer = new ArrayBuffer(0x100000);

	const source = await resolveMountConfig({ backend: Passthrough, fs: nodeFS, prefix: path.join(process.cwd(), 'tests/data') });
	mount('/src', source);

	const dest = await resolveMountConfig({ backend: SingleBuffer, buffer });
	mount('/dst', dest);

	fs.cpSync('/src', '/dst', { recursive: true, force: true, preserveTimestamps: true });

	// recursively walk through the directory and check that files are the same
	const files = fs.readdirSync('/src', { withFileTypes: true, recursive: true });
	for (const file of files) {
		const srcFile = path.join('/src', file.name);
		const dstFile = path.join('/dst', file.name);
		assert(fs.existsSync(dstFile));

		const stats = fs.statSync(srcFile);
		const dstStats = fs.statSync(dstFile);
		assert.strictEqual(stats, dstStats);

		if (file.isFile()) {
			const srcContent = fs.readFileSync(srcFile, 'utf-8');
			const dstContent = fs.readFileSync(dstFile, 'utf-8');
			assert.strictEqual(srcContent, dstContent);
		}
	}
})

theodores-MacBook-Pro:zenfs-core theodorebrockman$ npx tsx tests/backend/single-buffer.test.ts 
▶ SingleBuffer
  ✔ filesystem restoration from original buffer (with same metadata) (10.657334ms)
  ✔ cross-thread SharedArrayBuffer (99.360167ms)
  ✖ should recursively copy files with with same stats (28.865125ms)
✖ SingleBuffer (139.574667ms)
ℹ tests 3
ℹ suites 1
ℹ pass 2
ℹ fail 1
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 1017.996416

✖ failing tests:

test at tests/backend/single-buffer.test.ts:1:1499
✖ should recursively copy files with with same stats (28.865125ms)
  Exception [Error]: EISDIR: File is a directory, open '/dst'
      at Module._openSync (file:///Users/theodorebrockman/Documents/dev/zenfs-core/dist/vfs/sync.js:227:15)
      at Module.openSync (file:///Users/theodorebrockman/Documents/dev/zenfs-core/dist/vfs/sync.js:238:27)
      at Module.utimesSync (file:///Users/theodorebrockman/Documents/dev/zenfs-core/dist/vfs/sync.js:623:25)
      at Module.cpSync (file:///Users/theodorebrockman/Documents/dev/zenfs-core/dist/vfs/sync.js:850:20)
      at TestContext.<anonymous> (/Users/theodorebrockman/Documents/dev/zenfs-core/tests/backend/single-buffer.test.ts:66:6)
      at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
      at async Test.run (node:internal/test_runner/test:1054:7)
      at async Suite.processPendingSubtests (node:internal/test_runner/test:744:7) {
    errno: 21,
    code: 'EISDIR',
    path: '/dst',
    dest: undefined,
    syscall: 'open'
  }

[james-pre]

Feedback on the PR.