add add_metadata_plan.md - #5
Merged
messa merged 5 commits intoJan 26, 2026
Merged
Conversation
Document the main goal: after blockcopy sync, rsync should consider files identical. Added detailed analysis of rsync's "quick check" algorithm based on rsync source code (generator.c, util1.c): - rsync compares size first, then mtime (seconds only by default) - with -a flag also checks permissions and owner/group - plan will achieve compatibility when using --truncate --times --perms --owner --group options together
Rsync's "quick check" only considers size and mtime when deciding whether to transfer file content. Permissions, owner, and group are only updated via set_file_attrs() without triggering content transfer. This means --truncate and --times are sufficient for rsync compatibility. The --perms/--owner/--group options are optional extras.
New test file tests/test_rsync_compat.py with three tests: - test_rsync_sees_no_changes_after_blockcopy: main test verifying rsync sees no differences after blockcopy --truncate --times - test_rsync_sees_no_changes_with_full_metadata: with --perms too - test_rsync_without_times_would_transfer: negative test confirming rsync DOES see mtime difference without --times option Tests use rsync --dry-run --itemize-changes to verify behavior.
New test verifies blockcopy preserves full nanosecond mtime precision, compatible with rsync's strict nanosecond comparison mode (requires rsync >= 3.1.3).
- Skip rsync compat tests locally if rsync not installed - Fail with error on CI if rsync missing (CI must have rsync) - Detect CI via common env vars: CI, GITHUB_ACTIONS, GITLAB_CI, JENKINS_URL
There was a problem hiding this comment.
Pull request overview
Adds a design/implementation plan for preserving file metadata during save (to make blockcopy outputs rsync-friendly), plus a small robustness fix in the save protocol parser.
Changes:
- Add an explicit length check when reading the
Donepayload indo_save()to avoid silently parsing partial reads. - Add
add_metadata_plan.mddescribing proposed metadata-preservation flags and a newMetaprotocol command. - Add a short
TODO.mdfor follow-up work items.
Reviewed changes
Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
blockcopy.py |
Harden save by rejecting incomplete reads of the Done size field. |
add_metadata_plan.md |
New planning document for rsync-compatible metadata preservation and protocol changes. |
TODO.md |
Tracks next steps (version bump, README update). |
Comments suppressed due to low confidence (2)
add_metadata_plan.md:676
- The plan describes a new protocol command as both
Metaandmeta(e.g.,meta (4 bytes "Meta")and the command order showsmeta), while earlier examples writeb'Meta'but the save-side snippet checksb'meta'. Since the protocol command bytes are case-sensitive, standardize on one exact 4-byte token throughout the document (and in the eventual implementation) to avoid an incompatible sender/receiver.
Formát:
meta (4 bytes "Meta")
atime_ns (8 bytes, signed big-endian)
mtime_ns (8 bytes, signed big-endian)
mode (4 bytes, unsigned big-endian)
uid (4 bytes, unsigned big-endian)
gid (4 bytes, unsigned big-endian)
owner_name_len (2 bytes, unsigned big-endian)
owner_name (owner_name_len bytes, UTF-8)
group_name_len (2 bytes, unsigned big-endian)
group_name (group_name_len bytes, UTF-8)
Fixní část: 4 + 8 + 8 + 4 + 4 + 4 + 2 + 2 = 36 bajtů + délka jmen
Pořadí příkazů v protokolu: `data/dlzm...` → `meta` → `Done`
add_metadata_plan.md:681
- This section includes absolute local paths under
/home/..., which are machine-specific and leak local environment details. Prefer repository-relative paths (e.g.,blockcopy.py,tests/test_copy.py) or omit this section.
## Kritické soubory
- `/home/messa/code/gh/blockcopy/blockcopy.py` - hlavní změny
- `/home/messa/code/gh/blockcopy/tests/test_copy.py` - nové testy
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
| **Klíč k ověření:** `rsync --dry-run --itemize-changes` - prázdný výstup = soubory jsou shodné. | ||
|
|
||
| **Poznámka k nanosekudnám:** rsync od verze 3.1.3 podporuje `--modify-window=-1` pro porovnávání nanosekund. blockcopy používá `os.utime(ns=...)` který zachovává plnou přesnost, takže je kompatibilní i s tímto režimem. |
There was a problem hiding this comment.
Spelling: "nanosekudnám" should be "nanosekundám".
Suggested change
| **Poznámka k nanosekudnám:** rsync od verze 3.1.3 podporuje `--modify-window=-1` pro porovnávání nanosekund. blockcopy používá `os.utime(ns=...)` který zachovává plnou přesnost, takže je kompatibilní i s tímto režimem. | |
| **Poznámka k nanosekundám:** rsync od verze 3.1.3 podporuje `--modify-window=-1` pro porovnávání nanosekund. blockcopy používá `os.utime(ns=...)` který zachovává plnou přesnost, takže je kompatibilní i s tímto režimem. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.