Data Dedup Rehydrate

Data Dedup Rehydrate is a Python tool for scanning and restoring files from NTFS volumes that use Windows Data Deduplication.

It can parse a raw NTFS image, identify files represented by Dedup reparse points, locate the corresponding Dedup chunk-store records, reconstruct the original byte stream, and write restored files to disk. A Tkinter GUI and a small CLI are included.

Features

• Pure-Python NTFS raw-image parsing
• Automatic discovery of Dedup reparse-point files
• Stream key and logical file-size extraction from Dedup reparse data
• SMAP stream-map lookup in Dedup Stream containers
• Chunk lookup in Dedup Data containers
• XPRESS decompression for compressed chunks on Windows
• GUI workflow for scanning, searching, selecting, and restoring files
• CLI workflow for scripted scan and restore operations
• Optional support for an extracted Dedup folder instead of reading chunk containers only from the image

Requirements

• Python 3.10 or newer
• Windows is recommended when restoring compressed chunks, because XPRESS decompression uses ntdll.RtlDecompressBufferEx
• Tkinter for the GUI, included with most standard Python installations

No third-party Python package is required for the raw-image workflow.

Installation

This project can be run directly from its parent directory:

python -m data_dedup_rehydrate

If you are inside the package directory itself, move one directory up before running with -m:

cd ..
python -m data_dedup_rehydrate

GUI Usage

Launch the GUI:

python -m data_dedup_rehydrate

Input fields:

• Raw image: the NTFS image to scan. This is the primary required input.
• Extracted $MFT: reserved for workflows where an extracted MFT is available.
• Dedup folder: optional path to an extracted Dedup, ChunkStore, or .ddp folder.
• Output dir: destination directory for restored files.

Basic workflow:

1. Select a raw NTFS image.
2. Optionally select an extracted Dedup folder.
3. Click Scan.
4. Use the search box to filter files by path or filename.
5. Double-click a row, or select a row and press Space, to mark files for restore.
6. Click Restore selected.

CLI Usage

Scan a raw image:

python -m data_dedup_rehydrate --scan <raw-image>

Scan and verify that each listed file has a matching SMAP record:

python -m data_dedup_rehydrate --scan <raw-image> --verify

Restore a file by filename or path:

python -m data_dedup_rehydrate --restore <raw-image> <filename-or-path> --out <output-directory>

Use an extracted Dedup folder as the chunk-store source:

python -m data_dedup_rehydrate --restore <raw-image> <filename-or-path> --dedup <dedup-folder> --out <output-directory>

Supported Inputs

Raw NTFS image

The raw image path is the main input. The tool parses:

• NTFS boot sector
• $MFT
• file records
• resident and non-resident attributes
• data runs
• Dedup reparse attributes
• Dedup chunk-store files present inside the image

Extracted Dedup folder

The optional Dedup folder can point to any of these layouts:

Dedup\ChunkStore\{GUID}.ddp\Stream\*.ccc
Dedup\ChunkStore\{GUID}.ddp\Data\*.ccc

ChunkStore\{GUID}.ddp\Stream\*.ccc
ChunkStore\{GUID}.ddp\Data\*.ccc

{GUID}.ddp\Stream\*.ccc
{GUID}.ddp\Data\*.ccc

When provided, the extracted folder is searched before or alongside chunk-store data found in the raw image.

How Restoration Works

The restoration pipeline is:

1. Parse the NTFS image and load MFT records.
2. Find files with $REPARSE_POINT attributes.
3. Keep only reparse points with IO_REPARSE_TAG_DEDUP (0x80000013).
4. Parse the Dedup reparse payload to recover:
   • the 16-byte stream key
   • the logical file size
5. Search Dedup Stream containers for the stream key.
6. Locate the associated Smap record.
7. Parse SMAP entries to recover the ordered chunk hashes and chunk lengths.
8. Search Dedup Data containers for each chunk hash.
9. Validate each chunk record with its Ckhr header.
10. Extract each chunk payload.
11. Decompress XPRESS-compressed chunks when required.
12. Concatenate chunks and verify the restored size.
13. Write the restored file to the selected output directory.

Output Format

CLI scan output uses tab-separated fields:

<size>    <stream-key-hex>    <status>    <path>

Common statuses:

• dedup reparse: a Dedup reparse-point file was found
• ready: a matching SMAP record was found during verification
• smap not found: the file was detected, but no matching SMAP record was found

Restore output prints the path of the restored file on success.

Troubleshooting

No module named data_dedup_rehydrate

Run the command from the parent directory of the package:

cd <project-root>
python -m data_dedup_rehydrate

smap not found

The file was detected as a Dedup reparse point, but the matching Stream container was not found. Try providing an extracted Dedup or ChunkStore folder with --dedup.

chunk extraction failed

A chunk hash from the SMAP could not be found in the available Data containers, or the corresponding chunk record did not validate.

XPRESS unavailable

The target file uses compressed Dedup chunks, but XPRESS decompression is unavailable in the current runtime. Run the tool on Windows with a standard CPython build.

Limitations

• The raw-image workflow expects an NTFS volume image with readable MFT records.
• XPRESS decompression currently uses the Windows native ntdll API.
• The extracted $MFT field is present for workflow compatibility, but the primary implemented scanner is the raw-image scanner.
• Damaged or partial chunk stores may scan successfully but fail during restore if required Stream or Data containers are missing.

License

This project is released under the MIT License. See LICENSE.