🔍 Lucene 10 Term Vector Reconstructor

A command-line tool to reconstruct textual content from Apache Lucene 10.0 index files when the stored ("S") flag is not present, but term vectors with positions or offsets are available.

This tool rebuilds documents’ textual data from term vectors by mapping stored terms according to their positions and offsets.
It is useful for index recovery, forensic analysis, or Lucene debugging.

---

⚙️ Features

• Works exclusively for Lucene 10.0.
• Extracts term vector data (terms, positions, offsets) for each indexed document.
• Automatically maps and reconstructs text content for each document.
• Produces readable .txt files per document.
• Generates a manifest CSV file summarizing extraction details.
• Handles missing term vectors gracefully.

---

🧱 Project Structure

├── pom.xml
├── src/
│   └── main/
│       └── java/
│           └── ReconFromTermVectors10.java

The Maven configuration includes:

• lucene-core 10.0.0
• lucene-analysis-common 10.0.0
• maven-shade-plugin to create a runnable fat JAR

---

🧩 Requirements

To build and run this project, make sure you have the following installed:

Tool	Version	Description
Java JDK	17 or newer	Required for compiling and running Lucene 10.0-based projects
Apache Maven	3.8+	Used for dependency management and packaging the tool
Lucene Index Files	Built using Lucene 10.0.0	The tool is version-specific and may not work with other Lucene versions

---

Dependencies (managed via Maven):

• org.apache.lucene:lucene-core:10.0.0
• org.apache.lucene:lucene-analysis-common:10.0.0

These will be automatically downloaded when you run:

mvn clean package

---

🚀 Building the Tool

Prerequisites

• Java 17+
• Maven 3.8+
• Lucene 10.0.0 index files

Build Steps

git clone https://github.com/<your-username>/lucene10-termvector-reconstructor.git
cd lucene10-termvector-reconstructor

# Build and package runnable JAR
mvn clean package

After successful build, your JAR will be located at:

target/tvrecon10-1.0.jar

💻 Usage

java -jar target/tvrecon10-1.0.jar <indexDir> <outDir> <field> [pathField] [nameField]

Arguments

Argument	Description	Default
<indexDir>	Path to the Lucene 10 index directory	—
<outDir>	Output directory where reconstructed files are stored	—
<field>    	Name of the field whose term vectors should be used for             reconstruction	
[pathField]	(Optional) Field containing original document path	path
[nameField]	(Optional) Field containing document filename	filename

Example

java -jar target/tvrecon10-1.0.jar /data/lucene10_index ./reconstructed content path filename

This will:

• Read all term vectors from /data/lucene10_index

• Write reconstructed text files to ./reconstructed

• Generate a manifest.csv listing document IDs, paths, filenames, and field flags.

📄 Output

Each reconstructed document is saved as:

<filename>.txt

A manifest.csv is created in the output directory with columns:

docID, path, filename, hasTV, hasPos, hasOff, outfile

🧠 How It Works

• Reads all Lucene index segments using DirectoryReader.

• Extracts stored fields like path and filename.

• Fetches term vectors for the specified field.

• For each term:

  • Reads positions and offsets (if available).

  • Builds a list of term-position pairs (Tok objects).

• Reconstructs content:

  • By offsets, if available (exact sequence).

  • By positions, otherwise.

  • Falls back to sorted terms if neither available.

  • Writes reconstructed text and logs results to CSV.

This tool currently supports Apache Lucene 10.0 only. If you’d like to adapt or extend it for other Lucene versions (e.g., 8.x, 9.x, or future 11.x releases), feel free to fork the repository and contribute!