Detection Documentation Analysis

This tool evaluates how well humans versus AI can document investigative steps for detection rules. It scans a directory of detection files, extracts investigative guidance (if any), and prompts an AI to generate its own version of those steps. Then, it asks the AI to compare its output with the original documentation to measure consistency and quality.

The analysis is powered by OpenAI's ChatGPT and costs approximately $0.02 per detection file analyzed.

---

Features

• Analyze detection rule documentation quality.
• Generate AI-powered investigative steps.
• Compare AI output to existing human-authored steps.
• Built-in support for Elastic-style TOML files.
• Extensible model support via LangChain.

---

Setup

Prerequisites: Python 3.7+, pip

1. Create and activate a virtual environment:

   python3 -m venv dda_env
   source dda_env/bin/activate  # On Windows: dda_env\Scripts\activate

2. Install dependencies:

   pip3 install -r requirements.txt

3. Set your OpenAI API key:

   • macOS/Linux:

     export OPENAI_API_KEY="your_api_key"

   • Windows (Command Prompt):

     setx OPENAI_API_KEY "your_api_key"

---

Running the Program

To analyze a directory of detection rules:

python3 run_document_detection_analysis.py -d path/to/rules

Optional Arguments:

usage: run_document_detection_analysis.py [-h] [-d DIRECTORY] [-b BEGINNING]
                                          [-e END] [-s SUMMARY_ONLY]
                                          [-n DATASET_NAME]

Options:
  -h, --help              Show this help message and exit
  -d, --directory         Base directory of the detection rules
  -b, --beginning         Optional start index for partial processing
  -e, --end               Optional end index for partial processing
  -s, --summary_only      If set, only generate summary output
  -n, --dataset_name      Name of the dataset being analyzed

Example:

python3 run_document_detection_analysis.py -d ./rules -b 5 -e 10 -n "TestRules"

---

Customization

🔁 Changing the LLM

To use a different OpenAI model, edit the execute_graph function in llm_document_detection_analysis.py:

ChatOpenAI(model="gpt-4o-2024-08-06", temperature=0, rate_limiter=rate_limiter)

To use another provider (Anthropic, Mistral, etc.), refer to LangChain's official docs for integration examples.

🛠 Parsing Custom Rule Formats

If your detection files differ from Elastic's TOML format:

1. Edit the function parse_toml_file_node in get_data_from_file_node.py.
2. You can use an LLM to help you generate this function based on your rule file structure.
3. Test with a small subset of files before large-scale runs.

---

Troubleshooting

• No results or blank output?

  • Check that the detection files contain investigative guidance.
  • Ensure your API key is valid and not rate-limited.

• Incorrect parsing?

  • Confirm that the rule files follow the expected format.
  • Customize the parsing logic as described above.

• Unexpected charges?

  • Review the number of files and API model pricing. Consider using a cheaper model or limiting the number of processed files.

---

Contributing

Contributions are welcome! Please open an issue or submit a pull request for enhancements or bug fixes.

---

License

MIT License

---

Acknowledgments

This tool uses:

• LangChain
• OpenAI GPT models