feat: enhance augment prompt to detect disguised identifiers

[asteier2026]

Summary

• Adds an Additional extraction requirements block to the LLM entity augmentation prompt
• Requires verbatim span extraction (no normalization, reconstruction, or reformatting)
• Adds detection examples for identifiers written as digit words (phone/SSN/credit card) and letter-by-letter name spellings

Motivation

The augmenter was missing entities that were obfuscated or spoken aloud, e.g.:

• "nine o two, five five five, one two three four" (phone number)
• "J-O-H-N" (name spelled out with hyphens)

[greptile-apps]

Greptile Summary

Adds an Additional extraction requirements block to the LLM augment prompt in _get_augment_prompt, reinforcing verbatim span extraction and introducing detection hints for obfuscated identifiers (digit-word phone/SSN/credit-card numbers and letter-by-letter name spellings).

• The verbatim-extraction requirement is correctly motivated: the downstream anonymizer must locate and replace the exact text that appears in the document, so normalizing "nine o two…" to "902-…" would break replacement.
• The new detection examples show input patterns only; no corresponding expected output (label + reason) is included, leaving the LLM without a worked instance for the most novel cases.
• The requirements block is positioned after <<EXAMPLE_BLOCK>>, so the existing few-shot example does not demonstrate the new disguised-identifier behavior.

Confidence Score: 4/5

The change is prompt-only with no surrounding code logic altered; the verbatim extraction intent is sound and the new detection examples are well-chosen.

The prompt addition is logically coherent and solves a real gap. Two structural weaknesses — missing labeled output examples for the new cases and the post-example placement of the requirements — could cause inconsistent labeling or reduced LLM adherence across model versions, but neither causes an immediate breakage in the current workflow.

src/anonymizer/engine/detection/detection_workflow.py — specifically the ordering of the new requirements block relative to <<EXAMPLE_BLOCK>> and the absence of worked output examples for digit-word and letter-spelled identifiers.

Important Files Changed

Filename	Overview
src/anonymizer/engine/detection/detection_workflow.py	Adds an "Additional extraction requirements" block to the augment prompt in _get_augment_prompt, reinforcing verbatim span extraction and adding detection hints for disguised identifiers (digit-word phone/SSN/CC numbers, letter-spelled names). The block is placed after <<EXAMPLE_BLOCK>> and before the input separator. No code logic is changed, only the static prompt text.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Input text + Seed entities] --> B[_get_augment_prompt]
    B --> C{strict_labels?}
    C -- Yes --> D[label_block: ONLY allowed labels]
    C -- No --> E[label_block: prefer known labels, allow new]
    D --> F[example_block substitution]
    E --> F
    F --> G[Prompt: Task + LABEL_BLOCK + Rules + EXAMPLE_BLOCK]
    G --> H[NEW: Additional extraction requirements\n- Verbatim span extraction\n- No normalization or reconstruction\n- Detect disguised identifiers\n  e.g. digit-word phone numbers\n  e.g. letter-by-letter names]
    H --> I[--- Input text + Seed entities ---]
    I --> J[LLM augmentation inference]
    J --> K[New detected entities JSON]

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[Input text + Seed entities] --> B[_get_augment_prompt]
    B --> C{strict_labels?}
    C -- Yes --> D[label_block: ONLY allowed labels]
    C -- No --> E[label_block: prefer known labels, allow new]
    D --> F[example_block substitution]
    E --> F
    F --> G[Prompt: Task + LABEL_BLOCK + Rules + EXAMPLE_BLOCK]
    G --> H[NEW: Additional extraction requirements\n- Verbatim span extraction\n- No normalization or reconstruction\n- Detect disguised identifiers\n  e.g. digit-word phone numbers\n  e.g. letter-by-letter names]
    H --> I[--- Input text + Seed entities ---]
    I --> J[LLM augmentation inference]
    J --> K[New detected entities JSON]

Loading

_{Reviews (1): Last reviewed commit: "feat: enhance augment prompt to handle d..." | Re-trigger Greptile}

[greptile-apps]

Extraction requirements placed after the example block

Standard few-shot prompting places all instructions before the demonstrations so the example can illustrate those instructions. Here, "Additional extraction requirements" appears after <<EXAMPLE_BLOCK>>, so the concrete example does not show verbatim extraction of a disguised identifier — leaving the LLM without a worked instance of the most novel new behavior. Moving these requirements above <<EXAMPLE_BLOCK>> (or adding a second worked example that demonstrates digit-word extraction) would make the prompt self-consistent.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[asteier2026]

I think it's good the way it is.

[lipikaramaswamy]

I agree that a complete worked example would make the new behavior less ambiguous, particularly by demonstrating the canonical label and confirming that the output value must preserve the exact disguised span.

Could we move the exact-span requirements before the demonstrations and add at least one labeled input/output example for a digit-word identifier?

Input: `Call me at nine o two, five five five, one two three four.`

Output:
```json
{
  "entities": [
    {
      "value": "nine o two, five five five, one two three four",
      "label": "phone_number",
      "reason": "phone number written as digit words"
    }
  ]
}

[greptile-apps]

The full prompt in _get_augment_prompt confirms the ordering concern. Here's a concrete suggestion:

Move the verbatim-extraction requirements above <<EXAMPLE_BLOCK>> so the worked example can demonstrate them, and extend the example_block in both branches to include a digit-word case.

Step 1 — restructure the prompt string (move "Additional extraction requirements" above <<EXAMPLE_BLOCK>>):

    prompt = """Task: Find untagged sensitive entities in text (ignore already tagged entities). Focus on:
- Direct identifiers: Uniquely identify entities (names, emails, IDs), records (transaction IDs, case numbers), resources (file paths, URLs), or instances (server names, hostnames)
- Quasi-identifiers: Attributes that combine to narrow specificity (age, location, job title, timestamps, technical specs)
- Technical secrets: Credentials (passwords, API keys, tokens), access (internal URLs, endpoints), proprietary terms

We have the following type of data: <<DATA_SUMMARY>>

<<LABEL_BLOCK>>

Rules:
- Tag actual values, not placeholders
- Do not repeat already-tagged entities
- In structured formats, distinguish between:
  - Syntax/metadata: field names, column headers, function names, keywords
  - Data: assigned values, cell contents, literals, user input
  Only tag data, never syntax

Other information:
- unique_id rules: unique_id's are never only one character long (ex: "6" is not a unique_id). They are never event names or real words all strung together (ex: "ProcessManagementEvent" is NOT a unique_id).
- ipv4 label: This includes specific IP addresses as well as internal IP subnet ranges (ex: subnet=[224.0.0.0/4, 10.0.0.0/8]).
- Filename Exclusions: The "filename" label should be reserved for user-created documents or data exports (e.g., .pdf, .xlsx, .csv, .txt).
- Executable Distinction: Do not tag executable binaries (ending in .exe, .dll, or .sys) as filename. Treat these extensions as non-sensitive system identifiers.

Additional extraction requirements:
- The "value" field must be the EXACT verbatim span from the input text.
  Copy the text character-for-character exactly as it appears.
  Do NOT normalize, correct spelling, expand abbreviations, decode encodings,
  infer hidden values, translate text, reformat numbers, or otherwise modify
  the extracted span.
- Extract only text that is explicitly present in the input.
  Never reconstruct, guess, or generate a value that does not appear verbatim.
- Identifiers may be disguised, fragmented, hyphenated, misspelled, obfuscated,
  spaced out, mixed with punctuation, or written in words instead of digits.
  Detect the identifier and extract the exact text as written.

<<EXAMPLE_BLOCK>>

---
Input text: <<TAGGED_TEXT>>
Already-detected entities: <<SEED_ENTITIES>>
"""

Step 2 — append a digit-word worked example to both example_block strings. For the non-strict branch (same pattern for strict):

        example_block += """

Example (disguised identifier — digit words):
Input text: Reach me at nine o two, five five five, one two three four.
Already-detected entities: []
Output: {"entities": [{"value": "nine o two, five five five, one two three four", "label": "phone_number", "reason": "phone number written as digit words with 'o' for zero"}]}"""

The key points:

• Requirements appear before the example so the LLM reads the rules, then sees the example illustrating them — matching standard instruction-then-demonstration prompting order.
• The worked example shows the exact output schema (value, label, reason), verbatim span preservation, the phone_number label, and the "o for zero" pattern — all novel behaviors that were previously unillustrated.

Note that asteier2026 indicated they were happy with the current ordering. If the team prefers to keep it post-<<EXAMPLE_BLOCK>>, adding the worked example alone (Step 2) would still substantially close the gap by giving the LLM a concrete demonstration of the disguised-identifier behavior.

[lipikaramaswamy]

This block conflicts with strict custom-label mode. For example, when entity_labels=["hostname", "ipv4"], the prompt first says to skip entities outside the allowed list, but this unconditional block then instructs the model to detect phone numbers, SSNs, credit cards, and names.

Can we generate this guidance conditionally? So we include each disguised-identifier example only when its canonical label is allowed, or when the default permissive label set is being used. A strict hostname/IP prompt should contain none of the phone, SSN, card, or name instructions.