Data model duplication

[ayushgupta704]

Description

This PR implements Content-Addressable Storage (CAS) for Data Models to resolve the linear storage bloat issue identified in
Previously, IntelOwl would create a new database row for every successful analyzer execution, even if the intelligence gathered was bit-for-bit identical to an existing record. This led to redundant data and unnecessary index growth. This change transitions the system to deduplicate intelligence at the point of ingestion.

Key Architectural Changes:

• Stable Hashing: Added a normalize_dict helper to recursively sort dictionary keys and lists. This ensures the fingerprint stays the same even if an external API returns JSON keys in a different order.
• Unique Fingerprinting: Introduced an indexed fingerprint field to the BaseDataModel. This acts as a unique ID for the actual intelligence content, allowing for fast deduplication lookups.
• Smart Deduplication: Updated the create_data_model logic to use a "get-or-create" pattern. If the intelligence is already in the database, the system now just links it instead of creating a duplicate row.
• Concurrency & Safety: Wrapped the logic in an atomic database transaction to prevent race conditions during parallel scans, while strictly maintaining existing return types to ensure zero regressions.

This directly addresses the # TODO in api_app/analyzers_manager/models.py:124.

Type of change

• New feature (non-breaking change which adds functionality)

Checklist

• I have read and understood the rules about [how to Contribute
  (https://intelowlproject.github.io/docs/IntelOwl/contribute/) to this project
• I have inserted the copyright banner at the start of the file.
• Linters (Ruff) gave 0 errors.
• I have added tests for the feature I solved. All tests (including the existing analyzer suite
  and new deduplication tests) passed with 0 errors.
  closes Reusing identical Data Models to prevent Data Bloat (Found a # TODO) #3450

[ayushgupta704]

Hi @mlodic,
I’ve implemented the content-addressable storage (CAS) to fix the data model bloat. Identical intelligence reports now correctly reuse the same records via fingerprints. I've verified it manually and all tests are passing. Ready for your review when you have a moment!

[mlodic]

can you please clean up the commit history and fix the conflicts. thank you

[ayushgupta704]

Hi @mlodic, I've rebased, resolved conflicts, and squashed the history into one clean commit and I've verified the CAS logic and migration safety in the Docker environment. Ready for review!

[mlodic]

there are neither unittests nor any other proof that the migration work as intended. Please show how the data model is changed in the database before and after the migration is applied

[ayushgupta704]

Hi @mlodic,
I’ve refactored this PR to address your feedback. It is now a single clean commit that includes the core logic, required migrations, and dedicated unit tests for CAS verification.

• Database Schema Proof: The migration adds the fingerprint column and a performance index to the base data models.

• Passing CAS Verification Tests: I've added a test suite (test_cas_deduplication.py) that confirms identical reports correctly deduplicate across different jobs.

Ready for review Thanks!

[mlodic]

the problem about adding data migrations is that no failing tests show up by running classic CI because the CI would work on a fresh environment.
This is a very risky change if not tested with already existing environments. The benefit of this could be easily destroyed by introducing an unwanted breaking change. Additional more comprehensive tests should be required. We can't merge this in the next release, we would need to wait a major like we will do for other critical PRs

[ayushgupta704]

Hi @mlodic, I totally get the worry about the CI missing potential migration bugs it’s actually why I spent some extra time making sure this is really safe for anyone with a big existing database.
I ended up writing a Migration Test (test_migrations.py) that fakes an old installation, seeds it with messy duplicate data, and then runs the upgrade. Here's a breakdown of the changes

• Migration Integrity Test: I added tests/api_app/data_model_manager/test_migrations.py. Instead of starting fresh, it forces the DB backward to 0012, seeds "dirty" duplicate data, and verifies that the 0013 upgrade merges everything and re-links all Reports/Jobs/Events safely without data loss.

• Scale & Resilience: The migration now uses bulk_update (500-row batches) and try-except blocks. This prevents table-locking and ensures a single malformed row won't crash the upgrade on large production environments.

• Bug Fix: Caught and fixed a potential AttributeError in the signal handlers that triggers during the record pruning process.

I’ve verified the full suite in Docker.Here is the proof from my manual Before vs After check:

• Before Migration: Created 2 identical IP records linked to 2 separate reports.

• After Migration: Redundant IP row deleted. Both reports successfully re-linked to the single remaining canonical record.

• All 6 tests are passing in my local setup.

I also completely understand the decision to wait for a major release.so let me know if you want me to check anything else!
Thanks

[mlodic]

this will be reconsidered after the next release in April