Integrate evaluation capabilities for PII Column Identification in tables or JSONs with presidio-structured

[ebotiab]

I would like to propose the enhancement of the presidio-research repository by introducing functionalities that enable the evaluation of how accurately columns in a table or a JSON containing Personally Identifiable Information (PII) are identified, utilizing the capabilities of the newly introduced package presidio-structured.

A starting point could be simply assess the precision, recall, and F1 score of PII column identification.

[omri374]

This could be a great community contribution. If anyone's interested, feel free to raise a PR and we'll happily support. If anyone's interested in discussing this, please reach out to presidio@microsoft.com

[sahaj24]

Hi @omri374 👋

I'd like to work on this enhancement! I plan to add functionality to evaluate how accurately PII columns are identified in structured data like tables or JSON, using the presidio-structured package. My goal is to start with calculating precision, recall, and F1 score for column-level PII detection.

Let me know if you have any preferences or suggestions before I begin.

Thanks!

[omri374]

Hi @sahaj24, thanks! this is great. I would suggest to start with a high level design / ADR document. This would allow everyone to provide feedback prior to writing the actual code. Some things to think about:

• How this integrates with the rest of the code in presidio-evaluator
• How are metrics calculated? in PII, metrics are often different from other tasks. One example is a higher focus on recall vs. precision
• Which dataset(s) will you be testing this on? How is ground-truth data generated?