Skip to content

Commit 6a14cb2

Browse files
committed
Added caching for selected_with_duplicates, fixes rethreshold bug
1 parent b190260 commit 6a14cb2

1 file changed

Lines changed: 36 additions & 43 deletions

File tree

README.md

Lines changed: 36 additions & 43 deletions
Original file line numberDiff line numberDiff line change
@@ -124,11 +124,45 @@ filtered_texts = semhash.self_filter_outliers().selected
124124
representative_texts = semhash.self_find_representative().selected
125125
```
126126

127-
The `deduplicate` and `self_deduplicate` functions return a [DeduplicationResult](https://github.com/MinishLab/semhash/blob/main/semhash/datamodels.py#L30). This object stores the deduplicated corpus, a set of duplicate object (along with the objects that caused duplication), and several useful functions to further inspect the deduplication result. Examples of how these functions can be used can be found in the [usage](#usage) section.
127+
The `deduplicate` and `self_deduplicate` functions return a [DeduplicationResult](https://github.com/MinishLab/semhash/blob/main/semhash/datamodels.py#L58). This object stores the deduplicated corpus, a set of duplicate object (along with the objects that caused duplication), and several useful functions to further inspect the deduplication result.
128128

129-
The `filter_outliers`, `self_filter_outliers`, `find_representative`, and `self_find_representative` functions return a [FilterResult](https://github.com/MinishLab/semhash/blob/main/semhash/datamodels.py#106). This object stores the found outliers/representative samples.
129+
The `filter_outliers`, `self_filter_outliers`, `find_representative`, and `self_find_representative` functions return a [FilterResult](https://github.com/MinishLab/semhash/blob/main/semhash/datamodels.py#179). This object stores the found outliers/representative samples.
130130

131131
For both the `DeduplicationResult` and `FilterResult` objects, you can easily view the filtered records with the `selected` attribute (e.g. to view outliers: `outliers = semhash.self_filter_outliers().filtered`)
132+
133+
### Inspecting Deduplication Results
134+
135+
The `DeduplicationResult` object provides powerful tools for understanding and refining your deduplication:
136+
137+
```python
138+
from datasets import load_dataset
139+
from semhash import SemHash
140+
141+
# Load and deduplicate a dataset
142+
texts = load_dataset("ag_news", split="train")["text"]
143+
semhash = SemHash.from_records(records=texts)
144+
result = semhash.self_deduplicate()
145+
146+
# Access deduplicated and duplicate records
147+
deduplicated_texts = result.selected
148+
duplicate_texts = result.filtered
149+
150+
# View deduplication statistics
151+
print(f"Duplicate ratio: {result.duplicate_ratio}")
152+
print(f"Exact duplicate ratio: {result.exact_duplicate_ratio}")
153+
154+
# Find edge cases to tune your threshold
155+
least_similar = result.get_least_similar_from_duplicates(n=5)
156+
157+
# Adjust threshold without re-deduplicating
158+
result.rethreshold(0.95)
159+
160+
# View each kept record with its duplicate cluster
161+
for item in result.selected_with_duplicates:
162+
print(f"Kept: {item.record}")
163+
print(f"Duplicates: {item.duplicates}") # List of (duplicate_text, similarity_score)
164+
```
165+
132166
## Main Features
133167

134168
- **Fast**: SemHash uses [model2vec](https://github.com/MinishLab/model2vec) to embed texts and [vicinity](https://github.com/MinishLab/vicinity) to perform similarity search, making it extremely fast.
@@ -231,47 +265,6 @@ representative_records = semhash.self_find_representative().selected
231265

232266
</details>
233267

234-
<details>
235-
<summary> DeduplicationResult functionality </summary>
236-
<br>
237-
238-
The `DeduplicationResult` object returned by the `deduplicate` and `self_deduplicate` functions contains several useful functions to inspect the deduplication result. The following code snippet shows how to use these functions:
239-
240-
```python
241-
from datasets import load_dataset
242-
from semhash import SemHash
243-
244-
# Load a dataset to deduplicate
245-
texts = load_dataset("ag_news", split="train")["text"]
246-
247-
# Initialize a SemHash instance
248-
semhash = SemHash.from_records(records=texts)
249-
250-
# Deduplicate the texts
251-
deduplication_result = semhash.self_deduplicate()
252-
253-
# Check the deduplicated texts
254-
deduplication_result.selected
255-
# Check the duplicates
256-
deduplication_result.filtered
257-
# See what percentage of the texts were duplicates
258-
deduplication_result.duplicate_ratio
259-
# See what percentage of the texts were exact duplicates
260-
deduplication_result.exact_duplicate_ratio
261-
262-
# Get the least similar text from the duplicates. This is useful for finding the right threshold for deduplication.
263-
least_similar = deduplication_result.get_least_similar_from_duplicates()
264-
265-
# Rethreshold the duplicates. This allows you to instantly rethreshold the duplicates with a new threshold without having to re-deduplicate the texts.
266-
deduplication_result.rethreshold(0.95)
267-
268-
# View selected records along with their duplicates.
269-
# This is the opposite of the `filtered` attribute, which shows for every duplicate the record that caused it.
270-
deduplication_result.selected_with_duplicates
271-
```
272-
273-
</details>
274-
275268
<details>
276269
<summary> Using custom encoders </summary>
277270
<br>

0 commit comments

Comments
 (0)