Skip to content

Commit be8dcb8

Browse files
PringledPringled
committed
Moved benchmarks
1 parent 4c4c496 commit be8dcb8

2 files changed

Lines changed: 83 additions & 60 deletions

File tree

‎README.md‎

Lines changed: 2 additions & 60 deletions
Original file line numberDiff line numberDiff line change
@@ -405,67 +405,9 @@ representative_texts = semhash.self_find_representative().selected
405405

406406
## Benchmarks
407407

408-
We've benchmarked SemHash on a variety of datasets to measure the deduplication performance and speed. The benchmarks were run with the following setup:
409-
- The benchmarks were all run on CPU
410-
- The benchmarks were all run with the default ANN backend (usearch)
411-
- The used encoder is the default encoder ([potion-base-8M](https://huggingface.co/minishlab/potion-base-8M)).
412-
- The timings include the encoding time, index building time, and deduplication time.
413-
### Train Deduplication Benchmark
414-
415-
| Dataset | Original Train Size | Deduplicated Train Size | % Removed | Deduplication Time (s) |
416-
|----------------------|----------------------|--------------------------|------------|--------------------------|
417-
| bbc | 1225 | 1144 | 6.61 | 0.57 |
418-
| senteval_cr | 3012 | 2990 | 0.73 | 0.14 |
419-
| tweet_sentiment_extraction | 27481 | 26695 | 2.86 | 1.77 |
420-
| emotion | 16000 | 15695 | 1.91 | 0.77 |
421-
| amazon_counterfactual | 5000 | 4992 | 0.16 | 0.33 |
422-
| ag_news | 120000 | 106921 | 10.90 | 5.20 |
423-
| enron_spam | 31716 | 20540 | 35.24 | 2.03 |
424-
| subj | 8000 | 7990 | 0.12 | 0.63 |
425-
| sst5 | 8544 | 8526 | 0.21 | 0.58 |
426-
| 20_newgroups | 11314 | 10684 | 5.57 | 0.73 |
427-
| hatespeech_offensive | 22783 | 22090 | 3.04 | 0.92 |
428-
| ade | 17637 | 15718 | 10.88 | 0.73 |
429-
| imdb | 25000 | 24830 | 0.68 | 1.76 |
430-
| massive_scenario | 11514 | 9366 | 18.66 | 0.47 |
431-
| student | 117519 | 63856 | 45.66 | 8.80 |
432-
| squad_v2 | 130319 | 109698 | 15.82 | 8.81 |
433-
| wikitext | 1801350 | 884645 | 50.89 | 83.53 |
434-
435-
436-
### Train/Test Deduplication Benchmark
437-
438-
| Dataset | Train Size | Test Size | Deduplicated Test Size | % Removed | Deduplication Time (s) |
439-
|----------------------|--------------|--------------|--------------------------|------------|--------------------------|
440-
| bbc | 1225 | 1000 | 870 | 13.00 | 0.71 |
441-
| senteval_cr | 3012 | 753 | 750 | 0.40 | 0.13 |
442-
| tweet_sentiment_extraction | 27481 | 3534 | 3412 | 3.45 | 1.53 |
443-
| emotion | 16000 | 2000 | 1926 | 3.70 | 0.65 |
444-
| amazon_counterfactual | 5000 | 5000 | 4990 | 0.20 | 0.51 |
445-
| ag_news | 120000 | 7600 | 6198 | 18.45 | 3.74 |
446-
| enron_spam | 31716 | 2000 | 1060 | 47.00 | 1.94 |
447-
| subj | 8000 | 2000 | 1999 | 0.05 | 0.62 |
448-
| sst5 | 8544 | 2210 | 2205 | 0.23 | 0.59 |
449-
| 20_newgroups | 11314 | 7532 | 7098 | 5.76 | 2.25 |
450-
| hatespeech_offensive | 22783 | 2000 | 1925 | 3.75 | 0.77 |
451-
| ade | 17637 | 5879 | 4952 | 15.77 | 0.81 |
452-
| imdb | 25000 | 25000 | 24795 | 0.82 | 2.81 |
453-
| massive_scenario | 11514 | 2974 | 2190 | 26.36 | 0.46 |
454-
| student | 117519 | 5000 | 2393 | 52.14 | 3.78 |
455-
| squad_v2 | 130319 | 11873 | 11863 | 0.08 | 7.13 |
456-
| wikitext | 1801350 | 4358 | 2139 | 50.92 | 40.32 |
457-
458-
459-
As can be seen, SemHash is extremely fast, and scales to large datasets with millions of records. There are some notable examples of train/test leakage, such as `enron_spam` and `student`, where the test dataset contains a significant amount of semantic overlap with the training dataset.
460-
461-
### Reproducing the Benchmarks
462-
463-
To run the benchmarks yourself, you can use the following command (assuming you have the `datasets` library installed):
408+
SemHash is extremely fast and scales to large datasets with millions of records. We've benchmarked both single-dataset deduplication and train/test deduplication across a variety of datasets. For example, deduplicating 1.8M records takes only ~83 seconds on CPU.
464409

465-
```bash
466-
python -m benchmarks.run_benchmarks
467-
```
468-
Optionally, the datasets can be updated in the [data.py](https://github.com/MinishLab/semhash/blob/main/benchmarks/data.py) file.
410+
For detailed benchmark results including performance metrics across 17 datasets, see the [benchmarks directory](benchmarks/README.md).
469411

470412
## License
471413

‎benchmarks/README.md‎

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
# SemHash Benchmarks
2+
3+
This directory contains the benchmarking code and results for SemHash. The benchmarks measure deduplication performance and speed across a variety of datasets.
4+
5+
## Setup
6+
7+
All benchmarks were run with the following configuration:
8+
- **CPU-only**: All benchmarks run on CPU (no GPU acceleration)
9+
- **ANN backend**: Default backend (USearch)
10+
- **Encoder**: Default encoder ([potion-base-8M](https://huggingface.co/minishlab/potion-base-8M))
11+
- **Timing**: Includes encoding time, index building time, and deduplication time
12+
13+
## Results
14+
15+
### Train Deduplication Benchmark
16+
17+
This benchmark measures the performance of deduplicating within a single training dataset.
18+
19+
| Dataset | Original Train Size | Deduplicated Train Size | % Removed | Deduplication Time (s) |
20+
|----------------------|----------------------|--------------------------|------------|--------------------------|
21+
| bbc | 1225 | 1144 | 6.61 | 0.57 |
22+
| senteval_cr | 3012 | 2990 | 0.73 | 0.14 |
23+
| tweet_sentiment_extraction | 27481 | 26695 | 2.86 | 1.77 |
24+
| emotion | 16000 | 15695 | 1.91 | 0.77 |
25+
| amazon_counterfactual | 5000 | 4992 | 0.16 | 0.33 |
26+
| ag_news | 120000 | 106921 | 10.90 | 5.20 |
27+
| enron_spam | 31716 | 20540 | 35.24 | 2.03 |
28+
| subj | 8000 | 7990 | 0.12 | 0.63 |
29+
| sst5 | 8544 | 8526 | 0.21 | 0.58 |
30+
| 20_newgroups | 11314 | 10684 | 5.57 | 0.73 |
31+
| hatespeech_offensive | 22783 | 22090 | 3.04 | 0.92 |
32+
| ade | 17637 | 15718 | 10.88 | 0.73 |
33+
| imdb | 25000 | 24830 | 0.68 | 1.76 |
34+
| massive_scenario | 11514 | 9366 | 18.66 | 0.47 |
35+
| student | 117519 | 63856 | 45.66 | 8.80 |
36+
| squad_v2 | 130319 | 109698 | 15.82 | 8.81 |
37+
| wikitext | 1801350 | 884645 | 50.89 | 83.53 |
38+
39+
### Train/Test Deduplication Benchmark
40+
41+
This benchmark measures the performance of deduplicating a test dataset against a training dataset (detecting train/test leakage).
42+
43+
| Dataset | Train Size | Test Size | Deduplicated Test Size | % Removed | Deduplication Time (s) |
44+
|----------------------|--------------|--------------|--------------------------|------------|--------------------------|
45+
| bbc | 1225 | 1000 | 870 | 13.00 | 0.71 |
46+
| senteval_cr | 3012 | 753 | 750 | 0.40 | 0.13 |
47+
| tweet_sentiment_extraction | 27481 | 3534 | 3412 | 3.45 | 1.53 |
48+
| emotion | 16000 | 2000 | 1926 | 3.70 | 0.65 |
49+
| amazon_counterfactual | 5000 | 5000 | 4990 | 0.20 | 0.51 |
50+
| ag_news | 120000 | 7600 | 6198 | 18.45 | 3.74 |
51+
| enron_spam | 31716 | 2000 | 1060 | 47.00 | 1.94 |
52+
| subj | 8000 | 2000 | 1999 | 0.05 | 0.62 |
53+
| sst5 | 8544 | 2210 | 2205 | 0.23 | 0.59 |
54+
| 20_newgroups | 11314 | 7532 | 7098 | 5.76 | 2.25 |
55+
| hatespeech_offensive | 22783 | 2000 | 1925 | 3.75 | 0.77 |
56+
| ade | 17637 | 5879 | 4952 | 15.77 | 0.81 |
57+
| imdb | 25000 | 25000 | 24795 | 0.82 | 2.81 |
58+
| massive_scenario | 11514 | 2974 | 2190 | 26.36 | 0.46 |
59+
| student | 117519 | 5000 | 2393 | 52.14 | 3.78 |
60+
| squad_v2 | 130319 | 11873 | 11863 | 0.08 | 7.13 |
61+
| wikitext | 1801350 | 4358 | 2139 | 50.92 | 40.32 |
62+
63+
## Key Findings
64+
65+
SemHash is extremely fast and scales to large datasets with millions of records. Some notable findings include:
66+
67+
- **Speed**: Deduplication is fast even for large datasets (e.g., 1.8M records in ~83 seconds)
68+
- **Train/Test Leakage**: Several datasets show significant train/test overlap:
69+
- `enron_spam`: 47% of test data overlaps with training data
70+
- `student`: 52% of test data overlaps with training data
71+
- `wikitext`: 51% of test data overlaps with training data
72+
73+
## Running the Benchmarks
74+
75+
To run the benchmarks yourself:
76+
77+
```bash
78+
python -m benchmarks.run_benchmarks
79+
```
80+
81+
The datasets can be customized by editing `benchmarks/data.py`.

0 commit comments

Comments
 (0)