DAGE-107: Add a script to download bbc news dataset and support batch indexing in docker-services#250
Conversation
nseidan
commented
Oct 23, 2025
nseidan
commented
- Added a script to download a large bbc news dataset
- Added support for batch indexing in docker-services solr-init.py
- Updated READMe
nseidan
changed the title
nicolo-rinaldi
left a comment
nicolo-rinaldi
left a comment
There was a problem hiding this comment.
Sorry for the typos in the README and thanks for spotting them. I left minor comments
| truncated: str = txt[:CONTENT_MAX_LEN] | ||
|
|
||
| next_dot = txt.find('.', CONTENT_MAX_LEN) | ||
| if next_dot != -1 and next_dot - CONTENT_MAX_LEN < 200: |
There was a problem hiding this comment.
In this case, if we find a dot after the max len can become 1200, right?
Maybe we can check the last dot backward, so we keep the content len under CONTENT_MAX_LEN
There was a problem hiding this comment.
right, its max is 1200, because the reason is i noticed there is a long sentence which sometimes reaches at least 1100 and in case if we remove the sentence which reaches the last sentence after reaching 1000 chars and the content becomes around 800. That's why I set upper bound to 1200. But on average I saw about 1100 chars per content. Anyways, it is changeable.
There was a problem hiding this comment.
For me is fine like it is right now, indeed I approved. So feel free to keep it like this. The only thing is that when I read the code, CONTENT_MAX_LEN is set to 1000 but the "actual one" is 1200 (i.e., I often end up with more than 1k chars), since we are addressing the problem of finding the first dot after 1000. But this is just a little thing. I don't want to block anything
There was a problem hiding this comment.
updated the code for readability
| parser.add_argument('--filename', type=str, default='dataset.json', help='Output filename') | ||
| args = parser.parse_args() | ||
|
|
||
| months = ['2025-06', '2025-05', '2025-04', '2025-03', '2025-02', '2025-01', |
There was a problem hiding this comment.
How about something like
months = [
f"{date.year}-{date.month}"
for date in range (<something from X to Y increasing every month>)
]
3 participants
