-
Notifications
You must be signed in to change notification settings - Fork 1
adds new audiobook test and reorganizes tests #2838
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Changes from all commits
Commits
Show all changes
45 commits
Select commit
Hold shift + click to select a range
ba1882c
adds new audiobook test and reorganizes tests
omehes 715b991
resolves conflict
omehes e81a45b
Merge branch 'main' into audio
staxly[bot] 3385ae7
Merge branch 'main' into audio
staxly[bot] 5934bec
Merge branch 'main' into audio
staxly[bot] 0be7b3f
Merge branch 'main' into audio
staxly[bot] c4679ac
Merge branch 'main' into audio
staxly[bot] 93e2347
Merge branch 'main' into audio
staxly[bot] d769f69
Merge branch 'main' into audio
staxly[bot] 0e5d4d1
Merge branch 'main' into audio
staxly[bot] dbb8bad
Merge branch 'main' into audio
staxly[bot] f19f344
Merge branch 'main' into audio
staxly[bot] fd22385
Merge branch 'main' into audio
staxly[bot] 94cd82f
Merge branch 'main' into audio
staxly[bot] 1ac76e4
Merge branch 'main' into audio
staxly[bot] 118e904
Merge branch 'main' into audio
staxly[bot] 1b2f99c
Merge branch 'main' into audio
staxly[bot] 4b867ce
Merge branch 'main' into audio
staxly[bot] 920050a
Merge branch 'main' into audio
staxly[bot] 3ff9631
Merge branch 'main' into audio
staxly[bot] 3af6b68
Merge branch 'main' into audio
staxly[bot] 8ecd878
Merge branch 'main' into audio
staxly[bot] 16fbbee
Merge branch 'main' into audio
staxly[bot] 72727f1
Merge branch 'main' into audio
staxly[bot] c750497
Merge branch 'main' into audio
staxly[bot] ccfdb7f
Merge branch 'main' into audio
staxly[bot] 28500c8
Merge branch 'main' into audio
staxly[bot] d12ff6b
Merge branch 'main' into audio
staxly[bot] 1101efc
Merge branch 'main' into audio
staxly[bot] 6a80025
Merge branch 'main' into audio
staxly[bot] 6c4d784
Merge branch 'main' into audio
staxly[bot] 4f4110e
Merge branch 'main' into audio
staxly[bot] aef4bdc
Merge branch 'main' into audio
staxly[bot] d2e4b42
Merge branch 'main' into audio
staxly[bot] 0415241
Merge branch 'main' into audio
staxly[bot] f8771b1
Merge branch 'main' into audio
staxly[bot] 68481ca
Merge branch 'main' into audio
staxly[bot] 9c0bffc
Merge branch 'main' into audio
staxly[bot] ec93ae8
Merge branch 'main' into audio
staxly[bot] 1a3d456
Merge branch 'main' into audio
staxly[bot] ede8077
Merge branch 'main' into audio
staxly[bot] ca27609
Merge branch 'main' into audio
staxly[bot] 221897b
Merge branch 'main' into audio
staxly[bot] a53799b
Merge branch 'main' into audio
staxly[bot] 8dfacc6
Merge branch 'main' into audio
staxly[bot] File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file was deleted.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,144 @@ | ||
| import pytest | ||
| import os | ||
| import shutil | ||
| import zipfile | ||
| import textwrap | ||
| from deepdiff import DeepDiff | ||
| import docx2txt | ||
|
|
||
| """ | ||
| Compares docx files. To run: | ||
| 1. run two docx jobs in corgi | ||
| 2. download the docx zip files or docx files from corgi (by default to Downloads folder on Mac) | ||
| - make sure that BASE_TO_DIR exists and contains 2 subdirectories named 0 and 1 | ||
| 3. make sure that BASE_FROM_DIR variable is set correctly and files are copied there from Downloads folder | ||
| 4. run 'pytest -k test_compare_zip_or_docx_files.py docx-tools > docx_diffs.txt' | ||
|
|
||
| Note, that the test accepts two .zip files from corgi docx job, two .docx files (or one of each) | ||
|
|
||
| Latest update on March 31st, 2026 | ||
| """ | ||
|
|
||
| HOME_DIR = os.path.expanduser("~") | ||
| BASE_FROM_DIR = f"{HOME_DIR}/Downloads/2pdfs" | ||
| BASE_TO_DIR = f"{os.getcwd()}/docx_old_new" | ||
| MAX_WIDTH = 120 | ||
|
|
||
|
|
||
| def unzip_file(zip_path, extract_to): | ||
| """Unzips a zip file into extract_to directory.""" | ||
| with zipfile.ZipFile(zip_path) as z: | ||
| z.extractall(extract_to) | ||
|
|
||
|
|
||
| def get_docx_texts_from_dir(directory): | ||
| """Returns a dict of {book_name: {filename: text}} for all docx files in subdirs.""" | ||
| result = {} | ||
| for sub in os.listdir(directory): | ||
| sub_path = os.path.join(directory, sub) | ||
| if os.path.isdir(sub_path): | ||
| result[sub] = { | ||
| f: docx2txt.process(os.path.join(sub_path, f)) | ||
| for f in os.listdir(sub_path) | ||
| if f.endswith(".docx") | ||
| } | ||
| return result | ||
|
|
||
|
|
||
| def get_docx_texts_from_file(docx_path): | ||
| """Returns a dict for a single docx file.""" | ||
| filename = os.path.basename(docx_path) | ||
| return {filename: docx2txt.process(docx_path)} | ||
|
|
||
|
|
||
| def prepare_files(file_path, dest_dir): | ||
| """ | ||
| Copies and extracts zip files or copies docx files to dest_dir. | ||
| Returns a dict of docx texts. | ||
| """ | ||
| file_path = str(file_path) | ||
| dest_dir = str(dest_dir) | ||
|
|
||
| os.makedirs(dest_dir, exist_ok=True) | ||
|
|
||
| if file_path.endswith(".zip"): | ||
| dest_path = os.path.join(dest_dir, os.path.basename(file_path)) | ||
| shutil.copy(file_path, dest_path) | ||
| unzip_file(dest_path, dest_dir) | ||
| return get_docx_texts_from_dir(dest_dir) | ||
|
|
||
| elif file_path.endswith(".docx"): | ||
| dest_path = os.path.join(dest_dir, os.path.basename(file_path)) | ||
| shutil.copy(file_path, dest_path) | ||
| return get_docx_texts_from_file(dest_path) | ||
|
|
||
| else: | ||
| pytest.fail( | ||
| f"Unsupported file type: {file_path}. Only .zip and .docx are supported." | ||
| ) | ||
|
|
||
|
|
||
| def print_diffs(diffs): | ||
| for diff_type, changes in diffs.items(): | ||
| # Diff type header | ||
| print(f"\n{'#' * 60}") | ||
| print(f" {diff_type.replace('_', ' ').upper()}") | ||
| print(f"{'#' * 60}") | ||
|
|
||
| for key, value in changes.items(): | ||
| # Extract book/file name from key for readability | ||
| print(f"\n {'─' * 40}") | ||
| print(f" LOCATION: {key}") | ||
| print(f" {'─' * 40}") | ||
|
|
||
| # Handle old/new values separately if present | ||
| if hasattr(value, "get"): | ||
| old_val = str(value.get("old_value", "")) | ||
| new_val = str(value.get("new_value", "")) | ||
| if old_val: | ||
| print(f"\n OLD VALUE:") | ||
| for line in textwrap.wrap(old_val, width=MAX_WIDTH): | ||
| print(f" {line}") | ||
| if new_val: | ||
| print(f"\n NEW VALUE:") | ||
| for line in textwrap.wrap(new_val, width=MAX_WIDTH): | ||
| print(f" {line}") | ||
| else: | ||
| print(f"\n CHANGE:") | ||
| for line in textwrap.wrap(str(value), width=MAX_WIDTH): | ||
| print(f" {line}") | ||
|
|
||
|
|
||
| def test_compare_zip_or_docx_files(): | ||
| # Part 1: Find zip or docx files | ||
| all_files = [ | ||
| f | ||
| for f in os.listdir(BASE_FROM_DIR) | ||
| if f.endswith(".zip") or f.endswith(".docx") | ||
| ] | ||
|
|
||
| if len(all_files) != 2: | ||
| pytest.fail( | ||
| f"Two files required (.zip or .docx): {len(all_files)} file(s) available" | ||
| ) | ||
|
|
||
| file1 = os.path.join(BASE_FROM_DIR, all_files[0]) | ||
| file2 = os.path.join(BASE_FROM_DIR, all_files[1]) | ||
|
|
||
| print(f"\nFile 1: {all_files[0]}") | ||
| print(f"File 2: {all_files[1]}") | ||
|
|
||
| # Part 2: Prepare and extract text | ||
| dest_dir_0 = os.path.join(BASE_TO_DIR, "0") | ||
| dest_dir_1 = os.path.join(BASE_TO_DIR, "1") | ||
|
|
||
| dict_old = prepare_files(file1, dest_dir_0) | ||
| dict_new = prepare_files(file2, dest_dir_1) | ||
|
|
||
| # Part 3: Compare and print diffs | ||
| diffs = DeepDiff(dict_new, dict_old, verbose_level=2) | ||
|
|
||
| if not diffs: | ||
| print("------>>>>> NO DIFFERENCES <<<<<------") | ||
| else: | ||
| print_diffs(diffs) | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The test function is missing an assertion or failure case. When differences are found in the docx files, the test should fail rather than just printing the diffs. The test currently prints the diffs and exits successfully even if there are differences, which defeats the purpose of a test.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
no need for this