Skip to content

Commit e08e56f

Browse files
committed
chore: replaced bitshift by exponentiation for better readability in tokenized_file_writer.py
1 parent 6a17097 commit e08e56f

1 file changed

Lines changed: 1 addition & 1 deletion

File tree

src/modalities/dataloader/preprocessing/tokenization/tokenized_file_writer.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -93,7 +93,7 @@ def _write_data_segment(
9393
else:
9494
raise ValueError("Currently only support token byte sizes of 1, 2, and 4.")
9595

96-
max_allowed = (1 << (8 * token_size_in_bytes)) - 1
96+
max_allowed = 2 ** (8 * token_size_in_bytes) - 1
9797

9898
samples: list[bytes] = []
9999
index_list: list[tuple[int, int]] = []

0 commit comments

Comments
 (0)