Lazy Tokenization of Meta Type Features Causes Inconsistent Encoding in Block Dataset Building

[koriyyy]

Problem Description:
When building datasets in blocks, lazy tokenization of "meta" type features leads to inconsistent global mapping across different blocks, causing data mismatches in downstream tasks.

Context:
In fuxictr/preprocess/build_dataset.py, the build_dataset function calls transform, which processes data in blocks when block_size > 0. Each block is processed in parallel using transform_block.

Issue Details:
The problem occurs in fuxictr/preprocess/feature_processor.py during the transformation of "meta" type features. For each block:

1. The tokenizer's encode_meta method is called
2. A new vocabulary is updated per block
3. The block data is encoded using this block-specific vocabulary
4. The encoded block is saved to disk

This results in:

• Different blocks having different token-to-ID mappings for the same meta feature
• No consistent global vocabulary across the entire dataset
• Potential mismatches when aggregating or analyzing data across blocks

Code References:

1. Block Processing (build_dataset.py):

def transform_block(feature_encoder, df_block, filename):
    df_block = feature_encoder.transform(df_block)  # Problem occurs here
    # ... save to parquet

def transform(feature_encoder, ddf, filename, block_size=0):
    if block_size > 0:
        # Process each block independently
        pool.apply_async(transform_block, ...)  # Each block gets its own vocab

2. Meta Feature Encoding (feature_processor.py):

# feature encoder's transform, here the ddf is block ddf
def transform(self, ddf):
    for feature, feature_spec in self.feature_map.features.items():
        if feature_type == "meta":
            tokenizer = self.processor_dict[feature + "::tokenizer"]
            ddf[feature] = tokenizer.encode_meta(col_series)  # Updates vocab per block

3. Tokenizer Implementation (tokenizer.py):

def encode_meta(self, series):
        word_counts = dict(series.value_counts())
        if len(self.vocab) == 0:
            self.build_vocab(word_counts)
        else: # update vocabs with block data
            self.update_vocab(word_counts.keys())
        series = series.map(lambda x: self.vocab.get(x, self.vocab["__OOV__"]))
        return series.values

Suggested Solutions:
Pre-build global vocabulary before block processing

[pudovikoff]

Hi!

I did not verify the previous bug, however I caught one more similar bug. I introduced the meta class field in my dataset config and I caught the attribute error: AttributeError: 'dict' object has no attribute 'most_common'.

The call stack is:

• build_dataset(feature_encoder, **params)
• fuxictr/preprocess/build_dataset.py", line 104, in build_dataset
  transform(feature_encoder, train_ddf, 'train', block_size=data_block_size)
• fuxictr/preprocess/build_dataset.py", line 75, in transform
  transform_block(feature_encoder, ddf, filename + ".parquet")
• preprocess/build_dataset.py", line 52, in transform_block
  df_block = feature_encoder.transform(df_block)
• fuxictr/preprocess/feature_processor.py", line 347, in transform
  ddf[feature] = tokenizer.encode_meta(col_series)
• fuxictr/preprocess/tokenizer.py", line 108, in encode_meta
  self.build_vocab(word_counts)
• fuxictr/preprocess/tokenizer.py", line 60, in build_vocab
  word_counts = word_counts.most_common()

So, I added Counter to the dict of the word_counts in the tokenizer about line 108 to

    def encode_meta(self, series):
        word_counts = dict(series.value_counts())
        if len(self.vocab) == 0:
            self.build_vocab(Counter(word_counts))
        else: # for considering meta data in test data
            self.update_vocab(word_counts.keys())
        series = series.map(lambda x: self.vocab.get(x, self.vocab["__OOV__"]))
        return series.values

That works for me and I suppose that it is a necessary fix for the meta type supoprt.