Make embeddings optional

[de-code]

For ScienceBeam Parser, all my current DL models do not use external embeddings. That greatly simplifies deployment and overhead (since it doesn't need to ship and prepare them).

When I evaluated it in the past, it only made a 0.5 point difference.

That was achieved via the CustomBidLSTM_CRF.

Relevant code snippet

        model_inputs = []
        lstm_inputs = []
        # build input, directly feed with word embedding by the data generator
        word_input = Input(
            # shape=(None, config.word_embedding_size),
            batch_shape=(input_batch_size, None, config.word_embedding_size),
            name='word_input'
        )
        model_inputs.append(word_input)
        lstm_inputs.append(word_input)

        # build character based embedding
        char_input = Input(
            # shape=(None, config.max_char_length),
            batch_shape=(input_batch_size, None, config.max_char_length),
            dtype='int32',
            name='char_input'
        )
        model_inputs.append(char_input)

        if config.char_embedding_size:
            assert config.char_vocab_size, 'config.char_vocab_size required'
            char_embeddings = TimeDistributed(Embedding(
                input_dim=config.char_vocab_size,
                output_dim=config.char_embedding_size,
                mask_zero=config.char_input_mask_zero,
                name='char_embeddings_embedding'
            ), name='char_embeddings')(char_input)

            chars = TimeDistributed(
                Bidirectional(LSTM(
                    config.num_char_lstm_units,
                    dropout=config.char_input_dropout,
                    recurrent_dropout=config.char_lstm_dropout,
                    return_sequences=False
                )),
                name='char_lstm'
            )(char_embeddings)
            lstm_inputs.append(chars)

Looking at the fullext model from the ScienceBeam Parser default config: 2021-05-11-delft-grobid-fulltext-biorxiv-10k-auto-v0.0.21-train-1986-e159.tar.gz

fulltext model config

{
    "model_name": "fulltext",
    "model_type": "CustomBidLSTM_CRF",
    "embeddings_name": null,
    "char_vocab_size": 1018,
    "case_vocab_size": 8,
    "char_embedding_size": 25,
    "num_char_lstm_units": 30,
    "max_char_length": 30,
    "features_vocabulary_size": 12,
    "features_indices": [
        9,
        10,
        11,
        12,
        13,
        14,
        15,
        22,
        25
    ],
    "features_embedding_size": 0,
    "features_lstm_units": 0,
    "max_sequence_length": 2000,
    "word_embedding_size": 0,
    "num_word_lstm_units": 200,
    "case_embedding_size": 5,
    "dropout": 0.5,
    "recurrent_dropout": 0.5,
    "use_char_feature": true,
    "use_crf": true,
    "fold_number": 1,
    "batch_size": 32,
    "use_ELMo": false,
    "use_BERT": false,
    "use_word_embeddings": false,
    "additional_token_feature_indices": null,
    "text_feature_indices": null,
    "concatenated_embeddings_token_count": null,
    "use_features": true,
    "max_feature_size": 20,
    "use_features_indices_input": false,
    "stateful": null,
    "model_version": 2,
    "features_map_to_index": null
}

The relevant parts are:

{
    "model_type": "CustomBidLSTM_CRF",
    "embeddings_name": null,
    "char_vocab_size": 1018,
    "case_vocab_size": 8,
    "char_embedding_size": 25,
    "num_char_lstm_units": 30,
    "max_char_length": 30,
    "word_embedding_size": 0,
    "num_word_lstm_units": 200,
    "case_embedding_size": 5,
    "use_char_feature": true,
    "use_word_embeddings": false,
}

Combining that with the code...

The regular embedding input would be this:

        word_input = Input(
            # shape=(None, config.word_embedding_size),
            batch_shape=(input_batch_size, None, config.word_embedding_size),
            name='word_input'
        )

But that is effectively disabled (see word_embedding_size=0 and use_word_embeddings=false).

Then we have:

        char_input = Input(
            # shape=(None, config.max_char_length),
            batch_shape=(input_batch_size, None, config.max_char_length),
            dtype='int32',
            name='char_input'
        )
        model_inputs.append(char_input)

So for the fulltext model max_char_length=30. That means it looks at up to 30 characters of each token.

this then follows:

        if config.char_embedding_size:
            assert config.char_vocab_size, 'config.char_vocab_size required'
            char_embeddings = TimeDistributed(Embedding(
                input_dim=config.char_vocab_size,
                output_dim=config.char_embedding_size,
                mask_zero=config.char_input_mask_zero,
                name='char_embeddings_embedding'
            ), name='char_embeddings')(char_input)

            chars = TimeDistributed(
                Bidirectional(LSTM(
                    config.num_char_lstm_units,
                    dropout=config.char_input_dropout,
                    recurrent_dropout=config.char_lstm_dropout,
                    return_sequences=False
                )),
                name='char_lstm'
            )(char_embeddings)
            lstm_inputs.append(chars)

Characters use their own vocab, according to the config it found 1018 different characters (not sure what this would do to Chinese characters).

They are then reduced to an internal embedding vector of char_embedding_size=25.