Chat Template Processing Error with Orca AgentInstruct Dataset in Axolotl

[abdul-456]

Dataset: microsoft/orca-agentinstruct-1M-v1
Model: meta-llama/Llama-3.1-8B-Instruct
Chat Template: llama3

Problem Description:
When attempting to load and process the Orca AgentInstruct dataset, I'm encountering a KeyError: 'conversations' during dataset preparation. How can I resolve it?

The below is my config file:

import yaml

yaml_string = """
base_model: meta-llama/Llama-3.1-8B-Instruct
model_type: LlamaForCausalLM
tokenizer_type: AutoTokenizer

load_in_8bit: true
load_in_4bit: false
strict: false

chat_template: llama3
datasets:

• path: microsoft/orca-agentinstruct-1M-v1
  split: rag
  type: chat_template
  conversation_column: messages
  message_field_role: role
  message_field_content: content
  dataset_prepared_path: last_run_prepared
  val_set_size: 0.05
  output_dir: ./outputs/orca-llama-lora

sequence_len: 2048
sample_packing: true
eval_sample_packing: true
pad_to_sequence_len: true

adapter: lora
lora_model_dir:
lora_r: 32
lora_alpha: 16
lora_dropout: 0.05
lora_target_linear: true
lora_fan_in_fan_out:
lora_modules_to_save:

• embed_tokens
• lm_head

wandb_project:
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:

gradient_accumulation_steps: 2
micro_batch_size: 1
num_epochs: 1
optimizer: paged_adamw_8bit
lr_scheduler: cosine
learning_rate: 2e-5

train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false

gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
logging_steps: 1
xformers_attention:
flash_attention: false
sdp_attention: true

warmup_steps: 1
max_steps: 25
evals_per_epoch: 1
eval_table_size:
saves_per_epoch: 1
debug:
deepspeed:
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:
pad_token: <|end_of_text|>
"""

[NanoCode012]

Hello!

datasets:
  - path: microsoft/orca-agentinstruct-1M-v1
    type: chat_template
    split: rag
-    conversation_column: messages
+    field_messages: messages
    message_field_role: role
    message_field_content: content

We don't have a conversation_column field, it's called field_messages, although the default should work for this dataset.

---

I took a look at this split in particular. Their dataset is a bit weird.

Instead of messages: List[dict] (list of conversation), it's messages: str (stringified list).

https://huggingface.co/datasets/microsoft/orca-agentinstruct-1M-v1/viewer/default/rag?row=0

You'll need to do a small patch to axolotl if you want to get this to work.

axolotl/src/axolotl/prompt_strategies/chat_template.py

Lines 431 to 433 in 8fb72cb

	]
	for message in prompt[self.messages]:
	turn = {

Add this above the for loop

        if isinstance(prompt[self.messages], str):
            parsed_messages = json.loads(prompt[self.messages])
        else:
            parsed_messages = prompt[self.messages]
        assert isinstance(parsed_messages, list), f"Expected list, got {type(parsed_messages)}"
        for message in parsed_messages:

I would also recommend adding:

datasets:
    - path: ...
      drop_system_message: true

to drop the system message as I notice a lot are empty (didn't check if all are empty) and would cause Axolotl to warn out of these empty turns.

[abdul-456]

Thanks for the suggestions, it worked.

[NanoCode012]

Thanks for verifying. Will close this thread.