Add prompt_template parameter for per-sample prompt customization

[SokolovAnatoliy]

Add a prompt_template parameter to Task$new() that allows users to customize prompts on a per-sample basis using metadata columns from their dataset.

Motivation

Currently, if you want to vary the prompt context per sample (e.g., different instructions for each row), you need to build the prompts manually before creating the Task. This is a common need when:

• Running evaluations where each sample needs different instructions
• Testing how models respond to varied prompting strategies
• Parameterizing evaluations across different conditions

Solution (submitting as a PR)

Add a prompt_template argument to Task$new() that:

1. Uses {{ column_name }} syntax (via ellmer::interpolate()) to reference metadata columns
2. Automatically prepends the interpolated template to each sample's input
3. Works with both generate() and generate_structured() solvers

Examples

With generate()

library(vitals)                                                                 
library(ellmer)                                                                 
library(tibble)                                                                 
                                                                               
shapes_data <- tibble(                                                          
 input = c(                                                                    
   "The shapes are square, circle and rhombus",                                
   "The shapes are square, circle and rhombus",                                
   "The shapes are square, circle and rhombus"                                 
 ),                                                                            
 target = c("square", "circle", "rhombus"),                                    
 shape_to_pick = c("square", "circle", "rhombus")                              
)                                                                               
                                                                               
tsk <- Task$new(                                                                
 dataset = shapes_data,                                                        
 solver = generate(chat_anthropic(model = "claude-sonnet-4-5")),               
 scorer = detect_match("any"),                                                 
 prompt_template = "Always pick {{ shape_to_pick }}. Just return the single chosen shape as 1 word.",                                                       
 dir = tempdir()                                                               
)                                                                               
                                                                               
tsk$eval()                                                                      

Desired result:

tsk$get_samples()$result
[1] "square"  "circle"  "rhombus"

With generate_structured()

type_shape <- type_object(                                                      
 shape = type_string("The name of the shape that was picked")                  
)                                                                               
                                                                               
shapes_data <- tibble(                                                          
 input = c(                                                                    
   "The shapes are square, circle and rhombus",                                
   "The shapes are square, circle and rhombus",                                
   "The shapes are square, circle and rhombus"                                 
 ),                                                                            
 target = c("square", "circle", "rhombus"),                                    
 shape_to_pick = c("square", "circle", "rhombus")                              
)                                                                               
                                                                               
tsk_str <- Task$new(                                                            
 dataset = shapes_data,                                                        
 solver = generate_structured(                                                 
   solver_chat = chat_anthropic(model = "claude-sonnet-4-5"),                  
   type = type_shape                                                           
 ),                                                                            
 scorer = detect_match("any"),                                                 
 prompt_template = "Always pick {{ shape_to_pick }}. Just return the single chosen shape as 1 word.",                                                       
 dir = tempdir()                                                               
)                                                                               
                                                                               
tsk_str$eval()                                                                  

Desired result:

tsk_str$get_samples()$result
[1] "[{\"shape\":\"square\"}]"  "[{\"shape\":\"circle\"}]"  "[{\"shape\":\"rhombus\"}]"

Implementation Notes

• The template is interpolated at the Task level in $solve(), keeping solvers simple and unchanged
• Uses ellmer::interpolate() which uses {{ }} syntax (avoids conflicts with JSON/R code)
• The interpolated template is prepended to the input with \n\n separator
• When prompt_template = NULL (default), behavior is unchanged

[simonpcouch]

Thanks for the PR!

My reflex on first read is that this information may be able to live in the input itself. Note that input entries can be 1-row tibbles in the current package implementation, so if you wanted to have these per-sample customizations, you could have e.g.

shapes_data <- tibble(                                                          
 input = list(                                                                    
   tibble(shapes = "The shapes are square, circle and rhombus", pick = "square"),                                
   tibble(shapes = "The shapes are square, circle and rhombus", pick = "circle"),                                
   tibble(shapes = "The shapes are square, circle and rhombus", pick = "rhombus")                                 
 ),                                                                            
 target = c("square", "circle", "rhombus")                    
) 

Then, the solver could extract out input$shapes and input$pick as it needs, like:

  my_solver <- function(solver_chat = NULL) {
    chat <- solver_chat
    function(inputs, ..., solver_chat = chat) {
      ch <- if (is.function(solver_chat)) solver_chat() else solver_chat$clone()
      prompts <- lapply(inputs, function(inp) {
        paste0("Always pick ", inp$pick, ". Just return the single chosen shape.\n\n", inp$shapes)
      })
      res <- ellmer::parallel_chat(ch, prompts, ...)
      list(result = purrr::map_chr(res, \(c) c$last_turn()@text), solver_chat = res)
    }
  }

Would that suit your needs?

[SokolovAnatoliy]

Hi, yes, that works, and solved my issue. I didn't realize that the input could be a list of tibbles. SO in that case, for the custom structured data solver it would look something like this?

type_shape <- type_object(                                                   
  shape = type_string("The name of the shape that was picked")               
)                                                                            
                                                                             
shapes_data <- tibble(                                                       
  input = list(                                                              
    tibble(shapes = "The shapes are square, circle and rhombus", pick =      
"square"),                                                                   
    tibble(shapes = "The shapes are square, circle and rhombus", pick =      
"circle"),                                                                   
    tibble(shapes = "The shapes are square, circle and rhombus", pick =      
"rhombus")                                                                   
  ),                                                                         
  target = c("square", "circle", "rhombus")                                  
)                                                                            
                                                                             
generate_structured_with_context <- function(solver_chat = NULL, type = NULL)
 {                                                                           
  chat <- solver_chat                                                        
  type_spec <- type                                                          
                                                                             
  function(inputs, ..., solver_chat = chat, type = type_spec) {              
    if (is.function(solver_chat)) {                                          
      ch <- solver_chat()                                                    
    } else {                                                                 
      ch <- solver_chat$clone()                                              
    }                                                                        
                                                                             
    prompts <- purrr::map_chr(inputs, function(inp) {                        
      paste0("Always pick ", inp$pick, ". Just return the single chosen      
shape.\n\n", inp$shapes)                                                     
    })                                                                       
                                                                             
    res <- ellmer::parallel_chat_structured(ch, as.list(prompts), type =     
type, ...)                                                                   
                                                                             
    # Split results per input                                                
    res_per_input <- if (is.data.frame(res)) {                               
      lapply(seq_len(nrow(res)), function(i) res[i, , drop = FALSE])         
    } else if (is.list(res) && length(res) == length(inputs)) {              
      res                                                                    
    } else {                                                                 
      rep(list(res), length(inputs))                                         
    }                                                                        
                                                                             
    solver_chats <- purrr::map2(prompts, res_per_input, function(prompt,     
result) {                                                                    
      mock <- ch$clone()                                                     
      user_turn <- ellmer::UserTurn(                                         
        contents = list(ellmer::ContentText(prompt))                         
      )                                                                      
      result_json <- jsonlite::toJSON(result, auto_unbox = TRUE)             
      assistant_turn <- ellmer::AssistantTurn(                               
        contents = list(ellmer::ContentText(as.character(result_json)))      
      )                                                                      
      mock$set_turns(list(user_turn, assistant_turn))                        
      mock                                                                   
    })                                                                       
                                                                             
    result_strings <- purrr::map_chr(res_per_input, function(x) {            
      as.character(jsonlite::toJSON(x, auto_unbox = TRUE))                   
    })                                                                       
                                                                             
    list(                                                                    
      result = result_strings,                                               
      solver_chat = solver_chats,                                            
      solver_metadata = res_per_input                                        
    )                                                                        
  }                                                                          
}                                                                            
                                                                             
tsk <- Task$new(                                                             
  dataset = shapes_data,                                                     
  solver = generate_structured_with_context(                                 
    solver_chat = chat_anthropic(model = "claude-sonnet-4-5"),               
    type = type_shape                                                        
  ),                                                                         
  scorer = detect_match("any"),                                              
  dir = tempdir()                                                            
)                                                                            
                                                                             
tsk$eval()                                                                   
tsk$get_samples()$result                                                     
tsk$get_samples()$solver_metadata   

Thanks for the awesome package, I am having fun learning more about it and applying it in my work!

Is this clarification worth adding to your guides? I think this is a very helpful example!

[simonpcouch]

Good point, yes! This ought to be documented--thanks for that suggestion.