Feature Request: Testing externally generated programs

[wistuba]

The readme claims this is possible, but very strong assumptions are made about the input. For example, we assume that rust code is provided in a singe file called main.rs on the top-level. Docker files require an update to move all files with the code folder.

[mveroe]

Hey, thanks for raising this request.

Currently, you will have to follow the format expected by the BaxBench testing modules. This means that all the code has to be in the code folder under the correct model, prompt types, temperature, etc. settings path. Additionally, the entrypoint of the server as specified in the docker image (in the envs) has to be respected, as this is the command that will launch the server in any case. Apart from this, any extra or non-conforming files in the code directory should already be copied over into the container. Let me know if you experience issues here.

As in for testing general formats, I am not sure if we can anticipate any file structure. If you have suggestions here, please let us know.

Thanks!

[wistuba]

Thanks for the response. What I've tried to say is that some dockerfiles don't seem to copy paste all files over, e.g., https://github.com/logic-star-ai/baxbench/blob/main/src/env/rust.py#L24

[mveroe]

Ah, I see, thanks for the clarification and sorry for the initial confusion. I see the usecase. Originally, single-file frameworks were hard-saved to a specific filename, that is then passed as the only copied generated artifact to the image. However, all files are in the tar stream, so yes, copying everything over should be an easy update to the images. We did not need this usecase before, but we can make this change.

There are some other stability updates coming soon as well, I will then add this item to the changes.