Add mujoco/model to benchmarks

[ft-lab]

Description

Fixes #80

We have updated the implementation to allow multiple MuJoCo verification repositories to be included in the benchmark.

• The creation and updating of the "xxx_anotations.yaml" file is now handled by tools/manage_annotations.py (analyze_menagerie_models.py is no longer needed).
• tools/menagerie_annotations.yaml has been renamed to tools/mujoco_menagerie_annotations.yaml.
  • The naming convention tools/[repository name]_annotations.yaml

How to Use

After creating the "xxx_annotations.yaml" file using "tools/manage_annotations.py"(poe annotations), run the benchmark using "tools/benchmark_and_report.py"(poe benchmark).
The "xxx_annotations.yaml" file has already been created.

Create or update the "xxx_annotations.yaml" file

Use the --mjcf-repository-path argument to specify either the URL of a GitHub repository or the path to a repository located on your local machine.
Specifying --update creates a new “xxx_annotations.yaml” file or adds the differences.

poe annotations --mjcf-repository-path "https://github.com/google-deepmind/mujoco_menagerie" --update

As a result, tools/[repository name]_annotations.yaml will be created or updated.

Benchmark and Report

Runs the benchmark using the GitHub repository or local repository specified with --mjcf-repository-path and outputs a report of the results.

poe benchmark --mjcf-repository-path "https://github.com/google-deepmind/mujoco_menagerie" --mjcf-repository-path "https://github.com/google-deepmind/mujoco"

The "benchmarks" directory will contain csv, html, and md files, as well as a "benchmark_summary.txt" file and logs.

We can also run the benchmark as follows.

If you are not in a virtual environment.

uv run --group dev poe benchmark --all

If you are in a virtual environment.

poe benchmark --all

Test repositories

• https://github.com/google-deepmind/mujoco_menagerie
• https://github.com/google-deepmind/mujoco ( The "model" directory is the subject of the verification.)

TODO

• Reviewing benchmarks on GitHub Runners. ok
• Update benchmarks.md ok
• Delete tools/analyze_menagerie_models.py, tools/benchmark_menagerie.py ok
• Update README.md ok
• Update the description in the output report (HTML/markdown). ok

Checklist

• I am familiar with the Contributing Guidelines.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ All tests successful. No failed tests found.

📢 Thoughts on this report? Let us know!

[ft-lab]

https://github.com/google-deepmind/mujoco

I added this repository to the benchmark verification, so the number of models increased from 85 to 144.

Total Models	Successful	Failed	Total Warnings	Total Errors	Average Time	Total Time	Total File Size
144	132 (91.7%)	12 (8.3%)	3699	9	0.92s	2m 11.94s	422.85 MB

However, it seems that there are some conversion errors in this repository "mujoco" (located in the model directory).
Since there seem to be some patterns in these benchmark results (the mesh conversion is failing), I'll look into it.
Since the purpose of this PR is to adjust the benchmarks, I will limit myself here to investigating the cause of the error and providing comments.

[andrewkaufman]

there are 3500 new warnings ?!?

[ft-lab]

When I benchmarked only mujoco_menagerie using this PR, the results were: Total Models 85, Failed 0%, Total Warnings 157.
This is the same as it was before this PR.

The models in the mujoco repository might actually be worth testing more than we thought...

[ft-lab]

I’ll list the findings from my review of the mujoco repository in order.

Warnings

Warning: Recursive frames are not supported

This warning appeared 2523 times in the benchmarks for the two repositories.
Since the message Recursive frames are not supported appears multiple times during a single conversion, and it is unclear where in the file this issue is occurring, it might be a good idea to improve the way this warning is displayed.
It appears that the increase in these warnings is due to the replicate loop.

The following MJCFs had multiple instances of this warning.

• https://github.com/google-deepmind/mujoco/blob/main/model/adhesion/active_adhesion.xml

Warning: Unsupported or unknown geom type mjtGeom.mjGEOM_ELLIPSOID for geom 'Ellipsoid'

There was a sample that made extensive use of ELLIPSOID.

• https://github.com/google-deepmind/mujoco/blob/main/model/humanoid/humanoid100.xml
• https://github.com/google-deepmind/mujoco/blob/main/model/replicate/leaves.xml

"leaves.xml" is a pattern that uses replicate to create multiple instances of this ELLIPSOID.
In the benchmark, I found 873 times (number of warnings) that matched this criteria.
This may be resolved in Issue #84 .

Warning: Unsupported builtin texture type

Patterns that use ‘mjtBuiltin.mjBUILTIN_CHECKER’ or ‘mjtBuiltin.mjBUILTIN_FLAT’ in textures.
In the benchmark, I found 66 times (number of warnings) that matched this criteria.

• https://github.com/google-deepmind/mujoco/blob/main/model/hammock/hammock.xml

The other warnings were related to unsupported items such as lights, cameras, keys, flexes, and sensors.

Errors

Error: Mesh has no file

If the mesh does not contain a file.

    <mesh name="chasis" scale=".01 .006 .0015"
      vertex=" 9   2   0
              -10  10  10
               9  -2   0
               10  3  -10
               10 -3  -10
              -8   10 -10
              -10 -10  10
              -8  -10 -10
              -5   0   20"/>

• https://github.com/google-deepmind/mujoco/blob/main/model/car/car.xml
• https://github.com/google-deepmind/mujoco/blob/main/model/cube/cube_3x3x3.xml

In the benchmark, I found 4 files that matched this criteria.

Error: XML Error

• https://github.com/google-deepmind/mujoco/blob/main/model/flex/hollow_vs_solid.xml

ValueError: XML Error: strain constraint and elasticity (young) cannot both be present
Element 'flexcomp', line 49

Error: Segmentation fault (External XML Reference?)

• https://github.com/google-deepmind/mujoco/blob/main/model/hammock/hammock.xml

Conversion failed with return code -11. Stderr: [Status] [mujoco_usd_converter._impl.cli.run]

When I converted it manually, it just displayed "Segmentation fault".

When I looked at "hammock.xml", it appeared to be referencing another XML file.
However, I haven't yet determined whether this is the cause.

<model name="humanoid" file="../humanoid/humanoid.xml"/>

The following also resulted in a "Segmentation fault" error.
In both cases, it referenced an external “humanoid.xml” file.

• https://github.com/google-deepmind/mujoco/blob/main/model/humanoid/100_humanoids.xml
• https://github.com/google-deepmind/mujoco/blob/main/model/humanoid/22_humanoids.xml
• https://github.com/google-deepmind/mujoco/blob/main/model/humanoid/humanoid100.xml
• https://github.com/google-deepmind/mujoco/blob/main/model/sleep/100_humanoids.xml

These humanoid samples seem to be causing errors due to nested replicate statements.

Error: Segmentation fault

• https://github.com/google-deepmind/mujoco/blob/main/model/replicate/newton_cradle.xml

Conversion failed with return code -11. Stderr: [Status] [mujoco_usd_converter._impl.cli.run] Running mujoco_usd_converter

When I converted it manually, it just displayed "Segmentation fault".
It is unclear what might be causing this (there are no references to external XML files).

[andrewkaufman]

Can you log Issues for recursive frames & mesh-without-file? I think we should address those soon.

For the seg fault, you can run manually with --verbose to get a proper traceback

[ft-lab]

I’ll attach a few logs.

Warning: Recursive frames are not supported

https://github.com/google-deepmind/mujoco/blob/main/model/adhesion/active_adhesion.xml

Execute the conversion.

mujoco_usd_converter "/home/foo/GitHub/mujoco/model/adhesion/active_adhesion.xml" "./export_usds/adhesion" --verbose

log file: adhesion_log.txt

I’ll continue commenting on the logs for another conversion.

Added:
I've figured out what caused this.
This error - "Recursive frames are not supported" - occurs even with a simpler MJCF, and it appears to be unrelated to the content of this message.
This warning appeared when using replicate in a nested loop, and the conversion was not performed correctly.

replicate x 1 (Success)

The conversion was successful.
The results were the same as those from the MuJoCo Viewer.
No warnings were output.

<mujoco model="replicate_simple">
  <worldbody>
    <replicate count="5" offset="1.5 0 0">
        <geom type="box" size="0.4 0.4 0.4" />
    </replicate>
  </worldbody>
</mujoco>

replicate x 2 (Incorrect)

The following uses replicate twice.
I confirmed that 5x3 boxes were duplicated, resulting in a total of 15 boxes in USD as well.
However, the offset-based movement from the outer replicate has not been applied.
The warning Recursive frames are not supported has been displayed 15 times.

<mujoco model="replicate_simple_2">
  <worldbody>
    <replicate count="3" offset="0 1.5 0">
        <replicate count="5" offset="1.5 0 0">
            <geom type="box" size="0.4 0.4 0.4" />
        </replicate>
    </replicate>
  </worldbody>
</mujoco>

[ft-lab]

Error: Mesh has no file

https://github.com/google-deepmind/mujoco/blob/main/model/car/car.xml

Execute the conversion.

mujoco_usd_converter "/home/foo/GitHub/mujoco/model/car/car.xml" "./export_usds/car" --verbose

log file: car_log.txt

The following is specified in car.xml, but the current version of mujoco-usd-converter does not support this format, as it does not reference external mesh files (stl, obj).
However, this specification for direct mesh specification may be straightforward.

    <mesh name="chasis" scale=".01 .006 .0015"
      vertex=" 9   2   0
              -10  10  10
               9  -2   0
               10  3  -10
               10 -3  -10
              -8   10 -10
              -10 -10  10
              -8  -10 -10
              -5   0   20"/>

Added:

I found the specifications for this mesh described in the official documentation below.

• https://mujoco.readthedocs.io/en/latest/XMLreference.html#asset-mesh
• https://github.com/google-deepmind/mujoco/blob/main/doc/XMLreference.rst#el-prefixasset---mesh-

Direct meshes that do not use external references have the following specifications (Explanation from the cursor).

• Specify the XYZ coordinates of the vertex using vertex. This is a required parameter.
• If you do not specify face, faces will be automatically generated from the convex hull of the vertex positions.
• When writing a face, each face consists of three vertex indices (0 to nvert−1) in counterclockwise order.
  It is interpreted as an indexed triangular mesh.
• Other Options
  • normal: The same number of normals as there are vertices
  • texcoord: UV coordinates (0–1) for each vertex
  • scale / refpos / refquat: Scale and reference coordinate system

"automatically generated from the convex hull of the vertex positions" might be difficult to implement...

[ft-lab]

Error: Segmentation fault

https://github.com/google-deepmind/mujoco/blob/main/model/replicate/newton_cradle.xml

Execute the conversion.

mujoco_usd_converter "/home/foo/GitHub/mujoco/model/replicate/newton_cradle.xml" "./export_usds/newton_cradle" --verbose

Even when I added “--verbose,” the only output was “Segmentation fault,” so I couldn't determine where the problem was occurring.

[Status] [mujoco_usd_converter._impl.cli.run] Running mujoco_usd_converter
[Status] [mujoco_usd_converter._impl.cli.run] Version: 0.2.0
[Status] [mujoco_usd_converter._impl.cli.run] USD Version: (0, 25, 5)
[Status] [mujoco_usd_converter._impl.cli.run] USDEX Version: 2.2.2
[Status] [mujoco_usd_converter._impl.cli.run] MuJoCo Version: 3.8.0
[Status] [mujoco_usd_converter._impl.cli.run] Newton USD Schemas Version: 0.2.0
[Status] [mujoco_usd_converter._impl.convert.convert] Converting /home/foo/GitHub/mujoco/model/replicate/newton_cradle.xml into export_usds/newton_cradle
[Warning] [mujoco_usd_converter._impl.material.convert_texture] Unsupported builtin texture type mjtBuiltin.mjBUILTIN_CHECKER for texture 'texplane'
[Status] [usdex::core::saveStage] Saving "stage with rootLayer @/home/foo/GitHub/mujoco-usd-converter/export_usds/newton_cradle/Payload/MaterialsLibrary.usdc@, sessionLayer @anon:0x26950df0:MaterialsLibrary-session.usda@" with comment "Material Library for Newton's Cradle. "
Segmentation fault

I will output more tracking logs within mujoco-usd-converter to investigate what is causing this.

Added:
The conversion was successful after excluding replicate from MJCF.

[ft-lab]

Summary

Here is a list of the issues for which the causes have been identified.

• The conversion process does not properly handle replicate.
  • We might want to add a few replicate tests to our unit tests.
• Support for cases where external references are not specified in the mesh (Direct specification of mesh).

[ft-lab]

External MJCF Reference

When referencing a simple MJCF file externally as shown below, it converted without any issues.

<model name="sphere" file="sphere.xml"/>

sphere.xml

<mujoco model="sphere">
  <worldbody>
    <body name="sphere">
      <geom type="sphere" size="0.2" />
    </body>
  </worldbody>
</mujoco>

spheres.xml

<mujoco model="spheres">
  <asset>
    <model name="sphere" file="sphere.xml"/>
  </asset>

  <worldbody>
    <frame pos="0 -0.5 0">
      <attach model="sphere" body="sphere" prefix="_1" />
    </frame>
    <frame pos="0 0 0">
      <attach model="sphere" body="sphere" prefix="_2" />
    </frame>
    <frame pos="0 0.5 0">
      <attach model="sphere" body="sphere" prefix="_3" />
    </frame>
  </worldbody>
</mujoco>

It doesn't seem that the Error: Segmentation fault I mentioned a few posts ago was caused by a reference to an external XML file.

Added:
This also worked correctly when XML files that specified obj or stl files were referenced externally.

[ft-lab]

I've moved this PR to code review.

By adding the mujoco repository to our benchmarks, we were able to detect numerous warnings and 12 errors.
I have reported on this below.

#87 (comment)

It might be best to split some of these into separate issues.