Performance degrades over time in production searches

[gfaccin]

Hi guys, thank you for this code!

I'm using it to search for structures and noticed that, in production-grade situations, the simulation performance degrades a lot over time, resulting in computing nodes staying idle most of the time.

Using AI tools I tried to diagnose the issue and this is what it returned. I'm bolding specific itens worth of your attention. Specifically, it appears to be necessary for the user to have some level of control around how xtalopt autosaves data in order to avoid the degradation.

Perhaps a patch could be considered in this direction?

---

• Diagnosis

Job 11544719 is not stuck, and the node shows no evidence of failure. Slurm reports it as RUNNING on sdumont8069. The idle periods are caused by XtalOpt autosaving:

• XtalOpt saves the session whenever a structure is updated and maintains a structure.state file for each structure, as documented in the official manual (https://xtalopt.github.io/files/user_guide.pdf).
• There are currently about 2,265 structure states, 4,533 structure.state* files, 29,221 files in total, and a 349 MB working directory.
• During the idle period, there were no active srun steps. All four GPUs showed 0% utilization and only 1 MiB of memory usage.
• XtalOpt held an open structure.state.lock, while a single QtConcurrent thread traversed the structure states.
• Its cumulative counters showed approximately 181 GB read and 64 GB written through I/O calls, even though the current set of state files occupies only about 60 MB. This demonstrates repeated rewrites.
• The working directory is located on the shared Lustre filesystem, explicitly configured in scheduler/scheduler.env:5. The scheduler currently rejects any mode other than project in scheduler/scheduler.sh:170.

One observed cycle was:

• Four LAMMPS processes started at 23:42:38.
• The last one finished at 23:45:18.
• No LAMMPS process ran for 8 minutes and 10 seconds.
• At 23:53:28, the autosave finished and four new LAMMPS processes started simultaneously.
• Immediately afterward, all four GPUs returned to 33-38% utilization.

The maximum possible GPU occupancy decreased from approximately 86% during the first two hours to 33% on the following day and 22% during the two most recent hours. The autosave cost grows approximately quadratically with the number of structures. The 160000 limit in xtalopt.in:65 makes this degradation unavoidable.

Recommended Mitigations

1. Run work-cli on the node-local /tmp filesystem. Node sdumont8069 has 873 GB available on local XFS, while the calculation currently uses only 349 MB. Perform a stage-in at startup, periodic snapshots to Lustre, and a final stage-out after a soft exit. The scheduler must be adapted because it currently prohibits this mode.

2. Modify XtalOpt’s autosave behavior. Coalesce updates using a debounce mechanism, allow only one pending save, and save only modified structures. A complete checkpoint every 1-5 minutes, plus a final synchronous save, would eliminate the queue of repeated saves. The installed version, 14.3.0, remains the latest stable release (https://xtalopt.github.io/download.html), so a simple upgrade will not resolve the problem.

3. Reduce the size of each search. A limit of 160,000 structures is impractical with this persistence format. Prefer several smaller independent searches with different seeds, followed by comparison of their best results.

4. Reduce continuousStructures. Its current value is 64 for only four GPUs. A value around 8-16 would reduce the internal backlog and bursts of state updates, although it would not eliminate the main problem.

5. Do not increase filesystem striping as the first response. The workload consists of thousands of small files; the bottleneck is primarily metadata activity and repeated rewriting, not sequential bandwidth.

The job resumed calculating and was neither interrupted nor modified. However, these idle periods will continue and will probably become longer as additional structures are created.

[hajinazar]

Dear @gfaccin ,

Thank you very much for bringing up the issue.

Actually, we're aware of this problem through recent benchmarking tests. The original design of XtalOpt mostly predates the current trend of usage/access of classical/ML potentials and HPC infrastructures. As a result, and while being correct in implementation, the file/disk operations are now starting to become a sizable portion of the internal workload and run time; even in runs that are much shorter compared to the case you mentioned.

In the past few months, we have been making a new version of XtalOpt (a major re-write of the code) where -among other changes- the overall handling of scheduling/frequency of saving output files is revised, which hopefully improves on the issue you mentioned. We are now preparing a release candidate of this new version; on which we will appreciate your feedback!

[gfaccin]

Dear @hajinazar,

Thank you! I've deployed on my system an IA made, Frankenstein customization of the current public version with no new features and just an attempt at addressing this specific problem. Testing should take 1-2 weeks and if it works, I can share it with you, should anything be useful.

Obviously, in the long run, human made code is better, but for a quick bugfix until the official new version comes out, that's ok.

If you guys find it useful, feel free to share privately or publicly the new release candidate. I can test it on my system in production-grade runs and benchmark against the current public version and /or the Frankenstein version.

Regards,

Giovani

[hajinazar]

Hi @gfaccin , thanks for offering further testing of the upcoming version. The release candidate should be available on XtalOpt's github within a day or two. Once this is out, please feel free to use/test it for your system. One thing to note: 160K structures is well above the range needed for most target systems. So, while the changes in output logging are basically along the same lines as you noted in original posting; still some restrictions are applied to maintain a fairly current output for a typical (to large) run. That's, a customized version specifically made for that run size/need might be more efficient than the upcoming version.