Skip to content

Re-enable LeakSanitizer runs - #24765

Open
sevan wants to merge 1 commit into
Perl:bleadfrom
sevan:enable-leaksanitzer-runs-again
Open

sevan wants to merge 1 commit into
Perl:bleadfrom
sevan:enable-leaksanitzer-runs-again

Conversation

@sevan

@sevan sevan commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

It's been almost 6 years since the feature was disabled due to instability running GDBM tests in commit 229e23b. The original intent in the commit which introduced the ASAN CI job was described in commit 4e4a36d "Whilst this will detect various dangerous mistakes involving bad memory accesses, it's far more likely to spot if changes create memory leaks." I think it is time to revisit to see after many tooling releases if LeakSanitizer is choking. If so, perhaps it would be better to remove GDBM install as part of this job until whatever triggers failures can be identified and fixed rather than limiting ASAN features in an ASAN CI job.

  • This set of changes does not require a perldelta entry.

It's been almost 6 years since the feature was disabled due to instability
running GDBM tests in commit 229e23b.
The original intent in the commit which introduced the ASAN CI job was
described in commit 4e4a36d
"Whilst this *will* detect various dangerous mistakes involving bad memory
accesses, it's far more likely to spot if changes create memory leaks."
I think it is time to revisit to see after many tooling releases if
LeakSanitizer is choking. If so, perhaps it would be better to remove
GDBM install as part of this job until whatever triggers failures can
be identified and fixed rather than limiting ASAN features in an
ASAN CI job.
@sevan

sevan commented Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

ubuntu-latest target currently means Ubuntu 24.04, whilst the latest actual release of Ubuntu is 26.04.
There is an ubuntu-26.04 image available, but it's currently marked as preview.

In the logs from the CI run here on Ubuntu 24.04, I see

op/length.t .......................................................... ok
==23953==LeakSanitizer has encountered a fatal error.
==23953==HINT: For debugging, try setting environment variable LSAN_OPTIONS=verbosity=1:log_threads=1
==23953==HINT: LeakSanitizer does not work under ptrace (strace, gdb, etc)
op/fork.t ............................................................ ok

@sevan

sevan commented Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

Same error is observed on Ubuntu 26.04 in the logs from PR #24767

op/kvhslice.t ........................................................ ok
==23616==LeakSanitizer has encountered a fatal error.
==23616==HINT: For debugging, try setting environment variable LSAN_OPTIONS=verbosity=1:log_threads=1
==23616==HINT: LeakSanitizer does not work under ptrace (strace, gdb, etc)
op/fork.t ............................................................ ok

@sevan

sevan commented Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

The logs from the run on 26.04 shed a little more light.
From PERL_UNICODE job built with sanitizers, note the warning on the first line:

==29183==WARNING: TestPTrace() failed to fork (errno 11)
==29183==LeakSanitizer has encountered a fatal error.
==29183==HINT: For debugging, try setting environment variable LSAN_OPTIONS=verbosity=1:log_threads=1
==29183==HINT: LeakSanitizer does not work under ptrace (strace, gdb, etc)
op/fork.t ............................................................ ok

@sevan

sevan commented Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

In PR #24767, I skipped installing libgdbm and libdb, and the LeakSanitizer error is still there. Seems like it's always when it comes to running the op/fork.t test.

@tonycoz

tonycoz commented Aug 30, 2026

Copy link
Copy Markdown
Contributor

I think we need to re-run the CI a few times to check if the (intermittent) GDBM errors are fixed.

@tonycoz

tonycoz commented Aug 30, 2026

Copy link
Copy Markdown
Contributor

I think we need to re-run the CI a few times to check if the (intermittent) GDBM errors are fixed.

Unfortunately I can only restart one sub-job at a time. In future re-runs I'll just re-run all CI.

@jkeenan jkeenan added the Infrastructure Things needed to maintain Perl development label Aug 31, 2026
@sevan

sevan commented Aug 31, 2026

Copy link
Copy Markdown
Contributor Author

I think we need to re-run the CI a few times to check if the (intermittent) GDBM errors are fixed.

It might be a better idea to enable debugging on the sanitizer and let the CI re-run to see what is upsetting it exactly.
The commit which enables debugging should be sufficient to trigger the CI run.
We can do it on the other pull request which targets a newer Ubuntu and is a throw-away PR.

@sevan

sevan commented Aug 31, 2026

Copy link
Copy Markdown
Contributor Author
HINT: For debugging, try setting environment variable LSAN_OPTIONS=verbosity=1:log_threads=1

@tonycoz

tonycoz commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Ahh, sorry, I missed the failed run on the other ticket.

Unfortunately setting the verbosity on means the run fails due to the build log being too large, so we don't get any useful information from it.

If this is flaky enough that your tests showed failures I don't think we can enable it.

Note that we do get full ASAN runs from volunteer daily build reports, eg. see this.

Maybe I'll add ASAN to another configuration for my runs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Infrastructure Things needed to maintain Perl development

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants