There doesn't seem to be a way to get CFLite to upload files crashing fuzz targets if they crash the latest build

[evverx]

I've been using CFLite to test various patches and when they introduce new issues CFLite uploads files to the "summary" tab as expected. But if CFLite thinks that bugs aren't introduced in PRs it just skips that step. I think it makes sense in general but in some cases bugs can collide so to speak and crash PRs and the latest build differently. It would be great if I could flip a flag named, say, 'always_upload_crashes` to avoid running something like

./infra/helper.py build_fuzzers --external ~/elfutils
./infra/helper.py run_fuzzer --external ~/elfutils fuzz-dwfl-core

in my VMs and copying files to machines that have access to bug trackers to just attach those files to bug reports.

[jonathanmetzman]

How about a switch to shut off crash novelty checking instead?

[evverx]

It would make it harder to tell really new bugs and bugs that just happen to collide apart, which would probably mean that I wouldn't even try to turn it off for projects that have just started fixing bugs found by fuzz targets.

[jonathanmetzman]

Sorry, I think I missed something in your original request.
Do you want uploading of fuzz targets and crashes that are not novel?

[jonathanmetzman]

Should we reopen this and close google/oss-fuzz#6984 instead?

[evverx]

Should we reopen this and close google/oss-fuzz#6984 instead?

I thought I was doing exactly that :-) Looks like I lost track of all the issues I've opened :-)

Another feature that I think would be useful if CFLite could show backtraces of the last builds as well. I've just discovered an issue that I haven't seen before but

2021-12-08 23:31:17,581 - root - INFO - Fuzzer: fuzz-dwfl-core. Detected bug.
2021-12-08 23:31:17,582 - root - INFO - Trying to reproduce crash using: /tmp/tmp7n4idjys/oom-e7fed1fe8da16c1482845db8b8d1fac479a425b5.
2021-12-08 23:31:18,452 - root - INFO - Reproduce command returned: 71. Reproducible on /github/workspace/build-out/fuzz-dwfl-core.
2021-12-08 23:31:18,452 - root - INFO - Crash is reproducible.
2021-12-08 23:31:18,462 - root - INFO - Trying to downloading previous build 51106acb0d7f01845ebab85a3ce4e8b46ae95ab5.
2021-12-08 23:31:20,151 - root - INFO - Done downloading previous build.
2021-12-08 23:31:20,152 - root - INFO - Trying to reproduce crash using: /tmp/tmp7n4idjys/oom-e7fed1fe8da16c1482845db8b8d1fac479a425b5.
2021-12-08 23:31:20,982 - root - INFO - Reproduce command returned: 71. Reproducible on /github/workspace/cifuzz-prev-build/fuzz-dwfl-core

so I have a patch and a new backtrace and to figure out what is going there it seems I have to build and run the fuzz target locally to get the file triggering it, then I have to revert that patch and build the fuzz target once again to pass the testcase to it to get the backtrace. It wouldn't be necessary if CFLite uploaded files crashing fuzz targets unconditionally and showed "old" backtraces.

[jonathanmetzman]

Should we reopen this and close google/oss-fuzz#6984 instead?

I thought I was doing exactly that :-) Looks like I lost track of all the issues I've opened :-)

Another feature that I think would be useful if CFLite could show backtraces of the last builds as well. I've just discovered an issue that I haven't seen before but

2021-12-08 23:31:17,581 - root - INFO - Fuzzer: fuzz-dwfl-core. Detected bug.
2021-12-08 23:31:17,582 - root - INFO - Trying to reproduce crash using: /tmp/tmp7n4idjys/oom-e7fed1fe8da16c1482845db8b8d1fac479a425b5.
2021-12-08 23:31:18,452 - root - INFO - Reproduce command returned: 71. Reproducible on /github/workspace/build-out/fuzz-dwfl-core.
2021-12-08 23:31:18,452 - root - INFO - Crash is reproducible.
2021-12-08 23:31:18,462 - root - INFO - Trying to downloading previous build 51106acb0d7f01845ebab85a3ce4e8b46ae95ab5.
2021-12-08 23:31:20,151 - root - INFO - Done downloading previous build.
2021-12-08 23:31:20,152 - root - INFO - Trying to reproduce crash using: /tmp/tmp7n4idjys/oom-e7fed1fe8da16c1482845db8b8d1fac479a425b5.
2021-12-08 23:31:20,982 - root - INFO - Reproduce command returned: 71. Reproducible on /github/workspace/cifuzz-prev-build/fuzz-dwfl-core

so I have a patch and a new backtrace and to figure out what is going there it seems I have to build and run the fuzz target locally to get the file triggering it, then I have to revert that patch and build the fuzz target once again to pass the testcase to it to get the backtrace. It wouldn't be necessary if CFLite uploaded files crashing fuzz targets unconditionally and showed "old" backtraces.

Sorry, I'm having trouble understanding this. We already show the stacktrace of the crash on the newer binary, why do we need to show the crash from the older binary?

[evverx]

why do we need to show the crash from the older binary?

Because the backtrace can be different if PRs introduce new bugs on top of old bugs.

[evverx]

@jonathanmetzman judging by https://github.com/evverx/elfutils/actions/runs/1574802879, it seems to be working. Thanks!

[evverx]

I'm not sure why CFLite is red there though. Those issues are reproducible with the latest build as well so I think CFLite should be green.

[evverx]

I'm not sure if it helps but here's what I use in the CFLite workflow:

    env:
      UPLOAD_ALL_CRASHES: 'True'
      REPORT_OOMS: 'False'

My understanding it that UPLOAD_ALL_CRASHES=True combined with REPORT_OOMS=False should prevent PRs from turning red when those OOMs are reproducible with the latest builds.

[evverx]

I moved that to evverx/elfutils#53 to avoid interfering with evverx/elfutils#49 (which I'm using to test https://sourceware.org/pipermail/elfutils-devel/2021q4/004520.html)

[evverx]

Without UPLOAD_ALL_CRASHES: 'True' it's green there as expected

[evverx]

Turned it off for now because without google/oss-fuzz#6997 the fuzzers can't go any further

[jonathanmetzman]

Landed google/oss-fuzz#6997

[evverx]

Looks like those OOMs are gone. Thanks! The fuzz target triggers a timeout though but once https://sourceware.org/pipermail/elfutils-devel/2021q4/004463.html is merged it should disappear as well.

[evverx]

With that patch backported the fuzz target can run for 10 minutes on CFLite so it's probably safe to say that it's prepared for the next round of patches. It would be great if PRs crashing the latest builds didn't turn red with UPLOAD_ALL_CRASHES=True and it was possible to show old backtraces though.