Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Test failure baseservices/exceptions/simple/ParallelCrashMainThread/ParallelCrashMainThread.sh #80356

Open
v-wenyuxu opened this issue Jan 9, 2023 · 19 comments
Open

Test failure baseservices/exceptions/simple/ParallelCrashMainThread/ParallelCrashMainThread.sh #80356

v-wenyuxu opened this issue Jan 9, 2023 · 19 comments
Assignees
@mangod9
Labels
arch-x64 area-VM-coreclr disabled-test The test is disabled in source code against the issue GCStress os-mac-os-x macOS aka OSX
Milestone
Future

Comments

@v-wenyuxu
Copy link

v-wenyuxu commented Jan 9, 2023 •
edited by build-analysis bot
Loading

Run: runtime-coreclr outerloop 20230107.2

Failed test:

coreclr OSX x64 Checked no_tiered_compilation @ OSX.1200.Amd64.Open

- baseservices/exceptions/simple/ParallelCrashMainThread/ParallelCrashMainThread.sh
- baseservices/exceptions/simple/ParallelCrash/ParallelCrash.sh

Error message:

Process terminated. Parallel crash in main thread
at System.Environment.FailFast(System.String)
at ParallelCrash.Main(System.String[])
[createdump] Invalid process id: task_for_pid(99238) FAILED (os/kern) failure (5)
[createdump] This failure may be because createdump or the application is not properly signed and entitled.
[createdump] Failure took 0ms
[createdump] waitpid() returned successfully (wstatus 0000ff00)

cmdLine:/private/tmp/helix/working/B6BE098F/w/B7D109CE/e/baseservices/exceptions/simple/ParallelCrashMainThread/ParallelCrashMainThread.sh Timed Out (timeout in milliseconds: 600000 from variable __TestTimeout, start: 1/7/2023 6:00:33 PM, end: 1/7/2023 6:10:34 PM)

Return code:      -100
Raw output file:      /tmp/helix/working/B6BE098F/w/B7D109CE/uploads/Reports/baseservices.exceptions/simple/ParallelCrashMainThread/ParallelCrashMainThread.output.txt
Raw output:
BEGIN EXECUTION
export CDPATH=/private/tmp/helix/working/B6BE098F/w/B7D109CE/e/baseservices/exceptions/simple/ParallelCrashMainThread
/usr/bin/env bash ../ParallelCrash/ParallelCrash.sh '1'
BEGIN EXECUTION
/tmp/helix/working/B6BE098F/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false ParallelCrash.dll '1'
[createdump] Target process is alive

cmdLine:/private/tmp/helix/working/B6BE098F/w/B7D109CE/e/baseservices/exceptions/simple/ParallelCrashMainThread/ParallelCrashMainThread.sh Timed Out (timeout in milliseconds: 600000 from variable __TestTimeout, start: 1/7/2023 6:00:33 PM, end: 1/7/2023 6:10:34 PM)
Test Harness Exitcode is : -100
To run the test:

set CORE_ROOT=/tmp/helix/working/B6BE098F/p
/private/tmp/helix/working/B6BE098F/w/B7D109CE/e/baseservices/exceptions/simple/ParallelCrashMainThread/ParallelCrashMainThread.sh
Expected: True
Actual:   False


Stack trace
   at baseservices_exceptions._simple_ParallelCraMainThread_ParallelCraMainThread_._simple_ParallelCrashMainThread_ParallelCrashMainThread_sh()
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
Queued OS Arch Column1
2023-01-07T22:51:52.802Z osx.1200.amd64.open x64 runtime-coreclr outerloop Checked-no_tiered_compilation
2023-01-07T11:21:08.126Z osx.1200.amd64.open x64 runtime-coreclr outerloop Checked-no_tiered_compilation
2023-01-07T22:51:52.802Z osx.1200.amd64.open x64 runtime-coreclr outerloop Checked-no_tiered_compilation

Known Issue Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Report

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
0 0 0
@v-wenyuxu v-wenyuxu added os-mac-os-x macOS aka OSX arch-x64 blocking-outerloop Blocking the 'runtime-coreclr outerloop' and 'runtime-libraries-coreclr outerloop' runs labels Jan 9, 2023
@dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

@ghost ghost added the untriaged New issue has not been triaged by the area owner label Jan 9, 2023
@runfoapp runfoapp bot mentioned this issue Jan 9, 2023
Closed
@v-wenyuxu
Copy link
Author

Run: runtime-coreclr gcstress-extra 20230204.1

Failed test:

coreclr linux arm Checked gcstress0xc_zapdisable_jitstress2 @ (Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7

- baseservices/exceptions/simple/ParallelCrash/ParallelCrash.sh

coreclr linux arm Checked gcstress0xc_zapdisable @ (Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7

- baseservices/exceptions/simple/ParallelCrashMainThread/ParallelCrashMainThread.sh

coreclr linux arm64 Checked gcstress0xc_zapdisable_jitstress2 @ (Ubuntu.1804.Arm64.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8

- baseservices/exceptions/simple/ParallelCrashMainThread/ParallelCrashMainThread.sh

coreclr linux x64 Checked gcstress0xc_jitstress1 @ Ubuntu.1804.Amd64.Open

- baseservices/exceptions/simple/ParallelCrash/ParallelCrash.sh
- baseservices/exceptions/simple/ParallelCrashMainThread/ParallelCrashMainThread.sh

Error message:

Process terminated. Parallel crash in main thread
at System.Environment.FailFast(System.String)
at ParallelCrash.Main(System.String[])
[createdump] waitpid() returned successfully (wstatus 00000000)
/root/helix/work/workitem/e/baseservices/exceptions/simple/ParallelCrash/ParallelCrash.sh: line 425:  5533 Illegal instruction     (core dumped) $LAUNCHER $ExePath "${CLRTestExecutionArguments[@]}"

Return code:      1
Raw output file:      /root/helix/work/workitem/uploads/Reports/baseservices.exceptions/simple/ParallelCrash/ParallelCrash.output.txt
Raw output:
BEGIN EXECUTION
/root/helix/work/correlation/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false ParallelCrash.dll ''
[createdump] Gathering state for process 5533 corerun
[createdump] Crashing thread 0000159d signal 00000006
[createdump] Writing crash report to file /home/helixbot/dotnetbuild/dumps/coredump.5533.dmp.crashreport.json
[createdump] Crash report successfully written
[createdump] Writing minidump with heap to file /home/helixbot/dotnetbuild/dumps/coredump.5533.dmp
[createdump] Written 143884288 bytes (35128 pages) to core file
[createdump] Target process is alive
[createdump] Dump successfully written in 763ms
Expected: 134
Actual: 132
END EXECUTION - FAILED
Test failed. Trying to see if dump file was created in /home/helixbot/dotnetbuild/dumps since 2/5/2023 12:33:51 AM
Processing /home/helixbot/dotnetbuild/dumps/coredump.5533.dmp.crashreport.json
Printing stacktrace from '/home/helixbot/dotnetbuild/dumps/coredump.5533.dmp.crashreport.json'
Invoking llvm-symbolizer --pretty-print
Errors while running llvm-symbolizer --pretty-print
System.ComponentModel.Win32Exception (2): An error occurred trying to start process 'llvm-symbolizer' with working directory '/root/helix/work/workitem/e'. No such file or directory
at System.Diagnostics.Process.ForkAndExecProcess(ProcessStartInfo startInfo, String resolvedFilename, String[] argv, String[] envp, String cwd, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, Int32& stdinFd, Int32& stdoutFd, Int32& stderrFd, Boolean usesTerminal, Boolean throwOnNoExec)
at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo)
at CoreclrTestLib.CoreclrTestWrapperLib.TryPrintStackTraceFromCrashReport(String crashReportJsonFile, StreamWriter outputWriter)
Finish looking for *.crashreport.json. No new files created.
Test Harness Exitcode is : 1
To run the test:

set CORE_ROOT=/root/helix/work/correlation
/root/helix/work/workitem/e/baseservices/exceptions/simple/ParallelCrash/ParallelCrash.sh
Expected: True
Actual:   False


Stack trace
   at baseservices_exceptions._simple_ParallelCra_ParallelCra_._simple_ParallelCrash_ParallelCrash_sh()
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)

@v-wenyuxu
Copy link
Author

Failed in Run: runtime-coreclr gcstress-extra 20230212.1

Failed tests:

coreclr linux x64 Checked gcstress0xc_jitstress2 @ Ubuntu.1804.Amd64.Open
    - baseservices/exceptions/simple/ParallelCrash/ParallelCrash.sh
coreclr linux x64 Checked gcstress0xc_tailcallstress @ Ubuntu.1804.Amd64.Open
    - baseservices/exceptions/simple/ParallelCrash/ParallelCrash.sh

Error message:

Process terminated. Parallel crash in main thread
  at System.Environment.FailFast(System.String)
  at ParallelCrash.Main(System.String[])
[createdump] waitpid() returned successfully (wstatus 00000000)
/datadisks/disk1/work/9E9308D7/w/C40B09F9/e/baseservices/exceptions/simple/ParallelCrash/ParallelCrash.sh: line 425: 13244 Segmentation fault      (core dumped) $LAUNCHER $ExePath "${CLRTestExecutionArguments[@]}"

Return code:      1
Raw output file:      /datadisks/disk1/work/9E9308D7/w/C40B09F9/uploads/Reports/baseservices.exceptions/simple/ParallelCrash/ParallelCrash.output.txt
Raw output:
BEGIN EXECUTION
/datadisks/disk1/work/9E9308D7/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false ParallelCrash.dll ''
[createdump] Gathering state for process 13244 corerun
[createdump] Crashing thread 000033bc signal 00000006
[createdump] Writing crash report to file /home/helixbot/dotnetbuild/dumps/coredump.13244.dmp.crashreport.json
[createdump] Crash report successfully written
[createdump] Writing minidump with heap to file /home/helixbot/dotnetbuild/dumps/coredump.13244.dmp
[createdump] Written 145395712 bytes (35497 pages) to core file
[createdump] Target process is alive
[createdump] Dump successfully written in 597ms
Expected: 134
Actual: 139
END EXECUTION - FAILED
Test failed. Trying to see if dump file was created in /home/helixbot/dotnetbuild/dumps since 2/12/2023 10:50:58 PM
Processing /home/helixbot/dotnetbuild/dumps/coredump.13244.dmp.crashreport.json
Printing stacktrace from '/home/helixbot/dotnetbuild/dumps/coredump.13244.dmp.crashreport.json'
Invoking llvm-symbolizer --pretty-print
Stack trace:
----------------------------------
Thread Id: 0x33bc
     Child SP               IP Call Site
0x7ffc38633a10 0x7f4b6e20432a libpthread.so.0!waitpid
0x7ffc38633a40 0x7f4b6d3d0560 libcoreclr.so!PROCCreateCrashDump(std::vector<char const*, std::allocator<char const*> >&, char*, int) at /__w/1/s/src/coreclr/pal/src/thread/process.cpp:2223:22
0x7ffc38633ab0 0x7f4b6d3d1304 libcoreclr.so!PROCCreateCrashDumpIfEnabled at /__w/1/s/src/coreclr/pal/src/thread/process.cpp:2413:9
0x7ffc38633b40 0x7f4b6d3cdf21 libcoreclr.so!PROCAbort at /__w/1/s/src/coreclr/pal/src/thread/process.cpp:2441:5
0x7ffc38633b60 0x7f4b6d3cdcbf libcoreclr.so!PROCEndProcess(void*, unsigned int, int) at /__w/1/s/src/coreclr/pal/src/thread/process.cpp:1346:13
0x7ffc38633b90 0x7f4b6d3cdd55 libcoreclr.so!TerminateProcess at /__w/1/s/src/coreclr/pal/src/thread/process.cpp:1239:11
0x7ffc38633bc0 0x7f4b6cf7c220 libcoreclr.so!SafeExitProcess(unsigned int, ShutdownCompleteAction) at /__w/1/s/src/coreclr/vm/eepolicy.cpp:0:9
0x7ffc38633c00 0x7f4b6cf7d53b libcoreclr.so!CHECK::CHECK() at /__w/1/s/src/coreclr/inc/check.h:115:17
(inlined by) EEPolicy::HandleFatalError(unsigned int, unsigned long, char16_t const*, _EXCEPTION_POINTERS*, char16_t const*, char16_t const*) at /__w/1/s/src/coreclr/vm/eepolicy.cpp:795:5
0x7ffc386341f0 0x7f4b6d2ce3c7 libcoreclr.so!SBuffer::~SBuffer() at /__w/1/s/src/coreclr/inc/sbuffer.inl:180:9
(inlined by) SystemNative::GenericFailFast(REF<StringObject>, REF<ExceptionObject>, unsigned long, unsigned int, REF<StringObject>) at /__w/1/s/src/coreclr/classlibnative/bcltype/system.cpp:289:5
0x7ffc386344d0 0x7f4b6d2ce6ad libcoreclr.so!HelperMethodFrame_1OBJ::Pop() at /__w/1/s/src/coreclr/classlibnative/../vm/frames.h:1421:28
(inlined by) FrameWithCookie<HelperMethodFrame_1OBJ>::Pop() at /__w/1/s/src/coreclr/classlibnative/../vm/frames.h:3249:47
(inlined by) SystemNative::FailFast(StringObject*) at /__w/1/s/src/coreclr/classlibnative/bcltype/system.cpp:308:5
0x7ffc386346a0 0x7f4aeeee1b3b ParallelCrash.dll!ParallelCrash.Main(System.String[])
----------------------------------
Thread Id: 0x33bd
     Child SP               IP Call Site
0x7f4b6c8e6d60 0x7f4b6d5d6bb9 libc.so.6!__poll
0x7f4b6c8e6d90 0x7f4b6d3be83e libcoreclr.so!CorUnix::CPalSynchronizationManager::ThreadPrepareForShutdown() at /__w/1/s/src/coreclr/pal/src/sync

Stack trace:

  at baseservices_exceptions._simple_ParallelCra_ParallelCra_._simple_ParallelCrash_ParallelCrash_sh()
  at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
  at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)

@BruceForstall
Copy link
Member

BruceForstall commented Feb 14, 2023 •
edited
Loading

Some recent failures of ParallelCrash:



Queued Pipeline Pipeline_Configuration OS Arch
2023-02-13T07:55:46.964Z runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_jitstress2 ubuntu.1804.amd64.open.rt x64
2023-02-13T03:39:46.694Z runtime-coreclr r2r-extra Checked-gcstress0xc ubuntu.1804.amd64.open.rt x64
2023-02-12T22:39:43.294Z runtime-coreclr gcstress-extra Checked-gcstress0xc_tailcallstress ubuntu.1804.amd64.open.rt x64
2023-02-12T22:39:42.157Z runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress2 ubuntu.1804.amd64.open.rt x64
2023-02-12T07:41:18.287Z runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_zapdisable_heapverify1 ubuntu.1804.amd64.open.rt x64
2023-02-11T22:39:54.718Z runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress1 ubuntu.1804.amd64.open.rt x64
2023-02-06T08:23:41.332Z runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_jitstress1 osx.1200.arm64.open arm64
2023-02-06T08:00:30.958Z runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_jitstress2 ubuntu.1804.amd64.open.rt x64
2023-02-06T08:00:28.275Z runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_zapdisable ubuntu.1804.amd64.open.rt x64
2023-02-05T22:51:16.936Z runtime-coreclr gcstress-extra Checked-gcstress0xc_tailcallstress ubuntu.1804.armarch.open arm64
2023-02-05T07:51:45.652Z runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_jitstress1 ubuntu.1804.amd64.open.rt x64
2023-02-05T07:51:40.417Z runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_jitminopts_heapverify1 ubuntu.1804.amd64.open.rt x64
2023-02-05T07:51:40.417Z runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_jitminopts_heapverify1 ubuntu.1804.amd64.open.rt x64
2023-02-04T22:57:30.335Z runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress1 ubuntu.1804.amd64.open.rt x64
2023-02-04T22:56:11.956Z runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable_jitstress2 ubuntu.1804.armarch.open arm
2023-01-23T08:15:21.364Z runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_zapdisable ubuntu.1804.amd64.open.rt x64
2023-01-23T08:15:20.208Z runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_jitstress1 ubuntu.1804.amd64.open.rt x64
2023-01-22T22:36:00.945Z runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress1 ubuntu.1804.amd64.open.rt x64
2023-01-22T08:03:24.673Z runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_jitstress2 ubuntu.1804.amd64.open.rt x64
2023-01-21T22:40:06.836Z runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable ubuntu.1804.armarch.open arm64
2023-01-21T22:40:04.914Z runtime-coreclr gcstress-extra Checked-gcstress0xc_tailcallstress ubuntu.1804.armarch.open arm64
2023-01-07T22:51:52.802Z runtime-coreclr outerloop Checked-no_tiered_compilation osx.1200.amd64.open x64

@BruceForstall
Copy link
Member

Since all recent failures have been GCStress, I'm marking it as such. The OSX failure hasn't happened in a while.

@BruceForstall
Copy link
Member

Also, the runtime-coreclr outerloop pipeline hasn't failed, so changing the blocking label

@BruceForstall BruceForstall added blocking-clean-ci-optional Blocking optional rolling runs and removed blocking-outerloop Blocking the 'runtime-coreclr outerloop' and 'runtime-libraries-coreclr outerloop' runs labels Feb 14, 2023
BruceForstall added a commit to BruceForstall/runtime that referenced this issue Feb 14, 2023
@BruceForstall
Temporarily disable ParallelCrash, ParallelCrashMainThread under GCSt…
8dd7b2d 
…ress

These are regularly failing in GCStress testing.

Tracking: dotnet#80356
@BruceForstall BruceForstall mentioned this issue Feb 14, 2023
Merged
@BruceForstall
Copy link
Member

Dup? #80357

@BruceForstall BruceForstall mentioned this issue Feb 14, 2023
Closed
@kasperk81
Copy link
Contributor

caused by #81387?

@BruceForstall
Copy link
Member

@kunalspathak These tests are designed to crash -- that is considered success. Would #81387 intercept this crash and interfere with the test run? How would I test this when running locally?

@kunalspathak
Copy link
Member

Would #81387 intercept this crash and interfere with the test run?

I don't think so. It is an added functionality on top of existing one that collects crashdumps. Ideally, for this test, the test would have verified that it crashed as expected and then return the ExitCode = 0, but that doesn't seem be happening and the test crashes before that and the test driver doesn't get ExitCode=0 but gets a crash dump which #81387 will use to create stack trace. Additionally, this functionality was added 2 weeks back, while this test seems to be failing since January.

If it helps, here is the call stack that we got (through #81387) for x64. However, for Arm64, for some reason, we do not print the stack trace. @hoyosjs - isn't llvm-symbolizer installed on those boxes?

Stack trace on x64 
   Stack trace:
      ----------------------------------
      Thread Id: 0x2417
            Child SP               IP Call Site
       0x7fff6a8051a0 0x7fe268f6f32a libpthread.so.0!waitpid
       0x7fff6a8051d0 0x7fe26813b560 libcoreclr.so!PROCCreateCrashDump(std::vector<char const*, std::allocator<char const*> >&, char*, int) at /__w/1/s/src/coreclr/pal/src/thread/process.cpp:2223:22
       0x7fff6a805240 0x7fe26813c304 libcoreclr.so!PROCCreateCrashDumpIfEnabled at /__w/1/s/src/coreclr/pal/src/thread/process.cpp:2413:9
       0x7fff6a8052d0 0x7fe268138f21 libcoreclr.so!PROCAbort at /__w/1/s/src/coreclr/pal/src/thread/process.cpp:2441:5
       0x7fff6a8052f0 0x7fe268138cbf libcoreclr.so!PROCEndProcess(void*, unsigned int, int) at /__w/1/s/src/coreclr/pal/src/thread/process.cpp:1346:13
       0x7fff6a805320 0x7fe268138d55 libcoreclr.so!TerminateProcess at /__w/1/s/src/coreclr/pal/src/thread/process.cpp:1239:11
       0x7fff6a805350 0x7fe267ce7220 libcoreclr.so!SafeExitProcess(unsigned int, ShutdownCompleteAction) at /__w/1/s/src/coreclr/vm/eepolicy.cpp:0:9
       0x7fff6a805390 0x7fe267ce853b libcoreclr.so!CHECK::CHECK() at /__w/1/s/src/coreclr/inc/check.h:115:17
       (inlined by) EEPolicy::HandleFatalError(unsigned int, unsigned long, char16_t const*, _EXCEPTION_POINTERS*, char16_t const*, char16_t const*) at /__w/1/s/src/coreclr/vm/eepolicy.cpp:795:5
       0x7fff6a805980 0x7fe2680393c7 libcoreclr.so!SBuffer::~SBuffer() at /__w/1/s/src/coreclr/inc/sbuffer.inl:180:9
       (inlined by) SystemNative::GenericFailFast(REF<StringObject>, REF<ExceptionObject>, unsigned long, unsigned int, REF<StringObject>) at /__w/1/s/src/coreclr/classlibnative/bcltype/system.cpp:289:5
       0x7fff6a805c60 0x7fe2680396ad libcoreclr.so!HelperMethodFrame_1OBJ::Pop() at /__w/1/s/src/coreclr/classlibnative/../vm/frames.h:1421:28
       (inlined by) FrameWithCookie<HelperMethodFrame_1OBJ>::Pop() at /__w/1/s/src/coreclr/classlibnative/../vm/frames.h:3249:47
       (inlined by) SystemNative::FailFast(StringObject*) at /__w/1/s/src/coreclr/classlibnative/bcltype/system.cpp:308:5
       0x7fff6a805e30 0x7fe1e9c21b76 ParallelCrash.dll!ParallelCrash.Main(System.String[])
      ----------------------------------
      Thread Id: 0x2418
            Child SP               IP Call Site
       0x7fe267651d60 0x7fe268341bb9 libc.so.6!__poll
       0x7fe267651d90 0x7fe26812983e libcoreclr.so!CorUnix::CPalSynchronizationManager::ThreadPrepareForShutdown() at /__w/1/s/src/coreclr/pal/src/synchmgr/synchmanager.cpp:3250:13
       0x7fe267651db0 0x7fe26812c9c4 libcoreclr.so!?? at ??:0:0
       0x7fe267651dc0 0x7fe26813f142 libcoreclr.so!CorUnix::CPalThread::ThreadEntry(void*) at /__w/1/s/src/coreclr/pal/src/thread/thread.cpp:1760:16
       0x7fe267651f00 0x7fe268f646db libpthread.so.0!
       0x7fe267651fc0 0x7fe26834e61f libc.so.6!clone
      ----------------------------------
      Thread Id: 0x2419
            Child SP               IP Call Site
       0x7fe266e50a30 0x7fe268341bb9 libc.so.6!__poll
       0x7fe266e50a60 0x7fe267fa4cac libcoreclr.so!ipc_retry_syscall(int) at /__w/1/s/src/native/eventpipe/ds-ipc-pal-socket.c:283:16
       (inlined by) ipc_poll_fds(pollfd*, unsigned long, unsigned int) at /__w/1/s/src/native/eventpipe/ds-ipc-pal-socket.c:486:11
       (inlined by) ds_ipc_poll(_DiagnosticsIpcPollHandle*, unsigned long, unsigned int, void (*)(char const*, unsigned int)) at /__w/1/s/src/native/eventpipe/ds-ipc-pal-socket.c:1096:16
       0x7fe266e50aa0 0x7fe2680a9af6 libcoreclr.so!ds_ipc_stream_factory_get_next_available_stream(void (*)(char const*, unsigned int)) at /__w/1/s/src/native/eventpipe/ds-ipc.c:406:15
       0x7fe266e50dd0 0x7fe2680a7276 libcoreclr.so!server_thread(void*) at /__w/1/s/src/native/eventpipe/ds-server.c:130:8
       0x7fe266e50e40 0x7fe26813f142 libcoreclr.so!CorUnix::CPalThread::ThreadEntry(void*) at /__w/1/s/src/coreclr/pal/src/thread/thread.cpp:1760:16
       0x7fe266e50f00 0x7fe268f646db libpthread.so.0!
       0x7fe266e50fc0 0x7fe26834e61f libc.so.6!clone
      ----------------------------------
      Thread Id: 0x241a
            Child SP               IP Call Site
       0x7fe265a84670 0x7fe268f6ee9e libpthread.so.0!open64
       0x7fe265a846e0 0x7fe267fa5b3c libcoreclr.so!TwoWayPipe::WaitForConnection() at /__w/1/s/src/coreclr/debug/debug-pal/unix/twowaypipe.cpp:87:19
       0x7fe265a84710 0x7fe267f9d3ab libcoreclr.so!DbgTransportSession::TransportWorker() at /__w/1/s/src/coreclr/debug/ee/../shared/dbgtransportsession.cpp:1315:17
       0x7fe265a84e20 0x7fe267f9bf75 libcoreclr.so!DbgTransportSession::TransportWorkerStatic(void*) at /__w/1/s/src/coreclr/debug/ee/../shared/dbgtransportsession.cpp:1235:5
       0x7fe265a84e40 0x7fe26813f142 libcoreclr.so!CorUnix::CPalThread::ThreadEntry(void*) at /__w/1/s/src/coreclr/pal/src/thread/thread.cpp:1760:16
       0x7fe265a84f00 0x7fe268f646db libpthread.so.0!
       0x7fe265a84fc0 0x7fe26834e61f libc.so.6!clone
      ----------------------------------
      Thread Id: 0x241b
            Child SP               IP Call Site
       0x7fe265283990 0x7fe268f6aad3 libpthread.so.0!pthread_cond_wait
       0x7fe265283a60 0x7fe268129237 libcoreclr.so!CorUnix::CPalSynchronizationManager::ThreadNativeWait(CorUnix::_ThreadNativeWaitData*, unsigned int, CorUnix::ThreadWakeupReason*, unsigned int*) at /__w/1/s/src/coreclr/pal/src/synchmgr/synchmanager.cpp:0:28
       0x7fe265283ad0 0x7fe26812887b libcoreclr.so!CorUnix::CPalSynchronizationManager::BlockThread(CorUnix::CPalThread*, unsigned int, bool, bool, CorUnix::ThreadWakeupReason*, unsigned int*) at /__w/1/s/src/coreclr/pal/src/synchmgr/synchmanager.cpp:301:22
       0x7fe265283b30 0x7fe268131f7f libcoreclr.so!CorUnix::InternalWaitForMultipleObjectsEx(CorUnix::CPalThread*, unsigned int, void* const*, int, unsigned int, int, int) at /__w/1/s/src/coreclr/pal/src/synchmgr/wait.cpp:637:45
       0x7fe265283cf0 0x7fe26813290f libcoreclr.so!WaitForMultipleObjectsEx at /__w/1/s/src/coreclr/pal/src/synchmgr/wait.cpp:201:13
       0x7fe265283d40 0x7fe267f98b66 libcoreclr.so!DebuggerRCThread::MainLoop() at /__w/1/s/src/coreclr/debug/ee/rcthread.cpp:927:14
       0x7fe265283de0 0x7fe267f98989 libcoreclr.so!DebuggerRCThread::ThreadProc() at /__w/1/s/src/coreclr/debug/ee/rcthread.cpp:730:5
       0x7fe265283e20 0x7fe267f983e0 libcoreclr.so!DebuggerRCThread::ThreadProcStatic(void*) at /__w/1/s/src/coreclr/debug/ee/rcthread.cpp:1321:5
       0x7fe265283e40 0x7fe26813f142 libcoreclr.so!CorUnix::CPalThread::ThreadEntry(void*) at /__w/1/s/src/coreclr/pal/src/thread/thread.cpp:1760:16
       0x7fe265283f00 0x7fe268f646db libpthread.so.0!
       0x7fe265283fc0 0x7fe26834e61f libc.so.6!clone
      ----------------------------------
      Thread Id: 0x241c
            Child SP               IP Call Site
       0x7fe264a82530 0x7fe268f6afb9 libpthread.so.0!pthread_cond_timedwait
       0x7fe264a82630 0x7fe268129218 libcoreclr.so!CorUnix::CPalSynchronizationManager::ThreadNativeWait(CorUnix::_ThreadNativeWaitData*, unsigned int, CorUnix::ThreadWakeupReason*, unsigned int*) at /__w/1/s/src/coreclr/pal/src/synchmgr/synchmanager.cpp:0:0
       0x7fe264a826a0 0x7fe26812887b libcoreclr.so!CorUnix::CPalSynchronizationManager::BlockThread(CorUnix::CPalThread*, unsigned int, bool, bool, CorUnix::ThreadWakeupReason*, unsigned int*) at /__w/1/s/src/coreclr/pal/src/synchmgr/synchmanager.cpp:301:22
       0x7fe264a82700 0x7fe268131f7f libcoreclr.so!CorUnix::InternalWaitForMultipleObjectsEx(CorUnix::CPalThread*, unsigned int, void* const*, int, unsigned int, int, int) at /__w/1/s/src/coreclr/pal/src/synchmgr/wait.cpp:637:45
       0x7fe264a828c0 0x7fe26813268c libcoreclr.so!WaitForSingleObjectEx at /__w/1/s/src/coreclr/pal/src/synchmgr/wait.cpp:138:13
       0x7fe264a82910 0x7fe267df5721 libcoreclr.so!CLREventWaitHelper(void*, unsigned int, int)::$_1::operator()(CLREventWaitHelper(void*, unsigned int, int)::Param*) const at /__w/1/s/src/coreclr/vm/synch.cpp:397:24
       (inlined by) CLREventWaitHelper(void*, unsigned int, int) at /__w/1/s/src/coreclr/vm/synch.cpp:399:5
       (inlined by) CLREventBase::WaitEx(unsigned int, WaitMode, PendingSync*) at /__w/1/s/src/coreclr/vm/synch.cpp:466:20
       0x7fe264a829c0 0x7fe267d015af libcoreclr.so!FinalizerThread::WaitForFinalizerEvent(CLREvent*) at /__w/1/s/src/coreclr/vm/finalizerthread.cpp:130:5
       0x7fe264a82a00 0x7fe267d01932 libcoreclr.so!FinalizerThread::FinalizerThreadWorker(void*) at /__w/1/s/src/coreclr/vm/finalizerthread.cpp:262:13
       0x7fe264a82c60 0x7fe267c468be libcoreclr.so!ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) at /__w/1/s/src/coreclr/vm/threads.cpp:0:5
       0x7fe264a82dc0 0x7fe267c470dd libcoreclr.so!ManagedThreadBase_NoADTransition(void (*)(void*), UnhandledExceptionLocation) at /__w/1/s/src/coreclr/vm/threads.cpp:7517:5
       (inlined by) ManagedThreadBase::FinalizerBase(void (*)(void*)) at /__w/1/s/src/coreclr/vm/threads.cpp:7536:5
       0x7fe264a82df0 0x7fe267d01f78 libcoreclr.so!FinalizerThread::FinalizerThreadStart(void*) at /__w/1/s/src/coreclr/vm/finalizerthread.cpp:394:22
       0x7fe264a82e40 0x7fe26813f142 libcoreclr.so!CorUnix::CPalThread::ThreadEntry(void*) at /__w/1/s/src/coreclr/pal/src/thread/thread.cpp:1760:16
       0x7fe264a82f00 0x7fe268f646db libpthread.so.0!
       0x7fe264a82fc0 0x7fe26834e61f libc.so.6!clone
      ----------------------------------
      Thread Id: 0x241d
            Child SP               IP Call Site
       0x7fe2628fe070 0x7fe268f6aad3 libpthread.so.0!pthread_cond_wait
       0x7fe2628fe140 0x7fe2680cc616 libcoreclr.so!GCEvent::Impl::Wait(unsigned int, bool) at /__w/1/s/src/coreclr/gc/unix/events.cpp:149:22
       0x7fe2628fe190 0x7fe267dfa9b2 libcoreclr.so!Thread::RareDisablePreemptiveGC() at /__w/1/s/src/coreclr/vm/threadsuspend.cpp:2161:28
       0x7fe2628fe1e0 0x7fe267ad8c62 libcoreclr.so!void GCHolderBase::PopInternal<1>() at /__w/1/s/src/coreclr/vm/threads.h:5067:27
       0x7fe2628fe210 0x7fe267aeff39 libcoreclr.so!GCPreempThreadExists::~GCPreempThreadExists() at /__w/1/s/src/coreclr/vm/threads.h:5450:5
       0x7fe2628fe220 0x7fe267c09da7 libcoreclr.so!PreStubWorker at /__w/1/s/src/coreclr/vm/prestub.cpp:1957:13
       0x7fe2628fe340 0x7fe267f0dac4 libcoreclr.so!ThePreStub at /__w/1/s/src/coreclr/vm/amd64/theprestubamd64.S:18:0
       0x7fe2628fe430 0x7fe1e9c22246 System.Private.CoreLib.dll!System.PackedSpanHelpers.IndexOfAny[[System.SpanHelpers+DontNegate`1[[System.Int16, System.Private.CoreLib]], System.Private.CoreLib]](Int16 ByRef, Int16, Int16, Int32)
       0x7fe2628fe770 0x7fe1e8bbb532 System.Private.CoreLib.dll!System.Text.ValueStringBuilder.AppendFormatHelper(System.IFormatProvider, System.String, System.ReadOnlySpan`1<System.Object>)
       0x7fe2628fe850 0x7fe1e8a80d33 System.Private.CoreLib.dll!System.String.FormatHelper(System.IFormatProvider, System.String, System.ReadOnlySpan`1<System.Object>)
       0x7fe2628feab0 0x7fe1e9c21c36 ParallelCrash.dll!ParallelCrash.CrashInParallel()
       0x7fe2628fead0 0x7fe267f0cacf 0x7fe267f0cacf
       0x7fe2628feaf0 0x7fe267c93b6b libcoreclr.so!NativeExceptionHolderBase::~NativeExceptionHolderBase() at /__w/1/s/src/coreclr/pal/inc/pal.h:4663:5
       (inlined by) CallDescrWorkerWithHandler(CallDescrData*, int) at /__w/1/s/src/coreclr/vm/callhelpers.cpp:69:5
       0x7fe2628feb40 0x7fe267c93e9c libcoreclr.so!DispatchCallSimple(unsigned long*, unsigned int, unsigned long, unsigned int) at /__w/1/s/src/coreclr/vm/callhelpers.cpp:218:9
       0x7fe2628febc0 0x7fe267cb993b libcoreclr.so!ThreadNative::KickOffThread_Worker(void*) at /__w/1/s/src/coreclr/vm/comsynchronizable.cpp:0:5
       0x7fe2628fec30 0x7fe267c468be libcoreclr.so!ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) at /__w/1/s/src/coreclr/vm/threads.cpp:0:5
       0x7fe2628fed90 0x7fe267c4708d libcoreclr.so!ManagedThreadBase_FullTransition(void (*)(void*), void*, UnhandledExceptionLocation) at /__w/1/s/src/coreclr/vm/threads.cpp:0:5
       (inlined by) ManagedThreadBase::KickOff(void (*)(void*), void*) at /__w/1/s/src/coreclr/vm/threads.cpp:7528:5
       0x7fe2628fedc0 0x7fe267cb9a86 libcoreclr.so!ThreadNative::KickOffThread(void*) at /__w/1/s/src/coreclr/vm/comsynchronizable.cpp:230:9
       0x7fe2628fee40 0x7fe26813f142 libcoreclr.so!CorUnix::CPalThread::ThreadEntry(void*) at /__w/1/s/src/coreclr/pal/src/thread/thread.cpp:1760:16
       0x7fe2628fef00 0x7fe268f646db libpthread.so.0!
       0x7fe2628fefc0 0x7fe26834e61f libc.so.6!clone
      ----------------------------------
      Thread Id: 0x241e
            Child SP               IP Call Site
       0x7fe2620fc220 0x7fe267ecb172 libcoreclr.so!WKS::enter_spin_lock_noinstru(int volatile*) at /__w/1/s/src/coreclr/gc/gc.cpp:1403:39
       (inlined by) WKS::enter_spin_lock(WKS::GCDebugSpinLock*) at /__w/1/s/src/coreclr/gc/gc.cpp:1445:5
       (inlined by) WKS::GCHeap::GarbageCollectGeneration(unsigned int, gc_reason) at /__w/1/s/src/coreclr/gc/gc.cpp:47421:5
       0x7fe2620fc280 0x7fe267f047d0 libcoreclr.so!WKS::GCHeap::GarbageCollect(int, bool, int) at /__w/1/s/src/coreclr/gc/gc.cpp:46685:52
       0x7fe2620fc2c0 0x7fe267f03973 libcoreclr.so!WKS::GCHeap::StressHeap(gc_alloc_context*) at /__w/1/s/src/coreclr/gc/gc.cpp:46303:9
       0x7fe2620fc340 0x7fe267d0442c libcoreclr.so!Thread::HasPendingGCStressInstructionUpdate() at /__w/1/s/src/coreclr/vm/threads.h:4042:21
       (inlined by) DoGcStress(_CONTEXT*, NativeCodeVersion) at /__w/1/s/src/coreclr/vm/gccover.cpp:1768:17
       0x7fe2620fc470 0x7fe267d03f6a libcoreclr.so!OnGcCoverageInterrupt(_CONTEXT*) at /__w/1/s/src/coreclr/vm/gccover.cpp:0:9
       0x7fe2620fc560 0x7fe267b56e72 libcoreclr.so!IsGcMarker(_CONTEXT*, _EXCEPTION_RECORD*) at /__w/1/s/src/coreclr/vm/excep.cpp:6184:19
       0x7fe2620fc5c0 0x7fe267e340da libcoreclr.so!HandleHardwareException(PAL_SEHException*) at /__w/1/s/src/coreclr/vm/exceptionhandling.cpp:5257:59
       0x7fe2620fcb10 0x7fe2680d5fb3 libcoreclr.so!SEHProcessException(PAL_SEHException*) at /__w/1/s/src/coreclr/pal/src/exception/seh.cpp:268:21
       0x7fe2620fcb60 0x7fe2680d8017 libcoreclr.so!common_signal_handler(int, siginfo_t*, void*, int, ...) at /__w/1/s/src/coreclr/pal/src/exception/signal.cpp:1024:9
       0x7fe2620fd290 0x7fe2680d7d0f libcoreclr.so!signal_handler_worker at /__w/1/s/src/coreclr/pal/src/exception/signal.cpp:499:36
       0x7fe2620fd350 0x7fe26814321a libcoreclr.so!CallSignalHandlerWrapper0 at /__w/1/s/src/coreclr/pal/src/arch/amd64/callsignalhandlerwrapper.S:29:0
       0x7fe2620fd3e0 0x7fe1e9c226ec System.Private.CoreLib.dll!System.PackedSpanHelpers.NegateIfNeeded[[System.SpanHelpers+DontNegate`1[[System.Int16, System.Private.CoreLib]], System.Private.CoreLib]](System.Runtime.Intrinsics.Vector256`1<Byte>)
       0x7fe2620fd430 0x7fe1e9c22246 System.Private.CoreLib.dll!System.PackedSpanHelpers.IndexOfAny[[System.SpanHelpers+DontNegate`1[[System.Int16, System.Private.CoreLib]], System.Private.CoreLib]](Int16 ByRef, Int16, Int16, Int32)
       0x7fe2620fd770 0x7fe1e8bbb532 System.Private.CoreLib.dll!System.Text.ValueStringBuilder.AppendFormatHelper(System.IFormatProvider, System.String, System.ReadOnlySpan`1<System.Object>)
       0x7fe2620fd850 0x7fe1e8a80d33 System.Private.CoreLib.dll!System.String.FormatHelper(System.IFormatProvider, System.String, System.ReadOnlySpan`1<System.Object>)
       0x7fe2620fdab0 0x7fe1e9c21c36 ParallelCrash.dll!ParallelCrash.CrashInParallel()
       0x7fe2620fdad0 0x7fe267f0cacf 0x7fe267f0cacf
       0x7fe2620fdaf0 0x7fe267c93b6b libcoreclr.so!NativeExceptionHolderBase::~NativeExceptionHolderBase() at /__w/1/s/src/coreclr/pal/inc/pal.h:4663:5
       (inlined by) CallDescrWorkerWithHandler(CallDescrData*, int) at /__w/1/s/src/coreclr/vm/callhelpers.cpp:69:5
       0x7fe2620fdb40 0x7fe267c93e9c libcoreclr.so!DispatchCallSimple(unsigned long*, unsigned int, unsigned long, unsigned int) at /__w/1/s/src/coreclr/vm/callhelpers.cpp:218:9
       0x7fe2620fdbc0 0x7fe267cb993b libcoreclr.so!ThreadNative::KickOffThread_Worker(void*) at /__w/1/s/src/coreclr/vm/comsynchronizable.cpp:0:5
       0x7fe2620fdc30 0x7fe267c468be libcoreclr.so!ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) at /__w/1/s/src/coreclr/vm/threads.cpp:0:5
       0x7fe2620fdd90 0x7fe267c4708d libcoreclr.so!ManagedThreadBase_FullTransition(void (*)(void*), void*, UnhandledExceptionLocation) at /__w/1/s/src/coreclr/vm/threads.cpp:0:5
       (inlined by) ManagedThreadBase::KickOff(void (*)(void*), void*) at /__w/1/s/src/coreclr/vm/threads.cpp:7528:5
       0x7fe2620fddc0 0x7fe267cb9a86 libcoreclr.so!ThreadNative::KickOffThread(void*) at /__w/1/s/src/coreclr/vm/comsynchronizable.cpp:230:9
       0x7fe2620fde40 0x7fe26813f142 libcoreclr.so!CorUnix::CPalThread::ThreadEntry(void*) at /__w/1/s/src/coreclr/pal/src/thread/thread.cpp:1760:16
       0x7fe2620fdf00 0x7fe268f646db libpthread.so.0!
       0x7fe2620fdfc0 0x7fe26834e61f libc.so.6!clone
      ----------------------------------
      Thread Id: 0x241f
            Child SP               IP Call Site
       0x7fe2618fb170 0x7fe268f6b804 libpthread.so.0!pthread_cond_broadcast
       0x7fe2618fb1e0 0x7fe267dfe7f5 libcoreclr.so!IsGCSpecialThread() at /__w/1/s/src/coreclr/inc/utilcode.h:3672:15
       (inlined by) ThreadSuspend::RestartEE(int, int) at /__w/1/s/src/coreclr/vm/threadsuspend.cpp:5625:5
       0x7fe2618fb220 0x7fe267ecb64f libcoreclr.so!void VolatileStore<int>(int*, int) at /__w/1/s/src/coreclr/inc/volatile.h:272:5
       (inlined by) Volatile<int>::Store(int const&) at /__w/1/s/src/coreclr/inc/volatile.h:392:9
       (inlined by) Volatile<int>::operator=(int) at /__w/1/s/src/coreclr/inc/volatile.h:433:43
       (inlined by) WKS::GCHeap::GarbageCollectGeneration(unsigned int, gc_reason) at /__w/1/s/src/coreclr/gc/gc.cpp:47542:25
       0x7fe2618fb280 0x7fe267f047d0 libcoreclr.so!WKS::GCHeap::GarbageCollect(int, bool, int) at /__w/1/s/src/coreclr/gc/gc.cpp:46685:52
       0x7fe2618fb2c0 0x7fe267f03973 libcoreclr.so!WKS::GCHeap::StressHeap(gc_alloc_context*) at /__w/1/s/src/coreclr/gc/gc.cpp:46303:9
       0x7fe2618fb340 0x7fe267d0442c libcoreclr.so!Thread::HasPendingGCStressInstructionUpdate() at /__w/1/s/src/coreclr/vm/threads.h:4042:21
       (inlined by) DoGcStress(_CONTEXT*, NativeCodeVersion) at /__w/1/s/src/coreclr/vm/gccover.cpp:1768:17
       0x7fe2618fb470 0x7fe267d03f6a libcoreclr.so!OnGcCoverageInterrupt(_CONTEXT*) at /__w/1/s/src/coreclr/vm/gccover.cpp:0:9
       0x7fe2618fb560 0x7fe267b56e72 libcoreclr.so!IsGcMarker(_CONTEXT*, _EXCEPTION_RECORD*) at /__w/1/s/src/coreclr/vm/excep.cpp:6184:19
       0x7fe2618fb5c0 0x7fe267e340da libcoreclr.so!HandleHardwareException(PAL_SEHException*) at /__w/1/s/src/coreclr/vm/exceptionhandling.cpp:5257:59
       0x7fe2618fbb10 0x7fe2680d5fb3 libcoreclr.so!SEHProcessException(PAL_SEHException*) at /__w/1/s/src/coreclr/pal/src/exception/seh.cpp:268:21
       0x7fe2618fbb60 0x7fe2680d8017 libcoreclr.so!common_signal_handler(int, siginfo_t*, void*, int, ...) at /__w/1/s/src/coreclr/pal/src/exception/signal.cpp:1024:9
       0x7fe2618fc290 0x7fe2680d7d0f libcoreclr.so!signal_handler_worker at /__w/1/s/src/coreclr/pal/src/exception/signal.cpp:499:36
       0x7fe2618fc350 0x7fe26814321a libcoreclr.so!CallSignalHandlerWrapper0 at /__w/1/s/src/coreclr/pal/src/arch/amd64/callsignalhandlerwrapper.S:29:0
       0x7fe2618fc3e0 0x7fe1e9c226e8 System.Private.CoreLib.dll!System.PackedSpanHelpers.NegateIfNeeded[[System.SpanHelpers+DontNegate`1[[System.Int16, System.Private.CoreLib]], System.Private.CoreLib]](System.Runtime.Intrinsics.Vector256`1<Byte>)
       0x7fe2618fc430 0x7fe1e9c22246 System.Private.CoreLib.dll!System.PackedSpanHelpers.IndexOfAny[[System.SpanHelpers+DontNegate`1[[System.Int16, System.Private.CoreLib]], System.Private.CoreLib]](Int16 ByRef, Int16, Int16, Int32)
       0x7fe2618fc770 0x7fe1e8bbb532 System.Private.CoreLib.dll!System.Text.ValueStringBuilder.AppendFormatHelper(System.IFormatProvider, System.String, System.ReadOnlySpan`1<System.Object>)
       0x7fe2618fc850 0x7fe1e8a80d33 System.Private.CoreLib.dll!System.String.FormatHelper(System.IFormatProvider, System.String, System.ReadOnlySpan`1<System.Object>)
       0x7fe2618fcab0 0x7fe1e9c21c36 ParallelCrash.dll!ParallelCrash.CrashInParallel()
       0x7fe2618fcad0 0x7fe267f0cacf 0x7fe267f0cacf
       0x7fe2618fcaf0 0x7fe267c93b6b libcoreclr.so!NativeExceptionHolderBase::~NativeExceptionHolderBase() at /__w/1/s/src/coreclr/pal/inc/pal.h:4663:5
       (inlined by) CallDescrWorkerWithHandler(CallDescrData*, int) at /__w/1/s/src/coreclr/vm/callhelpers.cpp:69:5
       0x7fe2618fcb40 0x7fe267c93e9c libcoreclr.so!DispatchCallSimple(unsigned long*, unsigned int, unsigned long, unsigned int) at /__w/1/s/src/coreclr/vm/callhelpers.cpp:218:9
       0x7fe2618fcbc0 0x7fe267cb993b libcoreclr.so!ThreadNative::KickOffThread_Worker(void*) at /__w/1/s/src/coreclr/vm/comsynchronizable.cpp:0:5
       0x7fe2618fcc30 0x7fe267c468be libcoreclr.so!ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) at /__w/1/s/src/coreclr/vm/threads.cpp:0:5
       0x7fe2618fcd90 0x7fe267c4708d libcoreclr.so!ManagedThreadBase_FullTransition(void (*)(void*), void*, UnhandledExceptionLocation) at /__w/1/s/src/coreclr/vm/threads.cpp:0:5
       (inlined by) ManagedThreadBase::KickOff(void (*)(void*), void*) at /__w/1/s/src/coreclr/vm/threads.cpp:7528:5
       0x7fe2618fcdc0 0x7fe267cb9a86 libcoreclr.so!ThreadNative::KickOffThread(void*) at /__w/1/s/src/coreclr/vm/comsynchronizable.cpp:230:9
       0x7fe2618fce40 0x7fe26813f142 libcoreclr.so!CorUnix::CPalThread::ThreadEntry(void*) at /__w/1/s/src/coreclr/pal/src/thread/thread.cpp:1760:16
       0x7fe2618fcf00 0x7fe268f646db libpthread.so.0!
       0x7fe2618fcfc0 0x7fe26834e61f libc.so.6!clone
      ----------------------------------
      Thread Id: 0x2420
            Child SP               IP Call Site
       0x7fe2610fb070 0x7fe268f6aad3 libpthread.so.0!pthread_cond_wait
       0x7fe2610fb140 0x7fe2680cc616 libcoreclr.so!GCEvent::Impl::Wait(unsigned int, bool) at /__w/1/s/src/coreclr/gc/unix/events.cpp:149:22
       0x7fe2610fb190 0x7fe267dfa9b2 libcoreclr.so!Thread::RareDisablePreemptiveGC() at /__w/1/s/src/coreclr/vm/threadsuspend.cpp:2161:28
       0x7fe2610fb1e0 0x7fe267ad8c62 libcoreclr.so!void GCHolderBase::PopInternal<1>() at /__w/1/s/src/coreclr/vm/threads.h:5067:27
       0x7fe2610fb210 0x7fe267aeff39 libcoreclr.so!GCPreempThreadExists::~GCPreempThreadExists() at /__w/1/s/src/coreclr/vm/threads.h:5450:5
       0x7fe2610fb220 0x7fe267c09da7 libcoreclr.so!PreStubWorker at /__w/1/s/src/coreclr/vm/prestub.cpp:1957:13
       0x7fe2610fb340 0x7fe267f0dac4 libcoreclr.so!ThePreStub at /__w/1/s/src/coreclr/vm/amd64/theprestubamd64.S:18:0
       0x7fe2610fb430 0x7fe1e9c22246 System.Private.CoreLib.dll!System.PackedSpanHelpers.IndexOfAny[[System.SpanHelpers+DontNegate`1[[System.Int16, System.Private.CoreLib]], System.Private.CoreLib]](Int16 ByRef, Int16, Int16, Int32)
       0x7fe2610fb770 0x7fe1e8bbb532 System.Private.CoreLib.dll!System.Text.ValueStringBuilder.AppendFormatHelper(System.IFormatProvider, System.String, System.ReadOnlySpan`1<System.Object>)
       0x7fe2610fb850 0x7fe1e8a80d33 System.Private.CoreLib.dll!System.String.FormatHelper(System.IFormatProvider, System.String, System.ReadOnlySpan`1<System.Object>)
       0x7fe2610fbab0 0x7fe1e9c21c36 ParallelCrash.dll!ParallelCrash.CrashInParallel()
       0x7fe2610fbad0 0x7fe267f0cacf 0x7fe267f0cacf
       0x7fe2610fbaf0 0x7fe267c93b6b libcoreclr.so!NativeExceptionHolderBase::~NativeExceptionHolderBase() at /__w/1/s/src/coreclr/pal/inc/pal.h:4663:5
       (inlined by) CallDescrWorkerWithHandler(CallDescrData*, int) at /__w/1/s/src/coreclr/vm/callhelpers.cpp:69:5
       0x7fe2610fbb40 0x7fe267c93e9c libcoreclr.so!DispatchCallSimple(unsigned long*, unsigned int, unsigned long, unsigned int) at /__w/1/s/src/coreclr/vm/callhelpers.cpp:218:9
       0x7fe2610fbbc0 0x7fe267cb993b libcoreclr.so!ThreadNative::KickOffThread_Worker(void*) at /__w/1/s/src/coreclr/vm/comsynchronizable.cpp:0:5
       0x7fe2610fbc30 0x7fe267c468be libcoreclr.so!ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) at /__w/1/s/src/coreclr/vm/threads.cpp:0:5
       0x7fe2610fbd90 0x7fe267c4708d libcoreclr.so!ManagedThreadBase_FullTransition(void (*)(void*), void*, UnhandledExceptionLocation) at /__w/1/s/src/coreclr/vm/threads.cpp:0:5
       (inlined by) ManagedThreadBase::KickOff(void (*)(void*), void*) at /__w/1/s/src/coreclr/vm/threads.cpp:7528:5
       0x7fe2610fbdc0 0x7fe267cb9a86 libcoreclr.so!ThreadNative::KickOffThread(void*) at /__w/1/s/src/coreclr/vm/comsynchronizable.cpp:230:9
       0x7fe2610fbe40 0x7fe26813f142 libcoreclr.so!CorUnix::CPalThread::ThreadEntry(void*) at /__w/1/s/src/coreclr/pal/src/thread/thread.cpp:1760:16
       0x7fe2610fbf00 0x7fe268f646db libpthread.so.0!
       0x7fe2610fbfc0 0x7fe26834e61f libc.so.6!clone
      ----------------------------------
      Thread Id: 0x2421
            Child SP               IP Call Site
       0x7fe2608f90d0 0x7fe268f6aad3 libpthread.so.0!pthread_cond_wait
       0x7fe2608f91a0 0x7fe2680cc616 libcoreclr.so!GCEvent::Impl::Wait(unsigned int, bool) at /__w/1/s/src/coreclr/gc/unix/events.cpp:149:22
       0x7fe2608f91f0 0x7fe267eb7d5f libcoreclr.so!WKS::gc_heap::wait_for_gc_done(int) at /__w/1/s/src/coreclr/gc/gc.cpp:14139:12
       (inlined by) WKS::WaitLongerNoInstru(int) at /__w/1/s/src/coreclr/gc/gc.cpp:1359:13
       0x7fe2608f9220 0x7fe267ecb128 libcoreclr.so!int volatile VolatileLoad<int volatile>(int volatile const*) at /__w/1/s/src/coreclr/inc/volatile.h:188:13
       (inlined by) WKS::enter_spin_lock_noinstru(int volatile*) at /__w/1/s/src/coreclr/gc/gc.cpp:1392:16
       (inlined by) WKS::enter_spin_lock(WKS::GCDebugSpinLock*) at /__w/1/s/src/coreclr/gc/gc.cpp:1445:5
       (inlined by) WKS::GCHeap::GarbageCollectGeneration(unsigned int, gc_reason) at /__w/1/s/src/coreclr/gc/gc.cpp:47421:5
       0x7fe2608f9280 0x7fe267f047d0 libcoreclr.so!WKS::GCHeap::GarbageCollect(int, bool, int) at /__w/1/s/src/coreclr/gc/gc.cpp:46685:52
       0x7fe2608f92c0 0x7fe267f03973 libcoreclr.so!WKS::GCHeap::StressHeap(gc_alloc_context*) at /__w/1/s/src/coreclr/gc/gc.cpp:46303:9
       0x7fe2608f9340 0x7fe267d0442c libcoreclr.so!Thread::HasPendingGCStressInstructionUpdate() at /__w/1/s/src/coreclr/vm/threads.h:4042:21
       (inlined by) DoGcStress(_CONTEXT*, NativeCodeVersion) at /__w/1/s/src/coreclr/vm/gccover.cpp:1768:17
       0x7fe2608f9470 0x7fe267d03f6a libcoreclr.so!OnGcCoverageInterrupt(_CONTEXT*) at /__w/1/s/src/coreclr/vm/gccover.cpp:0:9
       0x7fe2608f9560 0x7fe267b56e72 libcoreclr.so!IsGcMarker(_CONTEXT*, _EXCEPTION_RECORD*) at /__w/1/s/src/coreclr/vm/excep.cpp:6184:19
       0x7fe2608f95c0 0x7fe267e340da libcoreclr.so!HandleHardwareException(PAL_SEHException*) at /__w/1/s/src/coreclr/vm/exceptionhandling.cpp:5257:59
       0x7fe2608f9b10 0x7fe2680d5fb3 libcoreclr.so!SEHProcessException(PAL_SEHException*) at /__w/1/s/src/coreclr/pal/src/exception/seh.cpp:268:21
       0x7fe2608f9b60 0x7fe2680d8017 libcoreclr.so!common_signal_handler(int, siginfo_t*, void*, int, ...) at /__w/1/s/src/coreclr/pal/src/exception/signal.cpp:1024:9
       0x7fe2608fa290 0x7fe2680d7d0f libcoreclr.so!signal_handler_worker at /__w/1/s/src/coreclr/pal/src/exception/signal.cpp:499:36
       0x7fe2608fa350 0x7fe26814321a libcoreclr.so!CallSignalHandlerWrapper0 at /__w/1/s/src/coreclr/pal/src/arch/amd64/callsignalhandlerwrapper.S:29:0
       0x7fe2608fa3e0 0x7fe1e9c226e4 System.Private.CoreLib.dll!System.PackedSpanHelpers.NegateIfNeeded[[System.SpanHelpers+DontNegate`1[[System.Int16, System.Private.CoreLib]], System.Private.CoreLib]](System.Runtime.Intrinsics.Vector256`1<Byte>)
       0x7fe2608fa430 0x7fe1e9c22246 System.Private.CoreLib.dll!System.PackedSpanHelpers.IndexOfAny[[System.SpanHelpers+DontNegate`1[[System.Int16, System.Private.CoreLib]], System.Private.CoreLib]](Int16 ByRef, Int16, Int16, Int32)
       0x7fe2608fa770 0x7fe1e8bbb532 System.Private.CoreLib.dll!System.Text.ValueStringBuilder.AppendFormatHelper(System.IFormatProvider, System.String, System.ReadOnlySpan`1<System.Object>)
       0x7fe2608fa850 0x7fe1e8a80d33 System.Private.CoreLib.dll!System.String.FormatHelper(System.IFormatProvider, System.String, System.ReadOnlySpan`1<System.Object>)
       0x7fe2608faab0 0x7fe1e9c21c36 ParallelCrash.dll!ParallelCrash.CrashInParallel()
       0x7fe2608faad0 0x7fe267f0cacf 0x7fe267f0cacf
       0x7fe2608faaf0 0x7fe267c93b6b libcoreclr.so!NativeExceptionHolderBase::~NativeExceptionHolderBase() at /__w/1/s/src/coreclr/pal/inc/pal.h:4663:5
       (inlined by) CallDescrWorkerWithHandler(CallDescrData*, int) at /__w/1/s/src/coreclr/vm/callhelpers.cpp:69:5
       0x7fe2608fab40 0x7fe267c93e9c libcoreclr.so!DispatchCallSimple(unsigned long*, unsigned int, unsigned long, unsigned int) at /__w/1/s/src/coreclr/vm/callhelpers.cpp:218:9
       0x7fe2608fabc0 0x7fe267cb993b libcoreclr.so!ThreadNative::KickOffThread_Worker(void*) at /__w/1/s/src/coreclr/vm/comsynchronizable.cpp:0:5
       0x7fe2608fac30 0x7fe267c468be libcoreclr.so!ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) at /__w/1/s/src/coreclr/vm/threads.cpp:0:5
       0x7fe2608fad90 0x7fe267c4708d libcoreclr.so!ManagedThreadBase_FullTransition(void (*)(void*), void*, UnhandledExceptionLocation) at /__w/1/s/src/coreclr/vm/threads.cpp:0:5
       (inlined by) ManagedThreadBase::KickOff(void (*)(void*), void*) at /__w/1/s/src/coreclr/vm/threads.cpp:7528:5
       0x7fe2608fadc0 0x7fe267cb9a86 libcoreclr.so!ThreadNative::KickOffThread(void*) at /__w/1/s/src/coreclr/vm/comsynchronizable.cpp:230:9
       0x7fe2608fae40 0x7fe26813f142 libcoreclr.so!CorUnix::CPalThread::ThreadEntry(void*) at /__w/1/s/src/coreclr/pal/src/thread/thread.cpp:1760:16
       0x7fe2608faf00 0x7fe268f646db libpthread.so.0!
       0x7fe2608fafc0 0x7fe26834e61f libc.so.6!clone
      ----------------------------------
      Thread Id: 0x2422
            Child SP               IP Call Site
       0x7fe2600f80d0 0x7fe268f6aad3 libpthread.so.0!pthread_cond_wait
       0x7fe2600f81a0 0x7fe2680cc616 libcoreclr.so!GCEvent::Impl::Wait(unsigned int, bool) at /__w/1/s/src/coreclr/gc/unix/events.cpp:149:22
       0x7fe2600f81f0 0x7fe267eb7d5f libcoreclr.so!WKS::gc_heap::wait_for_gc_done(int) at /__w/1/s/src/coreclr/gc/gc.cpp:14139:12
       (inlined by) WKS::WaitLongerNoInstru(int) at /__w/1/s/src/coreclr/gc/gc.cpp:1359:13
       0x7fe2600f8220 0x7fe267ecb128 libcoreclr.so!int volatile VolatileLoad<int volatile>(int volatile const*) at /__w/1/s/src/coreclr/inc/volatile.h:188:13
       (inlined by) WKS::enter_spin_lock_noinstru(int volatile*) at /__w/1/s/src/coreclr/gc/gc.cpp:1392:16
       (inlined by) WKS::enter_spin_lock(WKS::GCDebugSpinLock*) at /__w/1/s/src/coreclr/gc/gc.cpp:1445:5
       (inlined by) WKS::GCHeap::GarbageCollectGeneration(unsigned int, gc_reason) at /__w/1/s/src/coreclr/gc/gc.cpp:47421:5
       0x7fe2600f8280 0x7fe267f047d0 libcoreclr.so!WKS::GCHeap::GarbageCollect(int, bool, int) at /__w/1/s/src/coreclr/gc/gc.cpp:46685:52
       0x7fe2600f82c0 0x7fe267f03973 libcoreclr.so!WKS::GCHeap::StressHeap(gc_alloc_context*) at /__w/1/s/src/coreclr/gc/gc.cpp:46303:9
       0x7fe2600f8340 0x7fe267d0442c libcoreclr.so!Thread::HasPendingGCStressInstructionUpdate() at /__w/1/s/src/coreclr/vm/threads.h:4042:21
       (inlined by) DoGcStress(_CONTEXT*, NativeCodeVersion) at /__w/1/s/src/coreclr/vm/gccover.cpp:1768:17
       0x7fe2600f8470 0x7fe267d03f6a libcoreclr.so!OnGcCoverageInterrupt(_CONTEXT*) at /__w/1/s/src/coreclr/vm/gccover.cpp:0:9
       0x7fe2600f8560 0x7fe267b56e72 libcoreclr.so!IsGcMarker(_CONTEXT*, _EXCEPTION_RECORD*) at /__w/1/s/src/coreclr/vm/excep.cpp:6184:19
       0x7fe2600f85c0 0x7fe267e340da libcoreclr.so!HandleHardwareException(PAL_SEHException*) at /__w/1/s/src/coreclr/vm/exceptionhandling.cpp:5257:59
       0x7fe2600f8b10 0x7fe2680d5fb3 libcoreclr.so!SEHProcessException(PAL_SEHException*) at /__w/1/s/src/coreclr/pal/src/exception/seh.cpp:268:21
       0x7fe2600f8b60 0x7fe2680d8017 libcoreclr.so!common_signal_handler(int, siginfo_t*, void*, int, ...) at /__w/1/s/src/coreclr/pal/src/exception/signal.cpp:1024:9
       0x7fe2600f9290 0x7fe2680d7d0f libcoreclr.so!signal_handler_worker at /__w/1/s/src/coreclr/pal/src/exception/signal.cpp:499:36
       0x7fe2600f9350 0x7fe26814321a libcoreclr.so!CallSignalHandlerWrapper0 at /__w/1/s/src/coreclr/pal/src/arch/amd64/callsignalhandlerwrapper.S:29:0
       0x7fe2600f93e0 0x7fe1e9c226ec System.Private.CoreLib.dll!System.PackedSpanHelpers.NegateIfNeeded[[System.SpanHelpers+DontNegate`1[[System.Int16, System.Private.CoreLib]], System.Private.CoreLib]](System.Runtime.Intrinsics.Vector256`1<Byte>)
       0x7fe2600f9430 0x7fe1e9c22246 System.Private.CoreLib.dll!System.PackedSpanHelpers.IndexOfAny[[System.SpanHelpers+DontNegate`1[[System.Int16, System.Private.CoreLib]], System.Private.CoreLib]](Int16 ByRef, Int16, Int16, Int32)
       0x7fe2600f9770 0x7fe1e8bbb532 System.Private.CoreLib.dll!System.Text.ValueStringBuilder.AppendFormatHelper(System.IFormatProvider, System.String, System.ReadOnlySpan`1<System.Object>)
       0x7fe2600f9850 0x7fe1e8a80d33 System.Private.CoreLib.dll!System.String.FormatHelper(System.IFormatProvider, System.String, System.ReadOnlySpan`1<System.Object>)
       0x7fe2600f9ab0 0x7fe1e9c21c36 ParallelCrash.dll!ParallelCrash.CrashInParallel()
       0x7fe2600f9ad0 0x7fe267f0cacf 0x7fe267f0cacf
       0x7fe2600f9af0 0x7fe267c93b6b libcoreclr.so!NativeExceptionHolderBase::~NativeExceptionHolderBase() at /__w/1/s/src/coreclr/pal/inc/pal.h:4663:5
       (inlined by) CallDescrWorkerWithHandler(CallDescrData*, int) at /__w/1/s/src/coreclr/vm/callhelpers.cpp:69:5
       0x7fe2600f9b40 0x7fe267c93e9c libcoreclr.so!DispatchCallSimple(unsigned long*, unsigned int, unsigned long, unsigned int) at /__w/1/s/src/coreclr/vm/callhelpers.cpp:218:9
       0x7fe2600f9bc0 0x7fe267cb993b libcoreclr.so!ThreadNative::KickOffThread_Worker(void*) at /__w/1/s/src/coreclr/vm/comsynchronizable.cpp:0:5
       0x7fe2600f9c30 0x7fe267c468be libcoreclr.so!ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) at /__w/1/s/src/coreclr/vm/threads.cpp:0:5
       0x7fe2600f9d90 0x7fe267c4708d libcoreclr.so!ManagedThreadBase_FullTransition(void (*)(void*), void*, UnhandledExceptionLocation) at /__w/1/s/src/coreclr/vm/threads.cpp:0:5
       (inlined by) ManagedThreadBase::KickOff(void (*)(void*), void*) at /__w/1/s/src/coreclr/vm/threads.cpp:7528:5
       0x7fe2600f9dc0 0x7fe267cb9a86 libcoreclr.so!ThreadNative::KickOffThread(void*) at /__w/1/s/src/coreclr/vm/comsynchronizable.cpp:230:9
       0x7fe2600f9e40 0x7fe26813f142 libcoreclr.so!CorUnix::CPalThread::ThreadEntry(void*) at /__w/1/s/src/coreclr/pal/src/thread/thread.cpp:1760:16
       0x7fe2600f9f00 0x7fe268f646db libpthread.so.0!
       0x7fe2600f9fc0 0x7fe26834e61f libc.so.6!clone
      ----------------------------------
      Thread Id: 0x2423
            Child SP               IP Call Site
       0x7fe1e88680d0 0x7fe268f6aad3 libpthread.so.0!pthread_cond_wait
       0x7fe1e88681a0 0x7fe2680cc616 libcoreclr.so!GCEvent::Impl::Wait(unsigned int, bool) at /__w/1/s/src/coreclr/gc/unix/events.cpp:149:22
       0x7fe1e88681f0 0x7fe267eb7d5f libcoreclr.so!WKS::gc_heap::wait_for_gc_done(int) at /__w/1/s/src/coreclr/gc/gc.cpp:14139:12
       (inlined by) WKS::WaitLongerNoInstru(int) at /__w/1/s/src/coreclr/gc/gc.cpp:1359:13
       0x7fe1e8868220 0x7fe267ecb128 libcoreclr.so!int volatile VolatileLoad<int volatile>(int volatile const*) at /__w/1/s/src/coreclr/inc/volatile.h:188:13
       (inlined by) WKS::enter_spin_lock_noinstru(int volatile*) at /__w/1/s/src/coreclr/gc/gc.cpp:1392:16
       (inlined by) WKS::enter_spin_lock(WKS::GCDebugSpinLock*) at /__w/1/s/src/coreclr/gc/gc.cpp:1445:5
       (inlined by) WKS::GCHeap::GarbageCollectGeneration(unsigned int, gc_reason) at /__w/1/s/src/coreclr/gc/gc.cpp:47421:5
       0x7fe1e8868280 0x7fe267f047d0 libcoreclr.so!WKS::GCHeap::GarbageCollect(int, bool, int) at /__w/1/s/src/coreclr/gc/gc.cpp:46685:52
       0x7fe1e88682c0 0x7fe267f03973 libcoreclr.so!WKS::GCHeap::StressHeap(gc_alloc_context*) at /__w/1/s/src/coreclr/gc/gc.cpp:46303:9
       0x7fe1e8868340 0x7fe267d0442c libcoreclr.so!Thread::HasPendingGCStressInstructionUpdate() at /__w/1/s/src/coreclr/vm/threads.h:4042:21
       (inlined by) DoGcStress(_CONTEXT*, NativeCodeVersion) at /__w/1/s/src/coreclr/vm/gccover.cpp:1768:17
       0x7fe1e8868470 0x7fe267d03f6a libcoreclr.so!OnGcCoverageInterrupt(_CONTEXT*) at /__w/1/s/src/coreclr/vm/gccover.cpp:0:9
       0x7fe1e8868560 0x7fe267b56e72 libcoreclr.so!IsGcMarker(_CONTEXT*, _EXCEPTION_RECORD*) at /__w/1/s/src/coreclr/vm/excep.cpp:6184:19
       0x7fe1e88685c0 0x7fe267e340da libcoreclr.so!HandleHardwareException(PAL_SEHException*) at /__w/1/s/src/coreclr/vm/exceptionhandling.cpp:5257:59
       0x7fe1e8868b10 0x7fe2680d5fb3 libcoreclr.so!SEHProcessException(PAL_SEHException*) at /__w/1/s/src/coreclr/pal/src/exception/seh.cpp:268:21
       0x7fe1e8868b60 0x7fe2680d8017 libcoreclr.so!common_signal_handler(int, siginfo_t*, void*, int, ...) at /__w/1/s/src/coreclr/pal/src/exception/signal.cpp:1024:9
       0x7fe1e8869290 0x7fe2680d7d0f libcoreclr.so!signal_handler_worker at /__w/1/s/src/coreclr/pal/src/exception/signal.cpp:499:36
       0x7fe1e8869350 0x7fe26814321a libcoreclr.so!CallSignalHandlerWrapper0 at /__w/1/s/src/coreclr/pal/src/arch/amd64/callsignalhandlerwrapper.S:29:0
       0x7fe1e88693e0 0x7fe1e9c226e8 System.Private.CoreLib.dll!System.PackedSpanHelpers.NegateIfNeeded[[System.SpanHelpers+DontNegate`1[[System.Int16, System.Private.CoreLib]], System.Private.CoreLib]](System.Runtime.Intrinsics.Vector256`1<Byte>)
       0x7fe1e8869430 0x7fe1e9c22246 System.Private.CoreLib.dll!System.PackedSpanHelpers.IndexOfAny[[System.SpanHelpers+DontNegate`1[[System.Int16, System.Private.CoreLib]], System.Private.CoreLib]](Int16 ByRef, Int16, Int16, Int32)
       0x7fe1e8869770 0x7fe1e8bbb532 System.Private.CoreLib.dll!System.Text.ValueStringBuilder.AppendFormatHelper(System.IFormatProvider, System.String, System.ReadOnlySpan`1<System.Object>)
       0x7fe1e8869850 0x7fe1e8a80d33 System.Private.CoreLib.dll!System.String.FormatHelper(System.IFormatProvider, System.String, System.ReadOnlySpan`1<System.Object>)
       0x7fe1e8869ab0 0x7fe1e9c21c36 ParallelCrash.dll!ParallelCrash.CrashInParallel()
       0x7fe1e8869ad0 0x7fe267f0cacf 0x7fe267f0cacf
       0x7fe1e8869af0 0x7fe267c93b6b libcoreclr.so!NativeExceptionHolderBase::~NativeExceptionHolderBase() at /__w/1/s/src/coreclr/pal/inc/pal.h:4663:5
       (inlined by) CallDescrWorkerWithHandler(CallDescrData*, int) at /__w/1/s/src/coreclr/vm/callhelpers.cpp:69:5
       0x7fe1e8869b40 0x7fe267c93e9c libcoreclr.so!DispatchCallSimple(unsigned long*, unsigned int, unsigned long, unsigned int) at /__w/1/s/src/coreclr/vm/callhelpers.cpp:218:9
       0x7fe1e8869bc0 0x7fe267cb993b libcoreclr.so!ThreadNative::KickOffThread_Worker(void*) at /__w/1/s/src/coreclr/vm/comsynchronizable.cpp:0:5
       0x7fe1e8869c30 0x7fe267c468be libcoreclr.so!ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) at /__w/1/s/src/coreclr/vm/threads.cpp:0:5
       0x7fe1e8869d90 0x7fe267c4708d libcoreclr.so!ManagedThreadBase_FullTransition(void (*)(void*), void*, UnhandledExceptionLocation) at /__w/1/s/src/coreclr/vm/threads.cpp:0:5
       (inlined by) ManagedThreadBase::KickOff(void (*)(void*), void*) at /__w/1/s/src/coreclr/vm/threads.cpp:7528:5
       0x7fe1e8869dc0 0x7fe267cb9a86 libcoreclr.so!ThreadNative::KickOffThread(void*) at /__w/1/s/src/coreclr/vm/comsynchronizable.cpp:230:9
       0x7fe1e8869e40 0x7fe26813f142 libcoreclr.so!CorUnix::CPalThread::ThreadEntry(void*) at /__w/1/s/src/coreclr/pal/src/thread/thread.cpp:1760:16
       0x7fe1e8869f00 0x7fe268f646db libpthread.so.0!
       0x7fe1e8869fc0 0x7fe26834e61f libc.so.6!clone
      ----------------------------------
      Thread Id: 0x2424
            Child SP               IP Call Site
       0x7fa1c34cd070 0x7fe268f6aad3 libpthread.so.0!pthread_cond_wait
       0x7fa1c34cd140 0x7fe2680cc616 libcoreclr.so!GCEvent::Impl::Wait(unsigned int, bool) at /__w/1/s/src/coreclr/gc/unix/events.cpp:149:22
       0x7fa1c34cd190 0x7fe267dfa9b2 libcoreclr.so!Thread::RareDisablePreemptiveGC() at /__w/1/s/src/coreclr/vm/threadsuspend.cpp:2161:28
       0x7fa1c34cd1e0 0x7fe267ad8c62 libcoreclr.so!void GCHolderBase::PopInternal<1>() at /__w/1/s/src/coreclr/vm/threads.h:5067:27
       0x7fa1c34cd210 0x7fe267aeff39 libcoreclr.so!GCPreempThreadExists::~GCPreempThreadExists() at /__w/1/s/src/coreclr/vm/threads.h:5450:5
       0x7fa1c34cd220 0x7fe267c09da7 libcoreclr.so!PreStubWorker at /__w/1/s/src/coreclr/vm/prestub.cpp:1957:13
       0x7fa1c34cd340 0x7fe267f0dac4 libcoreclr.so!ThePreStub at /__w/1/s/src/coreclr/vm/amd64/theprestubamd64.S:18:0
       0x7fa1c34cd430 0x7fe1e9c22246 System.Private.CoreLib.dll!System.PackedSpanHelpers.IndexOfAny[[System.SpanHelpers+DontNegate`1[[System.Int16, System.Private.CoreLib]], System.Private.CoreLib]](Int16 ByRef, Int16, Int16, Int32)
       0x7fa1c34cd770 0x7fe1e8bbb532 System.Private.CoreLib.dll!System.Text.ValueStringBuilder.AppendFormatHelper(System.IFormatProvider, System.String, System.ReadOnlySpan`1<System.Object>)
       0x7fa1c34cd850 0x7fe1e8a80d33 System.Private.CoreLib.dll!System.String.FormatHelper(System.IFormatProvider, System.String, System.ReadOnlySpan`1<System.Object>)
       0x7fa1c34cdab0 0x7fe1e9c21c36 ParallelCrash.dll!ParallelCrash.CrashInParallel()
       0x7fa1c34cdad0 0x7fe267f0cacf 0x7fe267f0cacf
       0x7fa1c34cdaf0 0x7fe267c93b6b libcoreclr.so!NativeExceptionHolderBase::~NativeExceptionHolderBase() at /__w/1/s/src/coreclr/pal/inc/pal.h:4663:5
       (inlined by) CallDescrWorkerWithHandler(CallDescrData*, int) at /__w/1/s/src/coreclr/vm/callhelpers.cpp:69:5
       0x7fa1c34cdb40 0x7fe267c93e9c libcoreclr.so!DispatchCallSimple(unsigned long*, unsigned int, unsigned long, unsigned int) at /__w/1/s/src/coreclr/vm/callhelpers.cpp:218:9
       0x7fa1c34cdbc0 0x7fe267cb993b libcoreclr.so!ThreadNative::KickOffThread_Worker(void*) at /__w/1/s/src/coreclr/vm/comsynchronizable.cpp:0:5
       0x7fa1c34cdc30 0x7fe267c468be libcoreclr.so!ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) at /__w/1/s/src/coreclr/vm/threads.cpp:0:5
       0x7fa1c34cdd90 0x7fe267c4708d libcoreclr.so!ManagedThreadBase_FullTransition(void (*)(void*), void*, UnhandledExceptionLocation) at /__w/1/s/src/coreclr/vm/threads.cpp:0:5
       (inlined by) ManagedThreadBase::KickOff(void (*)(void*), void*) at /__w/1/s/src/coreclr/vm/threads.cpp:7528:5
       0x7fa1c34cddc0 0x7fe267cb9a86 libcoreclr.so!ThreadNative::KickOffThread(void*) at /__w/1/s/src/coreclr/vm/comsynchronizable.cpp:230:9
       0x7fa1c34cde40 0x7fe26813f142 libcoreclr.so!CorUnix::CPalThread::ThreadEntry(void*) at /__w/1/s/src/coreclr/pal/src/thread/thread.cpp:1760:16
       0x7fa1c34cdf00 0x7fe268f646db libpthread.so.0!
       0x7fa1c34cdfc0 0x7fe26834e61f libc.so.6!clone
      ----------------------------------
      Thread Id: 0x2425
            Child SP               IP Call Site
       0x7fa1c2ccb0d0 0x7fe268f6aad3 libpthread.so.0!pthread_cond_wait
       0x7fa1c2ccb1a0 0x7fe2680cc616 libcoreclr.so!GCEvent::Impl::Wait(unsigned int, bool) at /__w/1/s/src/coreclr/gc/unix/events.cpp:149:22
       0x7fa1c2ccb1f0 0x7fe267eb7d5f libcoreclr.so!WKS::gc_heap::wait_for_gc_done(int) at /__w/1/s/src/coreclr/gc/gc.cpp:14139:12
       (inlined by) WKS::WaitLongerNoInstru(int) at /__w/1/s/src/coreclr/gc/gc.cpp:1359:13
       0x7fa1c2ccb220 0x7fe267ecb128 libcoreclr.so!int volatile VolatileLoad<int volatile>(int volatile const*) at /__w/1/s/src/coreclr/inc/volatile.h:188:13
       (inlined by) WKS::enter_spin_lock_noinstru(int volatile*) at /__w/1/s/src/coreclr/gc/gc.cpp:1392:16
       (inlined by) WKS::enter_spin_lock(WKS::GCDebugSpinLock*) at /__w/1/s/src/coreclr/gc/gc.cpp:1445:5
       (inlined by) WKS::GCHeap::GarbageCollectGeneration(unsigned int, gc_reason) at /__w/1/s/src/coreclr/gc/gc.cpp:47421:5
       0x7fa1c2ccb280 0x7fe267f047d0 libcoreclr.so!WKS::GCHeap::GarbageCollect(int, bool, int) at /__w/1/s/src/coreclr/gc/gc.cpp:46685:52
       0x7fa1c2ccb2c0 0x7fe267f03973 libcoreclr.so!WKS::GCHeap::StressHeap(gc_alloc_context*) at /__w/1/s/src/coreclr/gc/gc.cpp:46303:9
       0x7fa1c2ccb340 0x7fe267d0442c libcoreclr.so!Thread::HasPendingGCStressInstructionUpdate() at /__w/1/s/src/coreclr/vm/threads.h:4042:21
       (inlined by) DoGcStress(_CONTEXT*, NativeCodeVersion) at /__w/1/s/src/coreclr/vm/gccover.cpp:1768:17
       0x7fa1c2ccb470 0x7fe267d03f6a libcoreclr.so!OnGcCoverageInterrupt(_CONTEXT*) at /__w/1/s/src/coreclr/vm/gccover.cpp:0:9
       0x7fa1c2ccb560 0x7fe267b56e72 libcoreclr.so!IsGcMarker(_CONTEXT*, _EXCEPTION_RECORD*) at /__w/1/s/src/coreclr/vm/excep.cpp:6184:19
       0x7fa1c2ccb5c0 0x7fe267e340da libcoreclr.so!HandleHardwareException(PAL_SEHException*) at /__w/1/s/src/coreclr/vm/exceptionhandling.cpp:5257:59
       0x7fa1c2ccbb10 0x7fe2680d5fb3 libcoreclr.so!SEHProcessException(PAL_SEHException*) at /__w/1/s/src/coreclr/pal/src/exception/seh.cpp:268:21
       0x7fa1c2ccbb60 0x7fe2680d8017 libcoreclr.so!common_signal_handler(int, siginfo_t*, void*, int, ...) at /__w/1/s/src/coreclr/pal/src/exception/signal.cpp:1024:9
       0x7fa1c2ccc290 0x7fe2680d7d0f libcoreclr.so!signal_handler_worker at /__w/1/s/src/coreclr/pal/src/exception/signal.cpp:499:36
       0x7fa1c2ccc350 0x7fe26814321a libcoreclr.so!CallSignalHandlerWrapper0 at /__w/1/s/src/coreclr/pal/src/arch/amd64/callsignalhandlerwrapper.S:29:0
       0x7fa1c2ccc3e0 0x7fe1e9c226e4 System.Private.CoreLib.dll!System.PackedSpanHelpers.NegateIfNeeded[[System.SpanHelpers+DontNegate`1[[System.Int16, System.Private.CoreLib]], System.Private.CoreLib]](System.Runtime.Intrinsics.Vector256`1<Byte>)
       0x7fa1c2ccc430 0x7fe1e9c22246 System.Private.CoreLib.dll!System.PackedSpanHelpers.IndexOfAny[[System.SpanHelpers+DontNegate`1[[System.Int16, System.Private.CoreLib]], System.Private.CoreLib]](Int16 ByRef, Int16, Int16, Int32)
       0x7fa1c2ccc770 0x7fe1e8bbb532 System.Private.CoreLib.dll!System.Text.ValueStringBuilder.AppendFormatHelper(System.IFormatProvider, System.String, System.ReadOnlySpan`1<System.Object>)
       0x7fa1c2ccc850 0x7fe1e8a80d33 System.Private.CoreLib.dll!System.String.FormatHelper(System.IFormatProvider, System.String, System.ReadOnlySpan`1<System.Object>)
       0x7fa1c2cccab0 0x7fe1e9c21c36 ParallelCrash.dll!ParallelCrash.CrashInParallel()
       0x7fa1c2cccad0 0x7fe267f0cacf 0x7fe267f0cacf
       0x7fa1c2cccaf0 0x7fe267c93b6b libcoreclr.so!NativeExceptionHolderBase::~NativeExceptionHolderBase() at /__w/1/s/src/coreclr/pal/inc/pal.h:4663:5
       (inlined by) CallDescrWorkerWithHandler(CallDescrData*, int) at /__w/1/s/src/coreclr/vm/callhelpers.cpp:69:5
       0x7fa1c2cccb40 0x7fe267c93e9c libcoreclr.so!DispatchCallSimple(unsigned long*, unsigned int, unsigned long, unsigned int) at /__w/1/s/src/coreclr/vm/callhelpers.cpp:218:9
       0x7fa1c2cccbc0 0x7fe267cb993b libcoreclr.so!ThreadNative::KickOffThread_Worker(void*) at /__w/1/s/src/coreclr/vm/comsynchronizable.cpp:0:5
       0x7fa1c2cccc30 0x7fe267c468be libcoreclr.so!ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) at /__w/1/s/src/coreclr/vm/threads.cpp:0:5
       0x7fa1c2cccd90 0x7fe267c4708d libcoreclr.so!ManagedThreadBase_FullTransition(void (*)(void*), void*, UnhandledExceptionLocation) at /__w/1/s/src/coreclr/vm/threads.cpp:0:5
       (inlined by) ManagedThreadBase::KickOff(void (*)(void*), void*) at /__w/1/s/src/coreclr/vm/threads.cpp:7528:5
       0x7fa1c2cccdc0 0x7fe267cb9a86 libcoreclr.so!ThreadNative::KickOffThread(void*) at /__w/1/s/src/coreclr/vm/comsynchronizable.cpp:230:9
       0x7fa1c2ccce40 0x7fe26813f142 libcoreclr.so!CorUnix::CPalThread::ThreadEntry(void*) at /__w/1/s/src/coreclr/pal/src/thread/thread.cpp:1760:16
       0x7fa1c2cccf00 0x7fe268f646db libpthread.so.0!
       0x7fa1c2cccfc0 0x7fe26834e61f libc.so.6!clone
      ----------------------------------
      Thread Id: 0x2426
            Child SP               IP Call Site
       0x7fa1c24cb070 0x7fe268f6aad3 libpthread.so.0!pthread_cond_wait
       0x7fa1c24cb140 0x7fe2680cc616 libcoreclr.so!GCEvent::Impl::Wait(unsigned int, bool) at /__w/1/s/src/coreclr/gc/unix/events.cpp:149:22
       0x7fa1c24cb190 0x7fe267dfa9b2 libcoreclr.so!Thread::RareDisablePreemptiveGC() at /__w/1/s/src/coreclr/vm/threadsuspend.cpp:2161:28
       0x7fa1c24cb1e0 0x7fe267ad8c62 libcoreclr.so!void GCHolderBase::PopInternal<1>() at /__w/1/s/src/coreclr/vm/threads.h:5067:27
       0x7fa1c24cb210 0x7fe267aeff39 libcoreclr.so!GCPreempThreadExists::~GCPreempThreadExists() at /__w/1/s/src/coreclr/vm/threads.h:5450:5
       0x7fa1c24cb220 0x7fe267c09da7 libcoreclr.so!PreStubWorker at /__w/1/s/src/coreclr/vm/prestub.cpp:1957:13
       0x7fa1c24cb340 0x7fe267f0dac4 libcoreclr.so!ThePreStub at /__w/1/s/src/coreclr/vm/amd64/theprestubamd64.S:18:0
       0x7fa1c24cb430 0x7fe1e9c22246 System.Private.CoreLib.dll!System.PackedSpanHelpers.IndexOfAny[[System.SpanHelpers+DontNegate`1[[System.Int16, System.Private.CoreLib]], System.Private.CoreLib]](Int16 ByRef, Int16, Int16, Int32)
       0x7fa1c24cb770 0x7fe1e8bbb532 System.Private.CoreLib.dll!System.Text.ValueStringBuilder.AppendFormatHelper(System.IFormatProvider, System.String, System.ReadOnlySpan`1<System.Object>)
       0x7fa1c24cb850 0x7fe1e8a80d33 System.Private.CoreLib.dll!System.String.FormatHelper(System.IFormatProvider, System.String, System.ReadOnlySpan`1<System.Object>)
       0x7fa1c24cbab0 0x7fe1e9c21c36 ParallelCrash.dll!ParallelCrash.CrashInParallel()
       0x7fa1c24cbad0 0x7fe267f0cacf 0x7fe267f0cacf
       0x7fa1c24cbaf0 0x7fe267c93b6b libcoreclr.so!NativeExceptionHolderBase::~NativeExceptionHolderBase() at /__w/1/s/src/coreclr/pal/inc/pal.h:4663:5
       (inlined by) CallDescrWorkerWithHandler(CallDescrData*, int) at /__w/1/s/src/coreclr/vm/callhelpers.cpp:69:5
       0x7fa1c24cbb40 0x7fe267c93e9c libcoreclr.so!DispatchCallSimple(unsigned long*, unsigned int, unsigned long, unsigned int) at /__w/1/s/src/coreclr/vm/callhelpers.cpp:218:9
       0x7fa1c24cbbc0 0x7fe267cb993b libcoreclr.so!ThreadNative::KickOffThread_Worker(void*) at /__w/1/s/src/coreclr/vm/comsynchronizable.cpp:0:5
       0x7fa1c24cbc30 0x7fe267c468be libcoreclr.so!ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) at /__w/1/s/src/coreclr/vm/threads.cpp:0:5
       0x7fa1c24cbd90 0x7fe267c4708d libcoreclr.so!ManagedThreadBase_FullTransition(void (*)(void*), void*, UnhandledExceptionLocation) at /__w/1/s/src/coreclr/vm/threads.cpp:0:5
       (inlined by) ManagedThreadBase::KickOff(void (*)(void*), void*) at /__w/1/s/src/coreclr/vm/threads.cpp:7528:5
       0x7fa1c24cbdc0 0x7fe267cb9a86 libcoreclr.so!ThreadNative::KickOffThread(void*) at /__w/1/s/src/coreclr/vm/comsynchronizable.cpp:230:9
       0x7fa1c24cbe40 0x7fe26813f142 libcoreclr.so!CorUnix::CPalThread::ThreadEntry(void*) at /__w/1/s/src/coreclr/pal/src/thread/thread.cpp:1760:16
       0x7fa1c24cbf00 0x7fe268f646db libpthread.so.0!
       0x7fa1c24cbfc0 0x7fe26834e61f libc.so.6!clone

BruceForstall added a commit that referenced this issue Feb 14, 2023
@BruceForstall
Temporarily disable ParallelCrash, ParallelCrashMainThread under GCSt…
adfccb1 
…ress (#82066)

These are regularly failing in GCStress testing.

Tracking: #80356
@BruceForstall BruceForstall added disabled-test The test is disabled in source code against the issue and removed blocking-clean-ci-optional Blocking optional rolling runs labels Feb 14, 2023
@BruceForstall
Copy link
Member

Now disabled under GCStress pending failure investigation.

@mangod9 mangod9 removed the untriaged New issue has not been triaged by the area owner label Jul 24, 2023
@mangod9 mangod9 added this to the Future milestone Jul 24, 2023
@v-wenyuxu
Copy link
Author

Failed again in: runtime-coreclr jitstress 20230917.1

Failed tests:

coreclr windows x86 Checked jitstress1 @ Windows.10.Amd64.Open
    - baseservices\\exceptions\\simple\\ParallelCrashWorkerThreads\\ParallelCrashWorkerThreads.cmd

Error message:

Process terminated. Parallel crash in thread 8!

   at System.Environment.FailFast(System.String)
   at ParallelCrash.CrashInParallel()
   at System.Threading.Thread.StartCallback()

cmdLine:C:\h\w\AF4809D1\w\B3E4098E\e\baseservices\exceptions\simple\ParallelCrashWorkerThreads\ParallelCrashWorkerThreads.cmd Timed Out (timeout in milliseconds: 1800000 from variable __TestTimeout, start: 9/18/2023 5:42:31 AM, end: 9/18/2023 6:12:31 AM)

Return code:      -100
Raw output file:      C:\h\w\AF4809D1\w\B3E4098E\uploads\Reports\baseservices.exceptions\simple\ParallelCrashWorkerThreads\ParallelCrashWorkerThreads.output.txt
Raw output:
BEGIN EXECUTION
 cmd /c ..\ParallelCrash\ParallelCrash.cmd
BEGIN EXECUTION
 "C:\h\w\AF4809D1\p\corerun.exe" -p "System.Reflection.Metadata.MetadataUpdater.IsSupported=false" -p "System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true"  ParallelCrash.dll 2 

cmdLine:C:\h\w\AF4809D1\w\B3E4098E\e\baseservices\exceptions\simple\ParallelCrashWorkerThreads\ParallelCrashWorkerThreads.cmd Timed Out (timeout in milliseconds: 1800000 from variable __TestTimeout, start: 9/18/2023 5:42:31 AM, end: 9/18/2023 6:12:31 AM)
Invoking: C:\Program Files (x86)\Windows Kits\10\Debuggers\x86\cdb.exe -c "$<C:\h\w\AF4809D1\t\tmp3iasdb.tmp" -z "C:\cores\crashdump_2428.dmp"
stdout: 
Microsoft (R) Windows Debugger Version 10.0.18362.1 X86
Copyright (c) Microsoft Corporation. All rights reserved.


Loading Dump File [C:\cores\crashdump_2428.dmp]
User Mini Dump File with Full Memory: Only application data is available


************* Path validation summary **************
Response                         Time (ms)     Location
OK                                             C:\h\w\AF4809D1\p\PDB
Symbol search path is: C:\h\w\AF4809D1\p\PDB
Executable search path is: 
Windows 10 Version 14393 MP (4 procs) Free x86 compatible
Product: Server, suite: TerminalServer DataCenter SingleUserTS
10.0.14393.5125 (rs1_release.220429-1732)
Machine Name:
Debug session time: Mon Sep 18 06:12:31.000 2023 (UTC + 0:00)
System Uptime: 0 days 0:34:49.118
Process Uptime: 0 days 0:29:52.000
ERROR: Reload failure, partially initialized target
WaitForEvent failed, HRESULT 0x8000FFFF
Catastrophic failure
When WaitForEvent fails on dump files the current state is not displayed.
If you are having symbol problems you will need to restart the debugger
after you correct the symbol path.  .restart can also be used.
?:???> cdb: Reading initial command '$<C:\h\w\AF4809D1\t\tmp3iasdb.tmp'
?:???> .load C:\Users\runner\.dotnet\sos\sos.dll
WARNING: The debugger does not have a current process or thread
WARNING: Many commands will not work
?:???> ~*k
WARNING: The debugger does not have a current process or thread
WARNING: Many commands will not work
       ^ Illegal thread error in '~*k'
?:???> !clrstack -f -all
WARNING: The debugger does not have a current process or thread
WARNING: Many commands will not work
Extension called without current PC
SOS does not support the current target architecture '' (0x0000). A 32 bit target may require a 32 bit debugger or vice versa. In general, try to use the same bitness for the debugger and target process.
?:???> q
WARNING: The debugger does not have a current process or thread
WARNING: Many commands will not work
quit:
NatVis script unloaded from 'C:\Program Files (x86)\Windows Kits\10\Debuggers\x86\Visualizers\atlmfc.natvis'
NatVis script unloaded from 'C:\Program Files (x86)\Windows Kits\10\Debuggers\x86\Visualizers\concurrency.natvis'
NatVis script unloaded from 'C:\Program Files (x86)\Windows Kits\10\Debuggers\x86\Visualizers\cpp_rest.natvis'
NatVis script unloaded from 'C:\Program Files (x86)\Windows Kits\10\Debuggers\x86\Visualizers\stl.natvis'
NatVis script unloaded from 'C:\Program Files (x86)\Windows Kits\10\Debuggers\x86\Visualizers\Windows.Data.Json.natvis'
NatVis script unloaded from 'C:\Program Files (x86)\Windows Kits\10\Debuggers\x

Stack trace:

   at baseservices_exceptions._simple_ParallelCrashWorkerThreads_ParallelCrashWorkerThreads_._simple_ParallelCrashWorkerThreads_ParallelCrashWorkerThreads_cmd()
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)

@v-wenyuxu
Copy link
Author

Failed again in: runtime-coreclr r2r-extra 20231015.1

Failed tests:

R2R-CG2 osx arm64 Checked jitstressregs0x80 @ OSX.1200.ARM64.Open
    - baseservices/exceptions/simple/ParallelCrashWorkerThreads/ParallelCrashWorkerThreads.sh

Error message:

../ParallelCrash/ParallelCrash.sh: line 358: -r:/private/tmp/helix/working/9EE6089F/w/A8BE0947/e/baseservices/exceptions/simple/ParallelCrash/IL-CG2/*.dll: No such file or directory
Process terminated. Parallel crash in thread 8!

   at System.Environment.FailFast(System.String)
   at ParallelCrash.CrashInParallel()
   at System.Threading.Thread.StartCallback()

cmdLine:/private/tmp/helix/working/9EE6089F/w/A8BE0947/e/baseservices/exceptions/simple/ParallelCrashWorkerThreads/ParallelCrashWorkerThreads.sh Timed Out (timeout in milliseconds: 5400000 from variable __TestTimeout, start: 10/15/2023 8:50:53 PM, end: 10/15/2023 10:20:53 PM)

Return code:      -100
Raw output file:      /tmp/helix/working/9EE6089F/w/A8BE0947/uploads/Reports/baseservices.exceptions/simple/ParallelCrashWorkerThreads/ParallelCrashWorkerThreads.output.txt
Raw output:
BEGIN EXECUTION
export CDPATH=/private/tmp/helix/working/9EE6089F/w/A8BE0947/e/baseservices/exceptions/simple/ParallelCrashWorkerThreads
/usr/bin/env bash ../ParallelCrash/ParallelCrash.sh '2'
BEGIN EXECUTION
in takeLock
/private/tmp/helix/working/9EE6089F/w/A8BE0947/e/baseservices/exceptions/simple/ParallelCrash/IL-CG2/ParallelCrash.dll
20:50:53
Response file: /private/tmp/helix/working/9EE6089F/w/A8BE0947/e/baseservices/exceptions/simple/ParallelCrash/ParallelCrash.dll.rsp
/private/tmp/helix/working/9EE6089F/w/A8BE0947/e/baseservices/exceptions/simple/ParallelCrash/IL-CG2/ParallelCrash.dll
-o:/private/tmp/helix/working/9EE6089F/w/A8BE0947/e/baseservices/exceptions/simple/ParallelCrash/ParallelCrash.dll
-r:/tmp/helix/working/9EE6089F/p/System.*.dll
-r:/tmp/helix/working/9EE6089F/p/Microsoft.*.dll
-r:/tmp/helix/working/9EE6089F/p/xunit.*.dll
-r:/tmp/helix/working/9EE6089F/p/mscorlib.dll
--verify-type-and-field-layout
--method-layout:random
--targetarch:arm64
--targetos:osx
Running CrossGen2:  dotnet /tmp/helix/working/9EE6089F/p/crossgen2/crossgen2.dll @/private/tmp/helix/working/9EE6089F/w/A8BE0947/e/baseservices/exceptions/simple/ParallelCrash/ParallelCrash.dll.rsp  
Emitting R2R PE file: /private/tmp/helix/working/9EE6089F/w/A8BE0947/e/baseservices/exceptions/simple/ParallelCrash/ParallelCrash.dll
Running R2RDump:  dotnet /tmp/helix/working/9EE6089F/p/R2RDump/R2RDump.dll --header --sc --in /private/tmp/helix/working/9EE6089F/w/A8BE0947/e/baseservices/exceptions/simple/ParallelCrash/ParallelCrash.dll --out /private/tmp/helix/working/9EE6089F/w/A8BE0947/e/baseservices/exceptions/simple/ParallelCrash/ParallelCrash.dll.r2rdump --val
20:50:55
in ReleaseLock
in takeLock
/tmp/helix/working/9EE6089F/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false -p System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true ParallelCrash.dll '2'

cmdLine:/private/tmp/helix/working/9EE6089F/w/A8BE0947/e/baseservices/exceptions/simple/ParallelCrashWorkerThreads/ParallelCrashWorkerThreads.sh Timed Out (timeout in milliseconds: 5400000 from variable __TestTimeout, start: 10/15/2023 8:50:53 PM, end: 10/15/2023 10:20:53 PM)
Test Harness Exitcode is : -100
To run the test:
> set CORE_ROOT=/tmp/helix/working/9EE6089F/p
> /private/tmp/helix/working/9EE6089F/w/A8BE0947/e/baseservices/exceptions/simple/ParallelCrashWorkerThreads/ParallelCrashWorkerThreads.sh
Expected: True
Actual:   False

Stack trace:

   at baseservices_exceptions._simple_ParallelCraWorkerThreads_ParallelCraWorkerThreads_._simple_ParallelCrashWorkerThreads_ParallelCrashWorkerThreads_sh()
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
Queued Pipeline Pipeline_Configuration OS Arch Test Outcome Properties BuildReason
2023-10-16T10:34:01.086Z runtime-coreclr outerloop Checked osx.1200.arm64.open arm64 baseservices/exceptions/simple/ParallelCrashWorkerThreads/ParallelCrashWorkerThreads.sh Failed {"AzurePipelinesTestRunId":"9782924","System.JobAttempt":"1","System.JobId":"765fd5e9-e47f-5cb0-4cbf-499eaaaee93e","operatingSystem":"OSX.1200.ARM64.Open","DefinitionName":"runtime-coreclr outerloop","BuildNumber":"20231016.1","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"439391","Project":"public","configuration":"Checked","System.PhaseAttempt":"1","architecture":"arm64","System.StageName":"Build","System.PhaseName":"run_test_p1__osx_arm64_checked","System.JobName":"__default","DefinitionId":"108","System.StageAttempt":"1","Reason":"Schedule"} Schedule
2023-10-16T10:34:01.086Z runtime-coreclr outerloop Checked osx.1200.arm64.open arm64 baseservices/exceptions/simple/ParallelCrashWorkerThreads/ParallelCrashWorkerThreads.sh Failed {"AzurePipelinesTestRunId":"9782924","System.JobAttempt":"1","System.JobId":"765fd5e9-e47f-5cb0-4cbf-499eaaaee93e","operatingSystem":"OSX.1200.ARM64.Open","DefinitionName":"runtime-coreclr outerloop","BuildNumber":"20231016.1","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"439391","Project":"public","configuration":"Checked","System.PhaseAttempt":"1","architecture":"arm64","System.StageName":"Build","System.PhaseName":"run_test_p1__osx_arm64_checked","System.JobName":"__default","DefinitionId":"108","System.StageAttempt":"1","Reason":"Schedule"} Schedule
2023-10-16T10:34:01.086Z runtime-coreclr outerloop Checked osx.1200.arm64.open arm64 baseservices/exceptions/simple/ParallelCrashWorkerThreads/ParallelCrashWorkerThreads.sh Failed {"AzurePipelinesTestRunId":"9782924","System.JobAttempt":"1","System.JobId":"765fd5e9-e47f-5cb0-4cbf-499eaaaee93e","operatingSystem":"OSX.1200.ARM64.Open","DefinitionName":"runtime-coreclr outerloop","BuildNumber":"20231016.1","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"439391","Project":"public","configuration":"Checked","System.PhaseAttempt":"1","architecture":"arm64","System.StageName":"Build","System.PhaseName":"run_test_p1__osx_arm64_checked","System.JobName":"__default","DefinitionId":"108","System.StageAttempt":"1","Reason":"Schedule"} Schedule
2023-10-16T10:34:01.086Z runtime-coreclr outerloop Checked osx.1200.arm64.open arm64 baseservices/exceptions/simple/ParallelCrashWorkerThreads/ParallelCrashWorkerThreads.sh Failed {"AzurePipelinesTestRunId":"9782924","System.JobAttempt":"1","System.JobId":"765fd5e9-e47f-5cb0-4cbf-499eaaaee93e","operatingSystem":"OSX.1200.ARM64.Open","DefinitionName":"runtime-coreclr outerloop","BuildNumber":"20231016.1","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"439391","Project":"public","configuration":"Checked","System.PhaseAttempt":"1","architecture":"arm64","System.StageName":"Build","System.PhaseName":"run_test_p1__osx_arm64_checked","System.JobName":"__default","DefinitionId":"108","System.StageAttempt":"1","Reason":"Schedule"} Schedule
2023-10-16T10:34:01.086Z runtime-coreclr outerloop Checked osx.1200.arm64.open arm64 baseservices/exceptions/simple/ParallelCrashWorkerThreads/ParallelCrashWorkerThreads.sh Failed {"AzurePipelinesTestRunId":"9782924","System.JobAttempt":"1","System.JobId":"765fd5e9-e47f-5cb0-4cbf-499eaaaee93e","operatingSystem":"OSX.1200.ARM64.Open","DefinitionName":"runtime-coreclr outerloop","BuildNumber":"20231016.1","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"439391","Project":"public","configuration":"Checked","System.PhaseAttempt":"1","architecture":"arm64","System.StageName":"Build","System.PhaseName":"run_test_p1__osx_arm64_checked","System.JobName":"__default","DefinitionId":"108","System.StageAttempt":"1","Reason":"Schedule"} Schedule
More failures
Queued Pipeline Pipeline_Configuration OS Arch Test Outcome Properties BuildReason
2023-10-16T10:34:01.086Z runtime-coreclr outerloop Checked osx.1200.arm64.open arm64 baseservices/exceptions/simple/ParallelCrashWorkerThreads/ParallelCrashWorkerThreads.sh Failed {"AzurePipelinesTestRunId":"9782924","System.JobAttempt":"1","System.JobId":"765fd5e9-e47f-5cb0-4cbf-499eaaaee93e","operatingSystem":"OSX.1200.ARM64.Open","DefinitionName":"runtime-coreclr outerloop","BuildNumber":"20231016.1","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"439391","Project":"public","configuration":"Checked","System.PhaseAttempt":"1","architecture":"arm64","System.StageName":"Build","System.PhaseName":"run_test_p1__osx_arm64_checked","System.JobName":"__default","DefinitionId":"108","System.StageAttempt":"1","Reason":"Schedule"} Schedule
2023-10-16T10:34:01.086Z runtime-coreclr outerloop Checked osx.1200.arm64.open arm64 baseservices/exceptions/simple/ParallelCrashWorkerThreads/ParallelCrashWorkerThreads.sh Failed {"AzurePipelinesTestRunId":"9782924","System.JobAttempt":"1","System.JobId":"765fd5e9-e47f-5cb0-4cbf-499eaaaee93e","operatingSystem":"OSX.1200.ARM64.Open","DefinitionName":"runtime-coreclr outerloop","BuildNumber":"20231016.1","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"439391","Project":"public","configuration":"Checked","System.PhaseAttempt":"1","architecture":"arm64","System.StageName":"Build","System.PhaseName":"run_test_p1__osx_arm64_checked","System.JobName":"__default","DefinitionId":"108","System.StageAttempt":"1","Reason":"Schedule"} Schedule
2023-10-16T10:34:01.086Z runtime-coreclr outerloop Checked osx.1200.arm64.open arm64 baseservices/exceptions/simple/ParallelCrashWorkerThreads/ParallelCrashWorkerThreads.sh Failed {"AzurePipelinesTestRunId":"9782924","System.JobAttempt":"1","System.JobId":"765fd5e9-e47f-5cb0-4cbf-499eaaaee93e","operatingSystem":"OSX.1200.ARM64.Open","DefinitionName":"runtime-coreclr outerloop","BuildNumber":"20231016.1","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"439391","Project":"public","configuration":"Checked","System.PhaseAttempt":"1","architecture":"arm64","System.StageName":"Build","System.PhaseName":"run_test_p1__osx_arm64_checked","System.JobName":"__default","DefinitionId":"108","System.StageAttempt":"1","Reason":"Schedule"} Schedule
2023-10-16T10:34:01.086Z runtime-coreclr outerloop Checked osx.1200.arm64.open arm64 baseservices/exceptions/simple/ParallelCrashWorkerThreads/ParallelCrashWorkerThreads.sh Failed {"AzurePipelinesTestRunId":"9782924","System.JobAttempt":"1","System.JobId":"765fd5e9-e47f-5cb0-4cbf-499eaaaee93e","operatingSystem":"OSX.1200.ARM64.Open","DefinitionName":"runtime-coreclr outerloop","BuildNumber":"20231016.1","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"439391","Project":"public","configuration":"Checked","System.PhaseAttempt":"1","architecture":"arm64","System.StageName":"Build","System.PhaseName":"run_test_p1__osx_arm64_checked","System.JobName":"__default","DefinitionId":"108","System.StageAttempt":"1","Reason":"Schedule"} Schedule
2023-10-16T10:34:01.086Z runtime-coreclr outerloop Checked osx.1200.arm64.open arm64 baseservices/exceptions/simple/ParallelCrashWorkerThreads/ParallelCrashWorkerThreads.sh Failed {"AzurePipelinesTestRunId":"9782924","System.JobAttempt":"1","System.JobId":"765fd5e9-e47f-5cb0-4cbf-499eaaaee93e","operatingSystem":"OSX.1200.ARM64.Open","DefinitionName":"runtime-coreclr outerloop","BuildNumber":"20231016.1","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"439391","Project":"public","configuration":"Checked","System.PhaseAttempt":"1","architecture":"arm64","System.StageName":"Build","System.PhaseName":"run_test_p1__osx_arm64_checked","System.JobName":"__default","DefinitionId":"108","System.StageAttempt":"1","Reason":"Schedule"} Schedule

@kunalspathak
Copy link
Member

Now disabled under GCStress pending failure investigation.

@BruceForstall - seems we didn't disable the test for all modes? They seem to be still failing - https://dev.azure.com/dnceng-public/public/_build/results?buildId=467989&view=ms.vss-test-web.build-test-results-tab&runId=10642174&resultId=143206&paneView=debug

@mangod9 - any plans of investigating this sooner? this is high hit issue.

@kunalspathak
Copy link
Member

Now disabled under GCStress pending failure investigation.

@BruceForstall - seems we didn't disable the test for all modes? They seem to be still failing - https://dev.azure.com/dnceng-public/public/_build/results?buildId=467989&view=ms.vss-test-web.build-test-results-tab&runId=10642174&resultId=143206&paneView=debug

never mind. This one turns out to be a different issue #94652

@BruceForstall BruceForstall mentioned this issue Mar 7, 2024
Closed
@v-wenyuxu
Copy link
Author

Failed in: runtime-coreclr outerloop 20240610.1

Failed tests:

coreclr osx x64 Checked no_tiered_compilation @ OSX.1200.Amd64.Open
    - baseservices/exceptions/simple/ParallelCrashTester/ParallelCrashTester.cmd

Error message:

 Process terminated. Parallel crash in main thread
   at System.Environment.FailFast(System.Runtime.CompilerServices.StackCrawlMarkHandle, System.String, System.Runtime.CompilerServices.ObjectHandleOnStack, System.String)
   at System.Environment.FailFast(System.Threading.StackCrawlMark ByRef, System.String, System.Exception, System.String)
   at System.Environment.FailFast(System.String)
   at ParallelCrash.Main(System.String[])
Process terminated. Parallel crash in thread 4!

   at System.Environment.FailFast(System.Runtime.CompilerServices.StackCrawlMarkHandle, System.String, System.Runtime.CompilerServices.ObjectHandleOnStack, System.String)
   at System.Environment.FailFast(System.Threading.StackCrawlMark ByRef, System.String, System.Exception, System.String)
   at System.Environment.FailFast(System.String)
   at ParallelCrash.CrashInParallel()
   at System.Threading.Thread.StartCallback()
Process terminated. Parallel crash in main thread
   at System.Environment.FailFast(System.Runtime.CompilerServices.StackCrawlMarkHandle, System.String, System.Runtime.CompilerServices.ObjectHandleOnStack, System.String)
   at System.Environment.FailFast(System.Threading.StackCrawlMark ByRef, System.String, System.Exception, System.String)
   at System.Environment.FailFast(System.String)
   at ParallelCrash.Main(System.String[])

cmdLine:/private/tmp/helix/working/AB0E09C4/w/AB1C09CE/e/baseservices/exceptions/baseservices-exceptions/../simple/ParallelCrashTester/ParallelCrashTester.sh Timed Out (timeout in milliseconds: 600000 from variable __TestTimeout, start: 6/10/2024 6:19:06 AM, end: 6/10/2024 6:29:06 AM)

Return code:      -100
Raw output file:      /tmp/helix/working/AB0E09C4/w/AB1C09CE/uploads/simple/ParallelCrashTester/output.txt
Raw output:
BEGIN EXECUTION
/tmp/helix/working/AB0E09C4/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false -p System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true ParallelCrashTester.dll ''
Running ParallelCrash test(1)
Running ParallelCrash test(2)
Running ParallelCrash test(0)

cmdLine:/private/tmp/helix/working/AB0E09C4/w/AB1C09CE/e/baseservices/exceptions/baseservices-exceptions/../simple/ParallelCrashTester/ParallelCrashTester.sh Timed Out (timeout in milliseconds: 600000 from variable __TestTimeout, start: 6/10/2024 6:19:06 AM, end: 6/10/2024 6:29:06 AM)
Printing stacktrace from '/cores/crashdump_62577.dmp.crashreport.json'
Invoking llvm-symbolizer --pretty-print
Errors while running llvm-symbolizer --pretty-print
System.ComponentModel.Win32Exception (2): An error occurred trying to start process 'llvm-symbolizer' with working directory '/private/tmp/helix/working/AB0E09C4/w/AB1C09CE/e/baseservices/exceptions/baseservices-exceptions'. No such file or directory
   at System.Diagnostics.Process.ForkAndExecProcess(ProcessStartInfo startInfo, String resolvedFilename, String[] argv, String[] envp, String cwd, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, Int32& stdinFd, Int32& stdoutFd, Int32& stderrFd, Boolean usesTerminal, Boolean throwOnNoExec) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.Unix.cs:line 496
   at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.Unix.cs:line 456
   at CoreclrTestLib.CoreclrTestWrapperLib.TryPrintStackTraceFromCrashReport(String crashReportJsonFile, TextWriter outputWriter)
Test Harness Exitcode is : -100
To run the test:
Set up CORE_ROOT and run.
> /private/tmp/helix/working/AB0E09C4/w/AB1C09CE/e/baseservices/exceptions/baseservices-exceptions/../simple/ParallelCrashTester/ParallelCrashTester.sh

Stack trace:

   at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/Microsoft.DotNet.XUnitAssert/src/BooleanAsserts.cs:line 146
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath, String testPathPrefix)
   at Program.<<Main>$>g__TestExecutor127|0_128(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)

@v-wenyuxu
Copy link
Author

Failed in: runtime-coreclr outerloop 20240721.3

Failed tests:

coreclr osx x64 Checked @ OSX.1200.Amd64.Open
    - baseservices/exceptions/simple/ParallelCrashTester/ParallelCrashTester.cmd

Error message:

 Process terminated. Parallel crash in main thread
   at System.Environment.FailFast(System.Runtime.CompilerServices.StackCrawlMarkHandle, System.String, System.Runtime.CompilerServices.ObjectHandleOnStack, System.String)
   at System.Environment.FailFast(System.Threading.StackCrawlMark ByRef, System.String, System.Exception, System.String)
   at System.Environment.FailFast(System.String)
   at ParallelCrash.Main(System.String[])
Process terminated. Parallel crash in thread 6!

   at System.Environment.FailFast(System.Runtime.CompilerServices.StackCrawlMarkHandle, System.String, System.Runtime.CompilerServices.ObjectHandleOnStack, System.String)
   at System.Environment.FailFast(System.Threading.StackCrawlMark ByRef, System.String, System.Exception, System.String)
   at System.Environment.FailFast(System.String)
   at ParallelCrash.CrashInParallel()
   at System.Threading.Thread.StartCallback()

cmdLine:/private/tmp/helix/working/AFE209A8/w/B882099F/e/baseservices/exceptions/baseservices-exceptions/../simple/ParallelCrashTester/ParallelCrashTester.sh Timed Out (timeout in milliseconds: 600000 from variable __TestTimeout, start: 7/21/2024 10:19:46 PM, end: 7/21/2024 10:29:46 PM)

Return code:      -100
Raw output file:      /tmp/helix/working/AFE209A8/w/B882099F/uploads/simple/ParallelCrashTester/output.txt
Raw output:
BEGIN EXECUTION
/tmp/helix/working/AFE209A8/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false -p System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true ParallelCrashTester.dll ''
Running ParallelCrash test(1)
Running ParallelCrash test(2)

cmdLine:/private/tmp/helix/working/AFE209A8/w/B882099F/e/baseservices/exceptions/baseservices-exceptions/../simple/ParallelCrashTester/ParallelCrashTester.sh Timed Out (timeout in milliseconds: 600000 from variable __TestTimeout, start: 7/21/2024 10:19:46 PM, end: 7/21/2024 10:29:46 PM)
Printing stacktrace from '/cores/crashdump_27666.dmp.crashreport.json'
Invoking llvm-symbolizer --pretty-print
Errors while running llvm-symbolizer --pretty-print
System.ComponentModel.Win32Exception (2): An error occurred trying to start process 'llvm-symbolizer' with working directory '/private/tmp/helix/working/AFE209A8/w/B882099F/e/baseservices/exceptions/baseservices-exceptions'. No such file or directory
   at System.Diagnostics.Process.ForkAndExecProcess(ProcessStartInfo startInfo, String resolvedFilename, String[] argv, String[] envp, String cwd, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, Int32& stdinFd, Int32& stdoutFd, Int32& stderrFd, Boolean usesTerminal, Boolean throwOnNoExec) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.Unix.cs:line 498
   at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.Unix.cs:line 456
   at System.Diagnostics.Process.Start() in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1284
   at CoreclrTestLib.CoreclrTestWrapperLib.TryPrintStackTraceFromCrashReport(String crashReportJsonFile, TextWriter outputWriter)
Printing stacktrace from '/cores/crashdump_28005.dmp.crashreport.json'
Invoking llvm-symbolizer --pretty-print
Errors while running llvm-symbolizer --pretty-print
System.ComponentModel.Win32Exception (2): An error occurred trying to start process 'llvm-symbolizer' with working directory '/private/tmp/helix/working/AFE209A8/w/B882099F/e/baseservices/exceptions/baseservices-exceptions'. No such file or directory
   at System.Diagnostics.Process.ForkAndExecProcess(ProcessStartInfo startInfo, String resolvedFilename, String[] argv, String[] envp, String cwd, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, Int32& stdinFd, Int32& stdoutFd, Int32& stderrFd, Boolean usesTerminal, Boolean throwOnNoExec) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.Unix.cs:line 498
   at System.Di

Stack trace:

   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath, String testPathPrefix)
   at Program.<<Main>$>g__TestExecutor131|0_132(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)

@JulieLeeMSFT JulieLeeMSFT added the Known Build Error Use this to report build issues in the .NET Helix tab label Jul 22, 2024
@JulieLeeMSFT JulieLeeMSFT modified the milestones: Future, 9.0.0 Jul 22, 2024
@mangod9 mangod9 self-assigned this Jul 22, 2024
@mangod9
Copy link
Member

mangod9 commented Jul 23, 2024

this looks similar to #105229.

@jkotas jkotas removed the Known Build Error Use this to report build issues in the .NET Helix tab label Jul 26, 2024
@mangod9 mangod9 modified the milestones: 9.0.0, Future Aug 28, 2024
@v-wenyuxu
Copy link
Author

Failed in: runtime-coreclr outerloop 20241016.7

Failed tests:

coreclr osx x64 Checked no_tiered_compilation @ OSX.1200.Amd64.Open
    - baseservices/exceptions/simple/ParallelCrashTester/ParallelCrashTester.cmd

Error message:

 Process terminated. Parallel crash in main thread
   at System.Environment.FailFast(System.Runtime.CompilerServices.StackCrawlMarkHandle, System.String, System.Runtime.CompilerServices.ObjectHandleOnStack, System.String)
   at System.Environment.FailFast(System.Threading.StackCrawlMark ByRef, System.String, System.Exception, System.String)
   at System.Environment.FailFast(System.String)
   at ParallelCrash.Main(System.String[])
Process terminated. Parallel crash in thread 10!

   at System.Environment.FailFast(System.Runtime.CompilerServices.StackCrawlMarkHandle, System.String, System.Runtime.CompilerServices.ObjectHandleOnStack, System.String)
   at System.Environment.FailFast(System.Threading.StackCrawlMark ByRef, System.String, System.Exception, System.String)
   at System.Environment.FailFast(System.String)
   at ParallelCrash.CrashInParallel()
   at System.Threading.Thread.StartCallback()

cmdLine:/private/tmp/helix/working/9A3708F3/w/A0C30901/e/baseservices/exceptions/baseservices-exceptions/../simple/ParallelCrashTester/ParallelCrashTester.sh Timed Out (timeout in milliseconds: 600000 from variable __TestTimeout, start: 10/16/2024 10:28:21 PM, end: 10/16/2024 10:38:21 PM)

Return code:      -100
Raw output file:      /tmp/helix/working/9A3708F3/w/A0C30901/uploads/simple/ParallelCrashTester/output.txt
Raw output:
BEGIN EXECUTION
/tmp/helix/working/9A3708F3/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false -p System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true ParallelCrashTester.dll ''
Running ParallelCrash test(1)
Running ParallelCrash test(2)

cmdLine:/private/tmp/helix/working/9A3708F3/w/A0C30901/e/baseservices/exceptions/baseservices-exceptions/../simple/ParallelCrashTester/ParallelCrashTester.sh Timed Out (timeout in milliseconds: 600000 from variable __TestTimeout, start: 10/16/2024 10:28:21 PM, end: 10/16/2024 10:38:21 PM)
Printing stacktrace from '/cores/crashdump_20636.dmp.crashreport.json'
Invoking llvm-symbolizer --pretty-print
Errors while running llvm-symbolizer --pretty-print
System.ComponentModel.Win32Exception (2): An error occurred trying to start process 'llvm-symbolizer' with working directory '/private/tmp/helix/working/9A3708F3/w/A0C30901/e/baseservices/exceptions/baseservices-exceptions'. No such file or directory
   at System.Diagnostics.Process.ForkAndExecProcess(ProcessStartInfo startInfo, String resolvedFilename, String[] argv, String[] envp, String cwd, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, Int32& stdinFd, Int32& stdoutFd, Int32& stderrFd, Boolean usesTerminal, Boolean throwOnNoExec) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.Unix.cs:line 496
   at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.Unix.cs:line 456
   at CoreclrTestLib.CoreclrTestWrapperLib.TryPrintStackTraceFromCrashReport(String crashReportJsonFile, TextWriter outputWriter)
Printing stacktrace from '/cores/crashdump_20939.dmp.crashreport.json'
Invoking llvm-symbolizer --pretty-print
Errors while running llvm-symbolizer --pretty-print
System.ComponentModel.Win32Exception (2): An error occurred trying to start process 'llvm-symbolizer' with working directory '/private/tmp/helix/working/9A3708F3/w/A0C30901/e/baseservices/exceptions/baseservices-exceptions'. No such file or directory
   at System.Diagnostics.Process.ForkAndExecProcess(ProcessStartInfo startInfo, String resolvedFilename, String[] argv, String[] envp, String cwd, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, Int32& stdinFd, Int32& stdoutFd, Int32& stderrFd, Boolean usesTerminal, Boolean throwOnNoExec) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.Unix.cs:line 496
   at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/P

Stack trace:

   at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/Microsoft.DotNet.XUnitAssert/src/BooleanAsserts.cs:line 146
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath, String testPathPrefix)
   at Program.<<Main>$>g__TestExecutor131|0_132(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mangod9 mangod9

Labels
arch-x64 area-VM-coreclr disabled-test The test is disabled in source code against the issue GCStress os-mac-os-x macOS aka OSX
Projects
None yet
Milestone
Future
Development

No branches or pull requests
8 participants
@jkotas @jeffschwMSFT @BruceForstall @kunalspathak @mangod9 @JulieLeeMSFT @kasperk81 @v-wenyuxu