Call rclcpp::shutdown in test_node for clean shutdown on Windows

[eboasson]

The graph_listener thread is started in the background in some of the tests and this thread is killed by Windows prior to executing global destructors if it is still running when leaving main(). This then corrupts state because the RMW layer is blocking in a waitset and causes Cyclone to hang trying to destroy the waitset.

This fixes the issue mentioned in ros2/rmw#293 (comment) (for me locally anyway!)

[clalancette]

Looks good to me. Since this problem seems to happen every time, a simple CI up to rclcpp should show whether it is fixed. Running that next.

[clalancette]

CI:

• Linux
• Linux-aarch64
• macOS
• Windows

[clalancette]

Looking good! I'm going to go ahead and merge this, thanks again @eboasson .

[wjwwood]

I don't know about this pull request, it sounds like we need to ensure this doesn't happen without an explicit shutdown.

I don't see how the thread can be terminated before globals can be destructed...

[wjwwood]

I don't know about this pull request, it sounds like we need to ensure this doesn't happen without an explicit shutdown.

Specifically, we allow and even encourage our users to not necessarily explicitly shutdown in their code and in our examples. So this would not work on Windows. It's not a very satisfying solution (requiring an explicit shutdown).

[ivanpauno]

Specifically, we allow and even encourage our users to not necessarily explicitly shutdown in their code and in our examples. So this would not work on Windows. It's not a very satisfying solution (requiring an explicit shutdown).

Yeah, I'm not sure if we will have to re-think that or not ....
Things hanging because of a forgotten rclcpp::shutdown() is definitely not "nice", and the only way I know to avoid that problem is to completely forbid objects with static lifetime that don't have "trivial" destructors (but this is a big restriction ...).

[wjwwood]

Things hanging because of a forgotten rclcpp::shutdown() is definitely not "nice",

That's my whole point. It's not forgotten, it's explicitly left out and that should work as far as I understand. I don't think an explicit shutdown should be required, unless someone can explain why that is the case from an architectural standpoint.

It may be the case that static destruction order is the problem, but that wasn't clear here. And if it is, then it's not rclcpp's fault. From my perspective all libraries in a stack for an application should make globals optional, so that this can be avoided. However, having globals be used optionally is ok, like we do in rclcpp. In the somewhat common case that the user is using rclcpp directly and utilizes the convenience singleton init/shutdown it should be fine since the user does not have static objects that rely on shutdown.

If the use of static objects in rcl/rmw or in cyclone are contributing to this problem we should aim to make their use optional, so they can coexist with rclcpp, anything else is (in my opinion) a bug of those layers, not rclcpp.

[ivanpauno]

That's my whole point. It's not forgotten, it's explicitly left out and that should work as far as I understand. I don't think an explicit shutdown should be required, unless someone can explain why that is the case from an architectural standpoint.

Yes, I agree that rclcpp::shutdown shouldn't be necessary.

And if it is, then it's not rclcpp's fault. From my perspective all libraries in a stack for an application should make globals optional, so that this can be avoided. However, having globals be used optionally is ok, like we do in rclcpp

I slightly disagree.
You can almost always avoid objects with non-trivial destructors and static lifetimes (at the expense of some extra verbosity), as really their initialization/destruction order is difficult to predict (between different platforms, and sometimes between different builds in the same platform).

e.g. the google style guide recommends to only use object that can be constexpr constructed and with trivial destructors.
If you want something that doesn't have a trivial destructor, the recommendation is to leak the object.
(they also support not constexpr constructible objects if initialization order doesn't matter or if it's an "static local" variable, but initialization isn't usually the big issue, destruction is)
e.g.2 rust only supports statics that can be constructed at compile time and drop is never called on those objects, or they also support lazily initialized statics which will never be dropped (e.g. once_cell or lazy_static).

Of course, this might be an opinionated decision of an style guide and a programming language, but it seems to me that's based in difficult to predict behavior of static constructors/destructors.

If the use of static objects in rcl/rmw or in cyclone are contributing to this problem we should aim to make their use optional, so they can coexist with rclcpp, anything else is (in my opinion) a bug of those layers, not rclcpp.

That's a good question, @eboasson can this problem by avoided in rmw_cyclonedds or cyclonedds?

[wjwwood]

If I would amend the style guide slightly it would be to say, you can use static objects but only if they are unconnected to one another. The problem comes in when a non-trivial destructor of a static global uses another static global, especially one from another shared library. If the static globals are completely decoupled, then there's no issue if their destruction order is unknown. So in the case of our tests and many of our examples, rclcpp is (or should be) the only thing with static globals and the user (no any other library on top of rclcpp but below user code) is not using static globals that are some how connected to rclcpp, and therefore there should be no problem. rclcpp, via rcl, rmw, etc... should opt not to use static globals from underlying libraries if possible to keep this isolation of static globals. For example, we shifted to using locally created and owned logging objects from spdlog rather than a global static singleton (one of the reasons to stop using log4cxx which did not allow this).

If the user (or some library on top of rclcpp) wants to have things with a static lifetime that do use rclcpp (specifically the context or signal handler), then they would need to use the non-global version of init/shutdown. Since rclcpp provides this as an option and does not require the use of globals, I think it's ok.

I understand why the guided probably prohibits their use, because it's easier to say never do it then to do it safely (ensuring static globals do not interact may be non-trivial), but I don't think it's impossible.

If we wanted to, we could do the same as rust and just not call the destructor for these classes, but that might require shifts in other places.

---

That all being said, the explanation for why this bug was happening was something like "memory corruption" which means there is a bug somewhere else that's affecting memory improperly.

it is then still blocked on a waitset by the time Windows decides that thread really shouldn't exist. Indiscriminately killing that thread from a global destructor then corrupts internal state of Cyclone and leaves it blocking on trying to delete the entity.

ros2/rmw#293 (comment)

So maybe the right fix is to conditionally "kill" the thread, if that is possible. By this I assume @eboasson meant join the thread or attempt to exit it? I'm not really sure. But presumably it is possible to know from Windows if a thread has been terminated and then we could attempt to join it or whatever only conditionally.

---

I'm not advocating for reverting this (I know we're trying to make progress on switching the default), but I don't think this solves the issue.

My problem with this pull request is that it seems to remove the thing that triggers the bug but doesn't fix it...

If the answer is, well we can't fix that bug, then I think we must change our architecture to require or ensure explicit shutdown, or change it so that shutdown isn't required in a different way (maybe leaking things). At the very least our documentation should change if we cannot fix this better.

[ivanpauno]

My problem with this pull request is that it seems to remove the thing that triggers the bug but doesn't fix it...

If the answer is, well we can't fix that bug, then I think we must change our architecture to require or ensure explicit shutdown, or change it so that shutdown isn't required in a different way (maybe leaking things). At the very least our documentation should change if we cannot fix this better.

👍

[eboasson]

Firstly, @wjwwood I totally agree with your statement that requiring an explicit shutdown should not be required, and indeed on Linux and macOS it all works just fine. However, on Windows it doesn't because the threads are "disappeared". As far as I have been able to find out all threads other than the main thread quite simply

are killed prior to the execution of the static destructors. It won’t give any notification, they’re just gone, and if they happen to be in a critical section or if there happens to be any state associated with them, tough luck.

test_node_cdb.log is the result of my efforts to capture test_node with Cyclone but without the explicit call to rclcpp::shutdown in place. This more-or-less traces the sequence of events, but it is an imperfect trace because I only set a limited number of breakpoints in Windows (ntdll!*Process* and ntdll!*Thread*) in addition to rclcpp::Context::shutdown. Each time a breakpoint is hit, it prints the list of threads and their stack traces.

The last point where all threads still exist (and in particular, the main thread on line 868 and the graph_listener on line 913)

    858 .  0  Id: 1754.4b4 Suspend: 1 Teb: 000000cf`85666000 Unfrozen
    859    1  Id: 1754.1804 Suspend: 0 Teb: 000000cf`85668000 Unfrozen
    860    2  Id: 1754.1a0 Suspend: 0 Teb: 000000cf`8566a000 Unfrozen
    861    3  Id: 1754.bdc Suspend: 0 Teb: 000000cf`8566c000 Unfrozen
    862    4  Id: 1754.1410 Suspend: 0 Teb: 000000cf`8566e000 Unfrozen
    863   11  Id: 1754.178 Suspend: 0 Teb: 000000cf`8567c000 Unfrozen
    864   12  Id: 1754.1dc0 Suspend: 0 Teb: 000000cf`85714000 Unfrozen
    866 .  0  Id: 1754.4b4 Suspend: 1 Teb: 000000cf`85666000 Unfrozen
    865
    867 Child-SP          RetAddr           Call Site
    868 000000cf`858ffa88 00007ffe`f8773e41 ntdll!RtlUnlockProcessHeapOnProcessTerminate
    869 000000cf`858ffa90 00007ffe`f6d9e0ab ntdll!RtlExitUserProcess+0x61
    870 000000cf`858ffac0 00007ffe`c1d3467a KERNEL32!ExitProcessImplementation+0xb
    871 000000cf`858ffaf0 00007ffe`c1d34629 ucrtbased!exit_or_terminate_process+0x3a [minkernel\crts\ucrt\src\appcrt\startup\exit.cpp @ 144]
    872 000000cf`858ffb20 00007ffe`c1d349a6 ucrtbased!common_exit+0x79 [minkernel\crts\ucrt\src\appcrt\startup\exit.cpp @ 282]
    873 000000cf`858ffb80 00007ff7`06695257 ucrtbased!exit+0x16 [minkernel\crts\ucrt\src\appcrt\startup\exit.cpp @ 294]
    874 000000cf`858ffbb0 00007ff7`066950fe test_node!__scrt_common_main_seh+0x147 [d:\agent\_work\63\s\src\vctools\crt\vcstartup\src\startup\e
    874 xe_common.inl @ 297]
    875 000000cf`858ffc20 00007ff7`066953e9 test_node!__scrt_common_main+0xe [d:\agent\_work\63\s\src\vctools\crt\vcstartup\src\startup\exe_com
    875 mon.inl @ 331]
    876 000000cf`858ffc50 00007ffe`f6d97034 test_node!mainCRTStartup+0x9 [d:\agent\_work\63\s\src\vctools\crt\vcstartup\src\startup\exe_main.cp
    876 p @ 17]
    877 000000cf`858ffc80 00007ffe`f875d0d1 KERNEL32!BaseThreadInitThunk+0x14
    878 000000cf`858ffcb0 00000000`00000000 ntdll!RtlUserThreadStart+0x21
…
    913   11  Id: 1754.178 Suspend: 0 Teb: 000000cf`8567c000 Unfrozen
    914 Child-SP          RetAddr           Call Site
    915 000000cf`863f7eb8 00007ffe`f8775281 ntdll!NtWaitForAlertByThreadId+0x14
    916 000000cf`863f7ec0 00007ffe`f63eba89 ntdll!RtlSleepConditionVariableSRW+0x131
    917 000000cf`863f7f40 00007ffe`ba9bccd7 KERNELBASE!SleepConditionVariableSRW+0x29
    918 000000cf`863f7f80 00007ffe`ba9bce80 ddsc!ddsrt_cond_wait+0x87 [C:\dev\ros2_ws\src\eclipse-cyclonedds\cyclonedds\src\ddsrt\src\sync\wind    918 ows\sync.c @ 64]
    919 000000cf`863f7fb0 00007ffe`ba9bcdd6 ddsc!ddsrt_cond_waitfor+0x90 [C:\dev\ros2_ws\src\eclipse-cyclonedds\cyclonedds\src\ddsrt\src\sync\w    919 indows\sync.c @ 104]
    920 000000cf`863f8000 00007ffe`ba98eaaf ddsc!ddsrt_cond_waituntil+0xe6 [C:\dev\ros2_ws\src\eclipse-cyclonedds\cyclonedds\src\ddsrt\src\sync    920 \windows\sync.c @ 88]
    921 000000cf`863f8050 00007ffe`ba98e702 ddsc!dds_waitset_wait_impl+0x25f [C:\dev\ros2_ws\src\eclipse-cyclonedds\cyclonedds\src\core\ddsc\sr    921 c\dds_waitset.c @ 79]
    922 000000cf`863f8100 00007ffe`c1f998a5 ddsc!dds_waitset_wait+0xa2 [C:\dev\ros2_ws\src\eclipse-cyclonedds\cyclonedds\src\core\ddsc\src\dds_    922 waitset.c @ 376]
    923 000000cf`863f8150 00007ffe`dd936466 rmw_cyclonedds_cpp!rmw_wait+0xa45 [C:\dev\ros2_ws\src\ros2\rmw_cyclonedds\rmw_cyclonedds_cpp\src\rm    923 w_node.cpp @ 3383]
    924 000000cf`863f8460 00007ffe`cc19744f rmw_implementation!rmw_wait+0xa6 [C:\dev\ros2_ws\src\ros2\rmw_implementation\rmw_implementation\src    924 \functions.cpp @ 491]
    925 000000cf`863f84c0 00007ffe`c2177d25 rcl!rcl_wait+0x95f [C:\dev\ros2_ws\src\ros2\rcl\rcl\src\rcl\wait.c @ 609]
    926 000000cf`863fd350 00007ffe`c21778c2 rclcpp!rclcpp::graph_listener::GraphListener::run_loop+0x425 [C:\dev\ros2_ws\src\ros2\rclcpp\rclcpp    926 \src\rclcpp\graph_listener.cpp @ 187]
    927 000000cf`863fd5b0 00007ffe`c217a4de rclcpp!rclcpp::graph_listener::GraphListener::run+0x52 [C:\dev\ros2_ws\src\ros2\rclcpp\rclcpp\src\r    927 clcpp\graph_listener.cpp @ 113]
    928 000000cf`863ff6d0 00007ffe`c2179ca5 rclcpp!std::invoke<void (__cdecl rclcpp::graph_listener::GraphListener::*)(void),rclcpp::graph_list    928 ener::GraphListener *>+0x3e [C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.28.29333\include\type_trait    928 s @ 1614]
    929 000000cf`863ff700 00007ffe`c1d3542c rclcpp!std::thread::_Invoke<std::tuple<void (__cdecl rclcpp::graph_listener::GraphListener::*)(void    929 ),rclcpp::graph_listener::GraphListener *>,0,1>+0x75 [C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.28    929 .29333\include\thread @ 44]
    930 000000cf`863ff760 00007ffe`f6d97034 ucrtbased!thread_start<unsigned int (__cdecl*)(void *),1>+0x9c [minkernel\crts\ucrt\src\appcrt\star    930 tup\thread.cpp @ 97]
    931 000000cf`863ff7c0 00007ffe`f875d0d1 KERNEL32!BaseThreadInitThunk+0x14
    932 000000cf`863ff7f0 00000000`00000000 ntdll!RtlUserThreadStart+0x21

At the next breakpoint there are no other threads left. I don’t know what function got called; I wouldn’t be surprised if it all happens within ntdll!RtlExitUserProcess. Anyway, at the point all you have is:

    946 .  0  Id: 1754.4b4 Suspend: 1 Teb: 000000cf`85666000 Unfrozen
    947 Child-SP          RetAddr           Call Site
    948 000000cf`858ffa88 00007ffe`f8773e88 ntdll!RtlReportSilentProcessExit
    949 000000cf`858ffa90 00007ffe`f6d9e0ab ntdll!RtlExitUserProcess+0xa8
    950 000000cf`858ffac0 00007ffe`c1d3467a KERNEL32!ExitProcessImplementation+0xb
    951 000000cf`858ffaf0 00007ffe`c1d34629 ucrtbased!exit_or_terminate_process+0x3a [minkernel\crts\ucrt\src\appcrt\startup\exit.cpp @ 144]
    952 000000cf`858ffb20 00007ffe`c1d349a6 ucrtbased!common_exit+0x79 [minkernel\crts\ucrt\src\appcrt\startup\exit.cpp @ 282]
    953 000000cf`858ffb80 00007ff7`06695257 ucrtbased!exit+0x16 [minkernel\crts\ucrt\src\appcrt\startup\exit.cpp @ 294]
    954 000000cf`858ffbb0 00007ff7`066950fe test_node!__scrt_common_main_seh+0x147 [d:\agent\_work\63\s\src\vctools\crt\vcstartup\src\startup\e    954 xe_common.inl @ 297]
    955 000000cf`858ffc20 00007ff7`066953e9 test_node!__scrt_common_main+0xe [d:\agent\_work\63\s\src\vctools\crt\vcstartup\src\startup\exe_com    955 mon.inl @ 331]
    956 000000cf`858ffc50 00007ffe`f6d97034 test_node!mainCRTStartup+0x9 [d:\agent\_work\63\s\src\vctools\crt\vcstartup\src\startup\exe_main.cp    956 p @ 17]
    957 000000cf`858ffc80 00007ffe`f875d0d1 KERNEL32!BaseThreadInitThunk+0x14
    958 000000cf`858ffcb0 00000000`00000000 ntdll!RtlUserThreadStart+0x21

A while later, the destructor finally runs:

   1046 .  0  Id: 1754.4b4 Suspend: 1 Teb: 000000cf`85666000 Unfrozen
   1047 Child-SP          RetAddr           Call Site
   1048 000000cf`858fd2e8 00007ffe`c211bb13 rclcpp!rclcpp::Context::shutdown [C:\dev\ros2_ws\src\ros2\rclcpp\rclcpp\src\rclcpp\context.cpp @ 30
   1048 2]
   1049 000000cf`858fd2f0 00007ffe`c212f918 rclcpp!rclcpp::Context::~Context+0x93 [C:\dev\ros2_ws\src\ros2\rclcpp\rclcpp\src\rclcpp\context.cpp
   1049  @ 158]
   1050 000000cf`858ff510 00007ffe`c212f9cc rclcpp!rclcpp::contexts::DefaultContext::~DefaultContext+0x28
   1051 000000cf`858ff540 00007ffe`c212f3af rclcpp!rclcpp::contexts::DefaultContext::`scalar deleting destructor'+0x2c
   1052 000000cf`858ff570 00007ffe`c212fb0f rclcpp!std::_Destroy_in_place<rclcpp::contexts::DefaultContext>+0x2f [C:\Program Files (x86)\Micros
   1052 oft Visual Studio\2019\Community\VC\Tools\MSVC\14.28.29333\include\xmemory @ 279]
   1053 000000cf`858ff5a0 00007ffe`c21046b7 rclcpp!std::_Ref_count_obj2<rclcpp::contexts::DefaultContext>::_Destroy+0x2f [C:\Program Files (x86
   1053 )\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.28.29333\include\memory @ 1582]
   1054 000000cf`858ff5d0 00007ffe`c212fa48 rclcpp!std::_Ref_count_base::_Decref+0x47 [C:\Program Files (x86)\Microsoft Visual Studio\2019\Comm
   1054 unity\VC\Tools\MSVC\14.28.29333\include\memory @ 644]
   1055 000000cf`858ff600 00007ffe`c212f8d8 rclcpp!std::_Ptr_base<rclcpp::contexts::DefaultContext>::_Decref+0x38 [C:\Program Files (x86)\Micro
   1055 soft Visual Studio\2019\Community\VC\Tools\MSVC\14.28.29333\include\memory @ 878]
   1056 000000cf`858ff630 00007ffe`c23677c1 rclcpp!std::shared_ptr<rclcpp::contexts::DefaultContext>::~shared_ptr<rclcpp::contexts::DefaultCont
   1056 ext>+0x28 [C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.28.29333\include\memory @ 1170]
   1057 000000cf`858ff660 00007ffe`c1d35107 rclcpp!`rclcpp::contexts::get_global_default_context'::`2'::`dynamic atexit destructor for 'default
   1057 _context''+0x21
   1058 000000cf`858ff690 00007ffe`c1d34b15 ucrtbased!<lambda_d121dba8a4adeaf3a9819e48611155df>::operator()+0x127 [minkernel\crts\ucrt\src\appc
   1058 rt\startup\onexit.cpp @ 206]
   1059 000000cf`858ff720 00007ffe`c1d34c4a ucrtbased!__crt_seh_guarded_call<int>::operator()<<lambda_6a47f4c8fd0152770a780fc1d70204eb>,<lambda
   1059 _d121dba8a4adeaf3a9819e48611155df> &,<lambda_6aaa2265f5b6a89667e7d7630012e97a> >+0x35 [vccrt\vcruntime\inc\internal_shared.h @ 204]
   1060 000000cf`858ff760 00007ffe`c1d352b1 ucrtbased!__acrt_lock_and_call<<lambda_d121dba8a4adeaf3a9819e48611155df> >+0x4a [minkernel\crts\ucr
   1060 t\inc\corecrt_internal.h @ 975]
   1061 000000cf`858ff7b0 00007ffe`c23283e9 ucrtbased!_execute_onexit_table+0x31 [minkernel\crts\ucrt\src\appcrt\startup\onexit.cpp @ 231]
   1062 000000cf`858ff7f0 00007ffe`c232901c rclcpp!__scrt_dllmain_uninitialize_c+0x19 [d:\agent\_work\63\s\src\vctools\crt\vcstartup\src\utilit   1062 y\utility.cpp @ 399]
   1063 000000cf`858ff820 00007ffe`c2328df7 rclcpp!dllmain_crt_process_detach+0x4c [d:\agent\_work\63\s\src\vctools\crt\vcstartup\src\startup\d   1063 ll_dllmain.cpp @ 182]
   1064 000000cf`858ff860 00007ffe`c232918e rclcpp!dllmain_crt_dispatch+0x67 [d:\agent\_work\63\s\src\vctools\crt\vcstartup\src\startup\dll_dll   1064 main.cpp @ 220]
   1065 000000cf`858ff8a0 00007ffe`c2329301 rclcpp!dllmain_dispatch+0xfe [d:\agent\_work\63\s\src\vctools\crt\vcstartup\src\startup\dll_dllmain   1065 .cpp @ 293]
   1066 000000cf`858ff8f0 00007ffe`f8747cad rclcpp!_DllMainCRTStartup+0x31 [d:\agent\_work\63\s\src\vctools\crt\vcstartup\src\startup\dll_dllma   1066 in.cpp @ 335]
   1067 000000cf`858ff920 00007ffe`f8773ff1 ntdll!LdrpCallInitRoutine+0x61
   1068 000000cf`858ff990 00007ffe`f8773e8d ntdll!LdrShutdownProcess+0x141
   1069 000000cf`858ffa90 00007ffe`f6d9e0ab ntdll!RtlExitUserProcess+0xad
   1070 000000cf`858ffac0 00007ffe`c1d3467a KERNEL32!ExitProcessImplementation+0xb
   1071 000000cf`858ffaf0 00007ffe`c1d34629 ucrtbased!exit_or_terminate_process+0x3a [minkernel\crts\ucrt\src\appcrt\startup\exit.cpp @ 144]
   1072 000000cf`858ffb20 00007ffe`c1d349a6 ucrtbased!common_exit+0x79 [minkernel\crts\ucrt\src\appcrt\startup\exit.cpp @ 282]
   1073 000000cf`858ffb80 00007ff7`06695257 ucrtbased!exit+0x16 [minkernel\crts\ucrt\src\appcrt\startup\exit.cpp @ 294]
   1074 000000cf`858ffbb0 00007ff7`066950fe test_node!__scrt_common_main_seh+0x147 [d:\agent\_work\63\s\src\vctools\crt\vcstartup\src\startup\e   1074 xe_common.inl @ 297]
   1075 000000cf`858ffc20 00007ff7`066953e9 test_node!__scrt_common_main+0xe [d:\agent\_work\63\s\src\vctools\crt\vcstartup\src\startup\exe_com   1075 mon.inl @ 331]
   1076 000000cf`858ffc50 00007ffe`f6d97034 test_node!mainCRTStartup+0x9 [d:\agent\_work\63\s\src\vctools\crt\vcstartup\src\startup\exe_main.cp   1076 p @ 17]
   1077 000000cf`858ffc80 00007ffe`f875d0d1 KERNEL32!BaseThreadInitThunk+0x14
   1078 000000cf`858ffcb0 00000000`00000000 ntdll!RtlUserThreadStart+0x21

It is always a tricky business guessing where to set breakpoints, but this at least substantiates my claim that Windows kills the threads, pretty much like kill -9 does with processes on Unix. And just like a kill -9 usually leaves processes using shared memory in a corrupt state, this leaves the process in an inconsistent state. If you look at the graph listener thread before it gets killed, it is blocked on a waitset (the Cyclone RMW layer maps waitsets to DDS waitsets). Cyclone properly protects the objects against race conditions such as trying to delete an entity/object while it is still in use, and hence it records the fact that there is currently a thread blocked in a call while holding a pointer to this waitset. When the thread is killed by Windows, that record remains.

The destructor then tries to trigger a guard condition to wake up the graph_listener, which ultimately ends up as a broadcast on a condition variable:

   1084 .  0  Id: 1754.4b4 Suspend: 1 Teb: 000000cf`85666000 Unfrozen
   1085 Child-SP          RetAddr           Call Site
   1086 000000cf`858fa758 00007ffe`f875d642 ntdll!NtAlertThreadByThreadId
   1087 000000cf`858fa760 00007ffe`ba9bcfe0 ntdll!RtlWakeAllConditionVariable+0x62
   1088 000000cf`858fa790 00007ffe`ba98ef8c ddsc!ddsrt_cond_broadcast+0x50 [C:\dev\ros2_ws\src\eclipse-cyclonedds\cyclonedds\src\ddsrt\src\sync
   1088 \windows\sync.c @ 128]
   1089 000000cf`858fa7c0 00007ffe`ba984057 ddsc!dds_waitset_observer+0x19c [C:\dev\ros2_ws\src\eclipse-cyclonedds\cyclonedds\src\core\ddsc\src
   1089 \dds_waitset.c @ 237]
   1090 000000cf`858fa840 00007ffe`ba9836a5 ddsc!dds_entity_observers_signal+0x67 [C:\dev\ros2_ws\src\eclipse-cyclonedds\cyclonedds\src\core\dd
   1090 sc\src\dds_entity.c @ 1393]
   1091 000000cf`858fa880 00007ffe`ba99007c ddsc!dds_entity_trigger_set+0xb5 [C:\dev\ros2_ws\src\eclipse-cyclonedds\cyclonedds\src\core\ddsc\sr
   1091 c\dds_entity.c @ 1442]
   1092 000000cf`858fa8c0 00007ffe`c1f9889a ddsc!dds_set_guardcondition+0x5c [C:\dev\ros2_ws\src\eclipse-cyclonedds\cyclonedds\src\core\ddsc\sr
   1092 c\dds_guardcond.c @ 87]
   1093 000000cf`858fa910 00007ffe`dd9362c1 rmw_cyclonedds_cpp!rmw_trigger_guard_condition+0xba [C:\dev\ros2_ws\src\ros2\rmw_cyclonedds\rmw_cyc   1093 lonedds_cpp\src\rmw_node.cpp @ 3092]
   1094 000000cf`858fa950 00007ffe`cc17559a rmw_implementation!rmw_trigger_guard_condition+0x61 [C:\dev\ros2_ws\src\ros2\rmw_implementation\rmw   1094 _implementation\src\functions.cpp @ 476]
   1095 000000cf`858fa990 00007ffe`c21782b8 rcl!rcl_trigger_guard_condition+0x5a [C:\dev\ros2_ws\src\ros2\rcl\rcl\src\rcl\guard_condition.c @ 1   1095 60]
   1096 000000cf`858fadd0 00007ffe`c21781b7 rclcpp!rclcpp::graph_listener::interrupt_+0x38 [C:\dev\ros2_ws\src\ros2\rclcpp\rclcpp\src\rclcpp\gr   1096 aph_listener.cpp @ 216]
   1097 000000cf`858fae40 00007ffe`c21777ee rclcpp!rclcpp::graph_listener::GraphListener::__shutdown+0x97 [C:\dev\ros2_ws\src\ros2\rclcpp\rclcp   1097 p\src\rclcpp\graph_listener.cpp @ 352]
   1098 000000cf`858faec0 00007ffe`c217bdb7 rclcpp!rclcpp::graph_listener::GraphListener::shutdown+0x4e [C:\dev\ros2_ws\src\ros2\rclcpp\rclcpp\   1098 src\rclcpp\graph_listener.cpp @ 374]
   1099 000000cf`858fd0e0 00007ffe`c217a488 rclcpp!<lambda_ce8062be14c7855075cc462eb8ce2d37>::operator()+0x67 [C:\dev\ros2_ws\src\ros2\rclcpp\r   1099 clcpp\src\rclcpp\graph_listener.cpp @ 100]
   1100 000000cf`858fd140 00007ffe`c21794d8 rclcpp!std::invoke<<lambda_ce8062be14c7855075cc462eb8ce2d37> &>+0x28 [C:\Program Files (x86)\Micros   1100 oft Visual Studio\2019\Community\VC\Tools\MSVC\14.28.29333\include\type_traits @ 1585]
   1101 000000cf`858fd170 00007ffe`c217c85f rclcpp!std::_Invoker_ret<void,1>::_Call<<lambda_ce8062be14c7855075cc462eb8ce2d37> &>+0x28 [C:\Progr   1101 am Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.28.29333\include\functional @ 745]
   1102 000000cf`858fd1a0 00007ffe`c211aca3 rclcpp!std::_Func_impl_no_alloc<<lambda_ce8062be14c7855075cc462eb8ce2d37>,void>::_Do_call+0x2f [C:\   1102 Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.28.29333\include\functional @ 948]
   1103 000000cf`858fd1d0 00007ffe`c211c845 rclcpp!std::_Func_class<void>::operator()+0x53 [C:\Program Files (x86)\Microsoft Visual Studio\2019   1103 \Community\VC\Tools\MSVC\14.28.29333\include\functional @ 996]
   1104 000000cf`858fd210 00007ffe`c211bb13 rclcpp!rclcpp::Context::shutdown+0x165 [C:\dev\ros2_ws\src\ros2\rclcpp\rclcpp\src\rclcpp\context.cp   1104 p @ 320]

It subsequently joins the thread (which succeeds immediately because the thread objects are signalled by the weird thread killing spree of Windows):

   1140 .  0  Id: 1754.4b4 Suspend: 1 Teb: 000000cf`85666000 Unfrozen
   1141 Child-SP          RetAddr           Call Site
   1142 000000cf`858fad28 00007ffe`f63f7a73 ntdll!NtQueryInformationThread
   1143 000000cf`858fad30 00007ffe`c429268c KERNELBASE!GetExitCodeThread+0x23
   1144 000000cf`858fada0 00007ffe`c2166ff3 MSVCP140D!_Thrd_join+0x3c [D:\a01\_work\6\s\src\vctools\crt\github\stl\src\cthread.cpp @ 57]
   1145 000000cf`858fade0 00007ffe`c21781cb rclcpp!std::thread::join+0x73 [C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Too
   1145 ls\MSVC\14.28.29333\include\thread @ 113]
   1146 000000cf`858fae40 00007ffe`c21777ee rclcpp!rclcpp::graph_listener::GraphListener::__shutdown+0xab [C:\dev\ros2_ws\src\ros2\rclcpp\rclcp
   1146 p\src\rclcpp\graph_listener.cpp @ 354]
   1147 000000cf`858faec0 00007ffe`c217bdb7 rclcpp!rclcpp::graph_listener::GraphListener::shutdown+0x4e [C:\dev\ros2_ws\src\ros2\rclcpp\rclcpp\
   1147 src\rclcpp\graph_listener.cpp @ 374]
   1148 000000cf`858fd0e0 00007ffe`c217a488 rclcpp!<lambda_ce8062be14c7855075cc462eb8ce2d37>::operator()+0x67 [C:\dev\ros2_ws\src\ros2\rclcpp\r
   1148 clcpp\src\rclcpp\graph_listener.cpp @ 100]
   1149 000000cf`858fd140 00007ffe`c21794d8 rclcpp!std::invoke<<lambda_ce8062be14c7855075cc462eb8ce2d37> &>+0x28 [C:\Program Files (x86)\Micros
   1149 oft Visual Studio\2019\Community\VC\Tools\MSVC\14.28.29333\include\type_traits @ 1585]
   1150 000000cf`858fd170 00007ffe`c217c85f rclcpp!std::_Invoker_ret<void,1>::_Call<<lambda_ce8062be14c7855075cc462eb8ce2d37> &>+0x28 [C:\Progr
   1150 am Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.28.29333\include\functional @ 745]
   1151 000000cf`858fd1a0 00007ffe`c211aca3 rclcpp!std::_Func_impl_no_alloc<<lambda_ce8062be14c7855075cc462eb8ce2d37>,void>::_Do_call+0x2f [C:\
   1151 Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.28.29333\include\functional @ 948]
   1152 000000cf`858fd1d0 00007ffe`c211c845 rclcpp!std::_Func_class<void>::operator()+0x53 [C:\Program Files (x86)\Microsoft Visual Studio\2019
   1152 \Community\VC\Tools\MSVC\14.28.29333\include\functional @ 996]
   1153 000000cf`858fd210 00007ffe`c211bb13 rclcpp!rclcpp::Context::shutdown+0x165 [C:\dev\ros2_ws\src\ros2\rclcpp\rclcpp\src\rclcpp\context.cp
   1153 p @ 320]

When it then tries to destroy the waitset, Cyclone notes that it is in use, interrupts the blocked operation and waits for the call to dds_waitset_wait made by the graph_listener to complete. But because the thread no longer exists (and only because it no longer exists), that never happens and the process hangs:

   1189 .  0  Id: 1754.4b4 Suspend: 1 Teb: 000000cf`85666000 Unfrozen
   1190 Child-SP          RetAddr           Call Site
   1191 000000cf`858fa548 00007ffe`f8775281 ntdll!NtWaitForAlertByThreadId
   1192 000000cf`858fa550 00007ffe`f63eba89 ntdll!RtlSleepConditionVariableSRW+0x131
   1193 000000cf`858fa5d0 00007ffe`ba9bccd7 KERNELBASE!SleepConditionVariableSRW+0x29
   1194 000000cf`858fa610 00007ffe`ba980620 ddsc!ddsrt_cond_wait+0x87 [C:\dev\ros2_ws\src\eclipse-cyclonedds\cyclonedds\src\ddsrt\src\sync\wind
   1194 ows\sync.c @ 64]
   1195 000000cf`858fa640 00007ffe`ba9842b8 ddsc!dds_handle_close_wait+0xc0 [C:\dev\ros2_ws\src\eclipse-cyclonedds\cyclonedds\src\core\ddsc\src
   1195 \dds_handles.c @ 502]
   1196 000000cf`858fa680 00007ffe`ba9839a1 ddsc!really_delete_pinned_closed_locked+0xf8 [C:\dev\ros2_ws\src\eclipse-cyclonedds\cyclonedds\src\
   1196 core\ddsc\src\dds_entity.c @ 450]
   1197 000000cf`858fa6e0 00007ffe`ba98417e ddsc!dds_delete_impl_pinned+0x71 [C:\dev\ros2_ws\src\eclipse-cyclonedds\cyclonedds\src\core\ddsc\sr
   1197 c\dds_entity.c @ 410]
   1198 000000cf`858fa710 00007ffe`ba981b37 ddsc!dds_delete_impl+0x7e [C:\dev\ros2_ws\src\eclipse-cyclonedds\cyclonedds\src\core\ddsc\src\dds_e
   1198 ntity.c @ 384]
   1199 000000cf`858fa760 00007ffe`c1f98d85 ddsc!dds_delete+0x27 [C:\dev\ros2_ws\src\eclipse-cyclonedds\cyclonedds\src\core\ddsc\src\dds_entity
   1199 .c @ 368]
   1200 000000cf`858fa790 00007ffe`dd9363b1 rmw_cyclonedds_cpp!rmw_destroy_wait_set+0x125 [C:\dev\ros2_ws\src\ros2\rmw_cyclonedds\rmw_cyclonedd
   1200 s_cpp\src\rmw_node.cpp @ 3164]
   1201 000000cf`858fa950 00007ffe`cc19483e rmw_implementation!rmw_destroy_wait_set+0x61 [C:\dev\ros2_ws\src\ros2\rmw_implementation\rmw_implem
   1201 entation\src\functions.cpp @ 486]
   1202 000000cf`858fa990 00007ffe`c21780c1 rcl!rcl_wait_set_fini+0x8e [C:\dev\ros2_ws\src\ros2\rcl\rcl\src\rcl\wait.c @ 195]
   1203 000000cf`858fadd0 00007ffe`c2178240 rclcpp!rclcpp::graph_listener::GraphListener::cleanup_wait_set+0x41 [C:\dev\ros2_ws\src\ros2\rclcpp
   1203 \rclcpp\src\rclcpp\graph_listener.cpp @ 339]
   1204 000000cf`858fae40 00007ffe`c21777ee rclcpp!rclcpp::graph_listener::GraphListener::__shutdown+0x120 [C:\dev\ros2_ws\src\ros2\rclcpp\rclc
   1204 pp\src\rclcpp\graph_listener.cpp @ 359]
   1205 000000cf`858faec0 00007ffe`c217bdb7 rclcpp!rclcpp::graph_listener::GraphListener::shutdown+0x4e [C:\dev\ros2_ws\src\ros2\rclcpp\rclcpp\
   1205 src\rclcpp\graph_listener.cpp @ 374]
   1206 000000cf`858fd0e0 00007ffe`c217a488 rclcpp!<lambda_ce8062be14c7855075cc462eb8ce2d37>::operator()+0x67 [C:\dev\ros2_ws\src\ros2\rclcpp\r
   1206 clcpp\src\rclcpp\graph_listener.cpp @ 100]
   1207 000000cf`858fd140 00007ffe`c21794d8 rclcpp!std::invoke<<lambda_ce8062be14c7855075cc462eb8ce2d37> &>+0x28 [C:\Program Files (x86)\Micros
   1207 oft Visual Studio\2019\Community\VC\Tools\MSVC\14.28.29333\include\type_traits @ 1585]
   1208 000000cf`858fd170 00007ffe`c217c85f rclcpp!std::_Invoker_ret<void,1>::_Call<<lambda_ce8062be14c7855075cc462eb8ce2d37> &>+0x28 [C:\Progr
   1208 am Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.28.29333\include\functional @ 745]
   1209 000000cf`858fd1a0 00007ffe`c211aca3 rclcpp!std::_Func_impl_no_alloc<<lambda_ce8062be14c7855075cc462eb8ce2d37>,void>::_Do_call+0x2f [C:\
   1209 Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.28.29333\include\functional @ 948]
   1210 000000cf`858fd1d0 00007ffe`c211c845 rclcpp!std::_Func_class<void>::operator()+0x53 [C:\Program Files (x86)\Microsoft Visual Studio\2019
   1210 \Community\VC\Tools\MSVC\14.28.29333\include\functional @ 996]
   1211 000000cf`858fd210 00007ffe`c211bb13 rclcpp!rclcpp::Context::shutdown+0x165 [C:\dev\ros2_ws\src\ros2\rclcpp\rclcpp\src\rclcpp\context.cp
   1211 p @ 320]

There are no static destructors involved in Cyclone, nor atexit handling, nor anything really, other than the assumption that threads don’t just disappear. An assumption that is perfectly valid in all circumstances, except process termination on Windows. I have a problem with faulting Cyclone for this.

I also have a problem with hacks like recording thread handles in blocking calls to then check whether the thread perhaps has been killed. If there happened to be a wake-up of the thread in parallel (and it might, spurious wake ups and timeouts can and do happen), the thread may have already re-acquired the associated mutex by and some other thread would hang in another place after running into a locked mutex.

Perhaps you could use error checking mutexes that will give an error code in such case; but then there is the state that is protected by the mutex. That state will likely be in the process of being modified, and so continuing despite the locked mutex will likely mean working with states in which invariants no longer hold.

In other words, such tricks are not solutions to the problem, even if they hide the problem most of the time.

Rewriting the waitsets in the Cyclone RMW to not use a DDS waitset and instead use just a condition variable (I believe this is how rmw_fastrtps_cpp implemented waitsets) also would only reduce the likelihood of encountering the problem, because you’d still have the same small window where a spurious wakeup will result in a mutex being locked when it isn’t expected. The likelihood of ever encountering this problem again will be (vastly) smaller, but it will still not be eliminated.

Elimination of the risk quite simply requires cleanly shutting down all threads before leaving main if you want reliable and clean shutdown on Windows. I think strongly advising explicitly calling rclcpp::shutdown if the code is to be used on Windows would be the most practical solution.

It really is beyond me why Microsoft chose this inane design.

[ivanpauno]

When it then tries to destroy the waitset, Cyclone notes that it is in use, interrupts the blocked operation and waits for the call to dds_waitset_wait made by the graph_listener to complete. But because the thread no longer exists (and only because it no longer exists), that never happens and the process hangs:

Thanks for the answer @eboasson !
That's interesting, so the hanging is completely generated in rclcpp and not in any global destructor of cyclone, and it doesn't happen in other rmw implementations because cyclone has the extra protection of not allowing you to remove an entity while you're using it in a wait set (which sounds like a reasonable thing).

Maybe, we could modify rclcpp so the graph listener thread is not always running, but it only runs while you have a node there (i.e. nodes retain shared ownership of the graph listener, and context only week ownership).
It doesn't sound ideal, but it's a fast workaround (and we don't need the graph listener running if there isn't any node).
Of course, creating nodes with static lifetimes would cause the same issue, but I think that's already not possible (?).

---

About what happens with mutex on Windows after threads are killed, it seems they start returning WAIT_ABANDONED (some sort of mutex poisoning in the windows kernel), but the windows stl will just throw an exception. But it seems that doesn't apply to condition variables, those just hang for ever 😮 .

[wjwwood]

There are no static destructors involved in Cyclone, nor atexit handling, nor anything really, other than the assumption that threads don’t just disappear. An assumption that is perfectly valid in all circumstances, except process termination on Windows. I have a problem with faulting Cyclone for this.

Ok, well I agree then it's not an issue in cyclone.

You do use static lifetime objects though? Is that for the domain participant factory? I would have assumed that it was both static and non-trivially destructed... but I'm guessing.

Though I don't 100% understand how using a condition variable in rmw_fastrtps_cpp is less likely to run into this problem. But I think that's just my limited context.

[wjwwood]

Maybe, we could modify rclcpp so the graph listener thread is not always running, but it only runs while you have a node there (i.e. nodes retain shared ownership of the graph listener, and context only week ownership).

Well, instead of changing the ownership mechanics, we could just change the run loop of graph listener to not wait on the wait set if there are no nodes. Instead using a condition variable to know when the graph listener should shutdown (due to sigint or context shutdown) and/or when a new has been added (and therefore it needs to wait on the wait set again).

[eboasson]

You do use static lifetime objects though? Is that for the domain participant factory? I would have assumed that it was both static and non-trivially destructed... but I'm guessing.

It is pure C code so only the simplest of objects ... All I really have is a bit of zero-initialised statically allocated memory that gets initialised on first use (first entity created) and de-initialised on last use (last entity deleted), after which it is back in its initial state. Managing that is done with a small state machine using some atomic operations and a bit of synchronisation (a bit of polling with a sleep for the very first bit, with a condition variable and a mutex once the run-time layer is initialised).

Though I don't 100% understand how using a condition variable in rmw_fastrtps_cpp is less likely to run into this problem. But I think that's just my limited context

In rmw_fastrtps_cpp, rmw_wait is essentially implemented as:

1. attach all specified objects
2. check objects for trigger conditions
3. if none triggered: lock mutex ; wait for condition variable ; unlock mutex
4. detach the objects

Attaching is

  void
  attachCondition(std::mutex * conditionMutex, std::condition_variable * conditionVariable)
  {
    std::lock_guard<std::mutex> lock(internalMutex_);
    conditionMutex_ = conditionMutex;
    conditionVariable_ = conditionVariable;
  }

and detaching correspondingly simple.

So in a case like this, the thread typically gets killed while blocked on the condition variable, after which the state will be inconsistent only in that the conditionMutex_ and conditionVariable_ pointers will never be reset to null, but the destructors of the guard condition and the wait set don't even inspect those and so complete without issue.

[ivanpauno]

Well, instead of changing the ownership mechanics, we could just change the run loop of graph listener to not wait on the wait set if there are no nodes. Instead using a condition variable to know when the graph listener should shutdown (due to sigint or context shutdown) and/or when a new has been added (and therefore it needs to wait on the wait set again).

The problem is that joining a thread in a static destructor on Windows seems to be broken (?).
Really, I don't understand Windows docs, but it sounds like that's either not allowed or leads to undefined behavior.

edit: I will give it a try to that approach, as after reading windows stl code I think that joining a thread in a static destructor is ok.
The issue is how you use synchronization primitives in that destructor, but let's see ...

[ivanpauno]

edit: I will give it a try to that approach, as after reading windows stl code I think that joining a thread in a static destructor is ok.
The issue is how you use synchronization primitives in that destructor, but let's see ...

I tried, but I don't see how to avoid the problem.

When I managed to avoid the deadlock when the thread "disappears", the code was deadlocking when the thread exited normally.
When I managed to get both "working", there was still the chance of the graph listener destructor trying to take a "poisoned mutex" (in windows terminology, ABANDONED_WAIT mutex) that was going to throw an exception (this can already happen in rmw_fastrtps I think, as the mutex the waitset is using internally can enter the WAIT_ABANDONED state, but that is not too "likely" too happen).

+ the code was impossible to understand.