Problem with Cartesian interface: cartesianSolver/left_arm suspended

[paliasgh]

Hello,

Recently, often when we send a movement command using iarm->goToPoseSync(), the robot suddenly freezes and we need to restart Cartesian interfaces. I found these errors happening in iKinCartesianSolver of left_arm:

	49.783905	WARNING			cartesianSolver/left_arm: timeout detected on part left_arm!
	49.804049	WARNING			cartesianSolver/left_arm: timeout detected on part left_arm!
	49.824174	WARNING			cartesianSolver/left_arm: timeout detected on part left_arm!
	49.844250	WARNING			cartesianSolver/left_arm: timeout detected on part left_arm!
	49.854303	INFO			cartesianSolver/left_arm suspended

At the same time, yarprobotinterface starts publishing these errors:

	49.281161	ERROR	yarp.devices.controlBoard_nws_yarp		Encoder Timestamps are not consistent! Data will not be published.
	49.291197	ERROR	yarp.devices.controlBoard_nws_yarp		Encoder Timestamps are not consistent! Data will not be published.
	49.301251	ERROR	yarp.devices.controlBoard_nws_yarp		Encoder Timestamps are not consistent! Data will not be published.
	49.311316	ERROR	yarp.devices.controlBoard_nws_yarp		Encoder Timestamps are not consistent! Data will not be published.

When I try to stop the modules in startup application, iKinCartesianSolver of right_arm and yarprobotinterface both fail to close properly and I will have to kill them. Here is a log session of the startup application:

yarprunlog_01_02_2024_16_51_45.log

We will appreciate your help!

[Nicogene]

x-ref:

• Rethink consistency checker in control board NWS/YARP yarp#2939

Not sure it is related

[pattacini]

Hi @paliasgh

This is not a problem with the Cartesian Solver per se but rather with the joints control board of the right arm (i.e., yarprobotinterface), which stops streaming out the encoders. The Solver simply detects that and pauses itself.

As @Nicogene correctly pointed out above there's an ongoing discussion in the PR addressing this situation.

cc @PeterBowman @randaz81

[randaz81]

If I remember correctly, the message "Encoder Timestamps are not consistent" is printed when the timestamps between two encoders/boards differ more than one second. This is a huge amount of time, and I suspect that the system is affected by some other lower-level issue e.g. hardware problem? network delays/bandwidth bottleneck? unstable wifi? cpu overload?
Therefore bypassing the check, and forcing a stream of inconsistent data will probably not solve the issue, it will just move it to another sw level (potentially dangerous: point of view of the controller)

[paliasgh]

Hello @randaz81 and @pattacini ,

This issue is now happening very often, unfortunately, and is affecting our work with the robot. Do you have any suggestions on how we can find the exact cause of the issue? Thanks a lot.

Something I tested was even after iKinCartesianSolver for left arm is no longer available, I can still move it through yarpmotorgui. The encoder values seem okay there. Also, the issue happens only for the left arm, never for the right arm.

[randaz81]

Which kind of robot?

[randaz81]

From the log I can understand that it is a ethernet based icub, however the log serms not complete. Since the issue seems of hardware nature, I'd proceed opening a support request on https://github.com/robotology/icub-tech-support/issues. I suggest to provide a full-sized log (maybe this one is limited to 10000 lines? Check the logger gui options). I'd try keeping a ping on the all the ethernet boards 10.0.1.x of the left arm, checking for internal network errors/delays/packets losses.