Synchronization error between OAK’s camera and IMU

[borongyuan]

Hi, @saching13 and @Serafadam

This may be an OAK firmware issue. But I want to mention it here so that users of RTAB-Map understand this issue and the workaround before fixing it.

Specifically, this issue is that when we set the timestamp of the image and IMU data in a reasonable way, there is still an offset between them. And this offset happens to be the interval between two adjacent IMU frames. In RTAB-Map the timestamp of the image frame is set to the midpoint of the exposure interval. Your documentation does not seem to specify how the timestamp of the IMU frame is set. This should generally be the sampling time of the IMU data. But I see that if the IMU frequency is set to 100 Hz, the timestamp of the data lags behind the actual sampling time by about 0.01 s. If the IMU frequency is set to 200 Hz, then this offset is about 0.005 s. The release note of depthai-core v2.23.0 mentions fixing latency of BMI270. But this issue exists in both v2.22.0 and v2.23.0. It's not clear whether it is related to the previous latency issue.

I noticed this issue when testing the performance of OpenVINS. OpenVINS can perform online calibration of camera to IMU offset. I highly recommend enabling this option for OAK cameras currently. The precision of data synchronization has a great influence on the accuracy and robustness of VIO. The general rule of thumb is that the system is basically workable when the synchronization error is within 10 ms. For severe motion, the synchronization error should be reduced to less than 1 ms. The video below demonstrates the excellent robustness of VIO after time offset compensation.

robust.2.mp4