Optical Flow's alignment with Images

[AbdessamadElK]

Dear authors,

I am interested in a new architecture where it is important that Optical Flow be aligned with Images. However, there is always a misalignment between images and their corresponding optical flow. Even after warping images towards events, the problem persists.

Please take a look at the following picture to have a better understanding of the issue:

Left : Optical flow (white) on events
Middle : Events on image
Right : Optical flow on image
Note that the images are warped toward events and that the events here are not rectified.

Is there a way to have the optical flow perfectly (or almost perfectly) aligned with the images ?

Looking forward for your reply.

Thank you in advance !

[magehrig]

the optical flow gt is on the rectified event camera (see https://dsec.ifi.uzh.ch/data-format/)

[AbdessamadElK]

Hello Mr Mathias,

Thank you very much for your answer!
So does that mean that there is no way for the images to be aligned with their corresponding optical flow ?

[Yaozhuwa]

@AbdessamadElK

So does that mean that there is no way for the images to be aligned with their corresponding optical flow ?

#25 may help.

[AbdessamadElK]

@Yaozhuwa
Thank you very much for your comment.

I already followed the steps in #25 to map the left image to the event camera frame, and also tried using the DSEC-det dataset directly. This allowed the images to be aligned with the events perfectly, but I am still struggling with the optical flow:

As you can see, the optical flow is in the rectified event camera as Mr. @magehrig indicated. You can also see that the warped images are indeed aligned with the distorted events but not with the optical flow.

Looking at the picture in your comment, It seems like you managed to align the optical flow with the images. Could you please tell me what kind of warping did you perform?

[Yaozhuwa]

@AbdessamadElK

Could you please tell me what kind of warping did you perform?

I did nothing to optical flow.

It seems you use the wrong optical flow to align with the image. For input image1 and image2, the optical flow will align with image1 rather than image2.

[AbdessamadElK]

@Yaozhuwa You might be right, I just loaded the flows and their corresponding images based on the file names, and I don't know if that's the right way to do it.

I will check again and see if that's the problem, thank you for your help!