bug: compaction task stuck for hours

[hzxa21]

Describe the bug

Rencently we have seen a small cg3 (MV compaction group) L0->L0 compaction task getting stuck for hours, which cause L0->base level compaction to block for the corresponding compaction group. Some findings:

• The stuck compaction task contains 60 SSTs with total size=~13MB so it is a relatively small task.
• Compactor is alive and can handle tasks from both the same and different compaction groups as usual.
• We saw "Ready to handle compaction task" log but not "Finished compaction task" log. This means compactor indeed received the task but not finished it.
• The task progress and heartbeat of the stuck task is continously reported to meta node. We use a guard to make sure the task progress is cleared if the task finishes or errors out so that means the task is indeed stuck in the compactor side.
• Metric "Compacting SSTable Count" reported by meta node stays at 60 for cg3 level0. This means meta indeed didn't receive ReportCompactionTasksRequest from the compactor.

To Reproduce

No response

Expected behavior

No response

Additional context

log.csv

[hzxa21]

cc @Li0k @Little-Wallace

[Little-Wallace]

We have temporarily fixed this issue by cancel a long task by #10183.
but we have not found why the task hung out.

[Li0k]

We have temporarily fixed this issue by cancel a long task by #10183. but we have not found why the task hung out.

Might fixed by #10584 ?