[Bug]: Subscription may not be cleaned by datacoord under corner cases, causing quota exceeded in milvus

[xiaofan-luan]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version:
- Deployment mode(standalone or cluster):
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

after #15353 is in, subscription will be cleaned up by datacoord.

However, if a node crash after a balance task, previous balance task subscription source may be deleted and there is nowhere to find the previous nodeId, causing subscription created from the crashed node can not be unsubscribed and leakage.

If this corner case happened, pulsar backlog will not be consumed due to the leaked subscription and system can not be produced by quota exceeded exception.

Expected Behavior

multiple task handled in parallel should be handled gracefully.

I would suggest add more states in etcd. A task life cycle in channel_manager should be as follow:

1. Add -> when channel is assigned
2. Offline(Added) -> when datacoord decide channel should be closed at a data node
3. Delete -> when channel is successfully removed from the datanode

Steps To Reproduce

No response

Anything else?

you can fix it by using pulsar admin to cleanup exist subscriptions.

[yanliang567]

@sunby would you take this improvement?
/assign @sunby
/unassign

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

[congqixia]

Working on extending channel_manager state. Let's keep this issue open.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.