BZPop bug

[adiholden]

E20240706 06:12:17.647733 2255 zset_family.cc:1343] BUG: Callback didn't run! BLOCK//claimed=1 coord_state=6 local_res=0/
E20240706 06:12:11.367316 2255 zset_family.cc:1343] BUG: Callback didn't run! BLOCK//claimed=1 coord_state=6 local_res=0/

[romange]

We should not put private links in our public repo (for security considerations).

[BorysTheDev]

@adiholden @romange I have found only one possible issue. We have 2 steps:

1. Check key presence
2. trans->execute()
   During 2nd step we lock db_slice::cb_mu_ and can preempt so if the key is removed we will execute a transaction with a deprecated iterator.
   This bug should be fixed by db_slice and journal RegisterOnChange method refactoring #3229

[BorysTheDev]

I haven't found using of cluster, replication but the save operation happened on July 10, 2024 2:59:31 PM (GMT)

[BorysTheDev]

I've found one more bug with KEY_MOVE error and have fixed it #3334

But it looks like we have at least one more

[BorysTheDev]

I haven't found the real reason for this bug, but it looks like the execution order was next:

1. run trans->WaitOnWatch(...)
2. run trans->CancelBlocking(...)
   2.1) Transaction::coordinator_state_ |= COORD_CANCELLED;
   2.2) Transaction::local_result_ = status; // where status != OK
3. Transaction::local_result_ somehow should be set OK
4. trans->WaitOnWatch(..) returns Transaction::local_result_ // that is already OK

The problem is that I don't see how the 3rd step can happen.
Also, it looks like the 2nd step was done from the connection closing code:

  owner->RegisterBreakHook([res, this](uint32_t) {
    if (res->transaction)
      res->transaction->CancelBlocking(nullptr);
    this->server_family().BreakOnShutdown();
  });

[romange]

ok, how can you prove that 2 has happened, and it was due to RegisterBreakHook?

[BorysTheDev]

coord_state=6 means that the operation was canceled, I think about RegisterBreakHook because we don't have a replication and a cluster so only one option is left

[dranikpg]

Based on the logs... isn't the key an empty string?

[BorysTheDev]

Based on the logs... isn't the key an empty string?

I took it into account that is why I have written about the 3rd step

[dranikpg]

Fixed with #3371