default connect timeouts are too low

[tarzanek]

default cassandra timeouts are usually
5s for reads or 10s for other operations

while https://github.com/gocql/gocql/blob/master/cluster.go#L54 defaults in go are
600ms for both above

As you all know, client timeouts should be bigger than server ones, otherwise a reconnect storm or request retry storm might happen.

Could you increase default timeouts above (Timeout and ConnectTimeout ) to be above server side defaults ?

e.g. 6s for Timeout and 11s for ConnectTimeout

While at the same topic the default retry policy is also a bit hardcore (I think 3x retry per 1s), wouldn't it make sense to change it to some backoff policy? (but that said, main focus of this issue should be default timeouts, please)

[martin-sucha]

Hi @tarzanek , thank you for the suggestion and sorry for the delayed reply.

Aligning the default values with the Cassandra defaults sounds good to me.

These are Cassandra 4.1 defaults:

Config option	Default value
read_request_timeout	5s
range_request_timeout	10s
write_request_timeout	2s
counter_write_request_timeout	5s
cas_contention_timeout	1s
truncate_request_timeout	60s
request_timeout	10s

Here is a survey of some other clients:

Driver	Version	Connect timeout	Request Timeout
DataStax Java Driver	4.16	connect-timeout=5s, init-query-timeout=5s	2s
DataStax Python Driver	3.28	5s	10s
scylla-go-driver	ce81923df69	500ms	500ms
DataStax C++ Driver	2.16.2	5s	12s

It seems it might be worth opening tickets there as well.

Regarding gocql, it seems to me that we should set both connect timeout and request timeout to slightly more than 10 seconds, because range requests have default timeout 10s in Cassandra. Truncate requests are not that often and 60s seems excessive.

What was the reason you suggested Timeout lower than ConnectTimeout in the original post? Is it that multiple requests need to happen during session initialization?

[tarzanek]

so my suggestion was based on assumption that connect timeout is the first contact - which should be 10s default
but subsequent queries e.g. system.peers, ... should already use read request
but better than me assuming, we should check it in code, I didn't have time to go that deep or ask a developer of that part of codebase

[mykaul]

It'd be great to move forward with this. We've had a use case which used the default and really hit the poor node hard - with more clients trying to reconnect way too fast due to the low timeout. Should we send a PR fixing the default connect timeout value to higher value?

[martin-sucha]

@mykaul @tarzanek I prepared PR #1719 fixing the defaults and updating the docs. Could you please review?