Token aware host policy chooses primary replicas only

[havaker]

If no keyspace is specified in ClusterConfig, then the token aware host policy has a degraded functionality.

What version of Gocql are you using?

master

What did you do?

I did not specify .Keyspace in ClusterConfig and used gocql.TokenAwareHostPolicy(gocql.RoundRobinHostPolicy(), gocql.ShuffleReplicas()) as an host selection policy. I then run a few SELECT queries against a 3 node cluster.

package main

import (
	"time"

	"github.com/gocql/gocql"
)

func main() {
	/* The example assumes the following CQL was used to setup the keyspace:
	   create keyspace example with replication = { 'class' : 'SimpleStrategy', 'replication_factor' : 3 };
	   create table example.tweet(timeline text, id UUID, text text, PRIMARY KEY(id));
	*/

	cluster := gocql.NewCluster("localhost:9042")

	//cluster.Keyspace = "example"

	cluster.PoolConfig.HostSelectionPolicy = gocql.TokenAwareHostPolicy(gocql.RoundRobinHostPolicy(), gocql.ShuffleReplicas())
	session, err := cluster.CreateSession()
	if err != nil {
		panic(err)
	}
	defer session.Close()

	someUUID := gocql.TimeUUID()
	for i := 0; i < 997; i++ {
		if err := session.Query(`SELECT id FROM example.tweet WHERE id = ?`, someUUID).Exec(); err != nil {
			panic(err)
		}

		time.Sleep(time.Second)
	}
}

What did you expect to see?

I expected each of 3 nodes to receive some queries.

What did you see instead?

Only the primary replica received the select queries (even though I was operating on a 3-node cluster, on a keyspace with replication factor 3 and passed gocql.ShuffleReplicas() to TokenAwareHostPolicy).

More

This issue persists if the .Keyspace field is not equal to a keyspace directly specified in a query string. For example:

cluster.Keyspace = "example"

// later

session.Query(`SELECT id FROM differentkeyspacename.tweet WHERE id = ?`, someUUID).Exec(

The root of this problem is the way that tokenAwareHostPolicy manages replica maps for keyspaces.

When a Pick method of tokenAwareHostPolicy is invoked, replica hosts are chosen using previously build replica maps. Because replica maps are only* built for keyspace returned by tokenAwareHostPolicy.getKeyspaceName(), this codepath is triggered resulting in a selection of a primary replica only.

* they are also build when a KeyspaceUpdateEvent is received by the policy. I haven't observed any of those events during my tests.

Sending every query to its primary replica can cause unequal load in a cluster. I did not find any records of this behavior in the documentation, so I'm considering it a bug.

[martin-sucha]

Hi! I haven't tried to reproduce yet, but indeed this seems like a bug. Thanks for reporting it!