S3 Kafka Sink: Tombstone encoded partition doesn't support timebased partioning

[joaopmoniz]

Hi there,

We are consuming the tombstones following the documentation, and the messages are stored in the path defined at tombstone.encoded.partition. We would like to also have the tombstone messages would be partitioned by a time based partition, which is the same we are using for regular messages, but this doesn't seem to be supported.
I would like to confirm if it's not supported, and ask for adding this as a feature if it's the case.

A simple S3 sink config to show this pattern would be the following:

  ...
  s3.bucket.name: "bucket"
  topics.dir: "kafka"
  partitioner.class: "io.confluent.connect.storage.partitioner.TimeBasedPartitioner"
  path.format: "YYYY/MM/dd/HH/"
  time.interval: "HOURLY"
  partition.duration.ms: "3600000"
  locale: "en-US"
  timezone: "UTC"
  store.kafka.keys: "true"
  behavior.on.null.values: "write"
  tombstone.encoded.partition: "tombstones"

What we would expect:

Regular messages:
s3://bucket/kafka/topic_name/YYYY/MM/DD/HH/message.format
s3://bucket/kafka/topic_name/YYYY/MM/DD/HH/message.keys.format

Tombstones:
s3://bucket/kafka/topic_name/tombstones/YYYY/MM/DD/HH/message.keys.format

What we have:

Regular messages:
s3://bucket/kafka/topic_name/YYYY/MM/DD/HH/message.format
s3://bucket/kafka/topic_name/YYYY/MM/DD/HH/message.keys.format

Tombstones:
s3://bucket/kafka/topic_name/tombstones/message.keys.format