feat : data bucket for persisting data to gcs

[k-anshul]

PrefixedBucket closes underlying bucket so no instance will be able to read other instance's data.

[begelundmuller]

It would be great if we could configure the data directory and data bucket together (instead of passing one as config vars and another as a constructor function). What do you think about grouping storage/persistence related functionality in a single place and dependency-injecting it similar to the logger and activity client?

For example, we could have a runtime/storage package:

// runtime/storage/storage.go
type Client struct {...}

func New(dataDir, tmpDir string, bucketCfg map[string]any) *Client {...}

func NewTmp() *Client {...}

func (c *Client) WithPrefix(instanceID string) *Client {...}

func (c *Client) DataDir(elem ..string) string {...}

func (c *Client) TempDir(elem ..string) string {...}

func (c *Client) OpenBucket() (*blob.Bucket, bool, error) {...}

Which we create in start.go and pass to the runtime, like runtime.New(ctx, opts, logger, ..., storage). In tests, we can have storage.NewTmp() similar to zap.NewNop().

Then when opening a driver, it can pass r.storage.WithPrefix(instanceID) to it.

The advantages I see to this are:

1. All disk persistence is managed in one place, making it simpler to understand and easier to track storage for billing and periodic cleanup
2. In the future, enables exposing buckets in other ways than *blob.Bucket. This might be useful for e.g. caching exports where writing directly from the OLAP might be faster.

[k-anshul]

It would be great if we could configure the data directory and data bucket together (instead of passing one as config vars and another as a constructor function). What do you think about grouping storage/persistence related functionality in a single place and dependency-injecting it similar to the logger and activity client?

Yeah sounds good.
I did the changes as per suggested design. Two changes/considerations though:

1. Even for tests we will need data_dir so better to let callers use same constructor and pass a temporary directory which can later be cleaned.
2. As of now it is not possible to set connector name so all connectors of same driver will use same name but this can be easily added by making some changes in AcquireConnection. I haven't done it yet, will make the changes as part of a different PR. As of now there is no impact because of this.

[begelundmuller]

Forgot to handle tempDirPath here

[k-anshul]

OOPS. Thanks :)