Do not implement batches using transactions #104
Conversation
Transactions have a size limit and will only accept a finite number of writes before returning ErrTxnTooBig. Batching should not be subjected to this limit. This PR fixes this by using a badger WriteBatch object to implement batching. The WriteBatch object packs writes into transactions, committing the transactions as they become full. This fixes Issue #92.
|
Thank you for submitting this PR!
Getting other community members to do a review would be great help too on complex PRs (you can ask in the chats/forums). If you are unsure about something, just leave us a comment.
We currently aim to provide initial feedback/triaging within two business days. Please keep an eye on any labelling actions, as these will indicate priorities and status of your contribution. |
| // Batch creats a new Batch object. This provides a way to do many writes, when | ||
| // there may be too many to fit into a single transaction. | ||
| // | ||
| // After writing to a Batch, always call Commit whether or not writing to the |
There was a problem hiding this comment.
This kind of contract needs to be specified at the interface level to be effective. I think your original solution of adding a Cancel method to the interface is the right way to go (we can add it here now, then add it to the interface later.
For now, we could also use runtime.SetFinalizer to ensure everything gets cleaned up eventually. The overhead should be relatively minimal.
There was a problem hiding this comment.
Added a Cancel function and finalizer. Will add Cancel to datastore.Batch interface later.
Transactions have a size limit and will only accept a finite number of writes before returning ErrTxnTooBig. Batching should not be subjected to this limit.
This PR fixes this by using a badger WriteBatch object to implement batching. The WriteBatch object packs writes into transactions, committing the transactions as they become full.
This fixes Issue #92.