-
Notifications
You must be signed in to change notification settings - Fork 39
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for DB migrations #225
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We need migration tests like what we do with dendrite. For now it will suffice to just use the v0.99.4 tag and then the checkout, to make sure things start up fine.
args := flags.Args() | ||
|
||
if len(args) < 2 { | ||
flags.Usage() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Need to document usage in README or somewhere.
state/device_data_table.go
Outdated
// The update to the DB would be a no-op; don't bother with it. | ||
// This helps reduce write usage and the contention on the unique index for | ||
// the device_data table. | ||
return nil | ||
} | ||
|
||
// re-marshal and write | ||
data, err := json.Marshal(writeBack) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I thought we were going to use JSONB to stop shuttling across MBs of data every time?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like we can only reduce the data we send when swapping the device lists?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Swapping or appending surely?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks great, thank you!
return nil | ||
} | ||
|
||
func downJSONB(ctx context.Context, tx *sql.Tx) error { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should probably run this in CI too, but I won't block this PR on it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please land this PR and do any further changes in a different PR.
@@ -119,6 +120,22 @@ func (t *DeviceDataTable) Upsert(dd *internal.DeviceData) (err error) { | |||
} | |||
tempDD.DeviceLists = tempDD.DeviceLists.Combine(dd.DeviceLists) | |||
|
|||
// we already got something in the database - update by just sending the new data | |||
if len(row.Data) > 0 { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I didn't envisage this. I thought we'd do something more like:
if dd.OTKCounts != nil {
txn.Exec(`UPDATE syncv3_device_data SET json_set(data, {otk}, $1) WHERE user_id = $2 AND device_id = $3`, dd.OTKCounts, userID, deviceID)
}
...and so on.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The only problem with this is, that we are hitting the database for every change in dd
. So if we have changes to OTKCounts
and FallbackKeyTypes
, we'd hit the DB twice, instead of once. One more hit to change the DeviceLists
.
Going to have a look at how we can possibly avoid the SELECT
first. (but in a different PR)
Co-authored-by: kegsay <[email protected]>
This doesn't magically resolve performance issues just yet, as we are still hitting the database quite often to do updates.
Setting a fillfactor may reduce the load on the database, since the indexes possibly don't need to be updated, if there is still space in the block. (For reference - https://www.cybertec-postgresql.com/en/hot-updates-in-postgresql-for-better-performance/)
As for the claim that DeepEqual uses fewer allocations, benchmark results from my machine:
This PR also embeds https://github.com/pressly/goose into the
syncv3
binary, allowing automatic database migrations (solving #229)New migrations can be created using
SQL migrations work out of the box, go migrations, once the first is added, need to
import _ "github.com/matrix-org/sliding-sync/state/migrations"
inv3.go
.the
migrate
command has all the sub-commands mentioned on the README of goose.