Database cleanup procedure for ACME data #473
Comments
hlobit
added
enhancement
needs triage
|
Hi @hlobit, I don't think right now there's another alternative to do it yourself. But I can tell you that we've started to change how ACME stores its data, and you will be able to switch to a PostgreSQL or MySQL db, using actual columns rather than just blobs of data, and you will be able to do As an alternative right now would be to start from scratch, the main problems that you can find are:
I can't think of anything else, |
|
@maraino You hit all the points that I wanted to suggest. Mainly that starting from scratch is not a bad option if you're not heavily relying on passive revocation. In the future we should have better tooling around this, but for now we kinda have to kick the bucket down the road. |
|
Am I doing something wrong? I’ve started from fresh a couple of times, each time the client renewal fails as the existing accounts don’t exist. I fix this by deleting the account config on the clients, we only have a few so it’s manageable, but it’s not something I’d like to do as more clients as added. |
|
@maraino, @dopey thanks for your answers, I see there are improvements in the pipe for that ; it will be nice to use a postgres DB in a near future to deal ACME with data.
Exactly my case, I tried to drop the database, but I had to restore a backup because of renewals failing with error Perhaps one option I have for now is to :
Does it sound good ? Is there something I should know before trying such a thing ? |
|
Hey @hlobit while that process may work, hold on before you go into it. I think it may be easier to just go in and delete the tables that you don't need. Let me see if I can put something quick and dirty together and send it out. I'll test it out locally to verify it doesn't kill my system. I forgot that most ACME users will be reusing the same one or multiple accounts. Our cli ACME client generates a new ACME account for each order. Wasteful, but hasn't caused an issue with our use case. Because of this I forget that most users are reusing accounts when they use the more popular clients. |
I've run this locally and confirmed that afterwards I'm able to create ACME certs using the same account (the same as I had been using before running the script). I tested with certbot and acme.sh clients. My |
|
Hi @dopey, thanks for this one, I was just trying on our side. This is the output I get on first run: I have tried with all other tables, looks like all other Maybe a different version of step-ca ? OK I finally could get the former list of acme tables from 0.13.3. I restarted with remaining tables: Fine ! |
|
Hey @hlobit, what version of Badger are you running? Looks like it may be V2. If so, then you should be able to install the badger cmd line tool and try out some operations like flatten/garbage-collection. I'm thinking the space that you freed just needs to be reclaimed via garbage collection. |
|
@dopey thanks for the clue. Yeah I'm not familiar with package main
import (
"fmt"
"os"
"github.com/dgraph-io/badger"
)
func main() {
opts := badger.DefaultOptions(os.Args[1])
opts.ValueDir = os.Args[1]
db, err := badger.Open(opts)
if err != nil {
panic(err)
}
defer db.Close()
again:
if err := db.RunValueLogGC(0.7); err == nil {
fmt.Println("success")
goto again
} else {
fmt.Println(err)
}
}Now I have the expected outcome, I'm interested in a way to automate it:
Thanks for your help 👍 |
|
My hunch is that even for higher-volume CAs, a GC routine that runs periodically and by default would have a negligible impact on load. (And, at the volume where it does have a big impact, the recommendation is simple: Switch to mysql or postgres. We should consider making this our recommendation for high-volume installs anyway.) However, migrating existing folks to a periodic GC—for CAs that have been running for a while and have a lot of GC backlog—is where trouble could arise, and we’d just have to get the comms right around that and give people a migration process. Maybe we say, shut down the CA and run a program that does full GC in this case. Ultimately, because GC is an online operation for Badger, and because high-volume CAs have mysql and postgres as an alternative option, I think Badger GC should be on by default and not be a separate URL that they have to hit. |
|
FYI: badger already provides a cli to do a gc, backup, restore, ... The CA should be stopped to do for example a GC, and the tool might need to be the one specific for the version of badger used. |
What would you like to be added
Wouldn't it be a great addition if there was a documented procedure or a CLI command to cleanup the DB from old ACME data ?
I would be happy to get a clue on how you would deal with DB data growth.
Why this is needed
We are using step-ca on production for more than 1 year now, and then our disk holds a badger database with ~40GB data. When using ACME challenges in the long run the database grows, and I found no relevant information about what could be done to get rid of old data (let's say all data for certificates issued before the last 3 months). I found the link to the export gist in the documentation as a starting point, but why would I stop the server and write Go code for such a task ?
I could migrate the DB from badger to mysql backend for easier maintenance, but even in that case, it seems to me there is no easy way to migrate without writing Go code...
The text was updated successfully, but these errors were encountered: