Dump github/gitlab/gitea repository data to a local directory and restore to gitea

[lunny]

Currently online migrating is possible but in fact, it depends on the internet network status what's a mess. This PR makes it possible to dump a repository on GitHub/GitLab/Gitea to a disk directory and then transfer it to another Gitea instance via the possible transfer method. And then restore the repository on the target Gitea instance.

i.e. Gitea repository itself in GitHub spent about 80 GB but transfer 80GB online is almost impossible. Then we can dump the repository to a host nearby Github's host and then compress it and transfer it to another host.

A new Gitea command has been added to do the things, you can use it as below:

./gitea dump-repo --repo_dir ./dumps --clone_addr https://github.com/go-xorm/builder.git --auth_username <token> --owner_name xorm --repo_name builder

A dumped repository(go-gitea/test_repo) directory structure like this.

Another command is restore-repo added from this PR which could restore the repository to a Gitea instance from the dumped folder.

./gitea restore-repo --repo_dir ./dumps/xorm/builder --owner_name xorm --repo_name builder

And this could be also used as a backup tool to backup repositories in Github/Gitlab/Gitea .

• Add a command gitea dump-repo
• Add a command gitea restore-repo

[6543]

@lunny one thing left: we should also save the git-service type and source url somewhere

[6543]

... to be able to migrate issue/pull ownership to existing users as we can do with the normal github/gitlab migration now

[codecov-commenter]

Codecov Report

Merging #12244 into master will increase coverage by 0.66%.
The diff coverage is 1.49%.

@@            Coverage Diff             @@
##           master   #12244      +/-   ##
==========================================
+ Coverage   42.59%   43.25%   +0.66%     
==========================================
  Files         671      645      -26     
  Lines       73635    71468    -2167     
==========================================
- Hits        31365    30917     -448     
+ Misses      37184    35543    -1641     
+ Partials     5086     5008      -78     

Impacted Files	Coverage Δ
cmd/dump_repo.go	0.00% <0.00%> (ø)
modules/migrations/base/pullrequest.go	0.00% <ø> (ø)
modules/migrations/dump.go	0.00% <0.00%> (ø)
modules/structs/repo.go	54.54% <ø> (+4.54%)	⬆️
modules/migrations/migrate.go	22.47% <41.66%> (+0.91%)	⬆️
routers/api/v1/settings/settings.go	44.44% <0.00%> (-55.56%)	⬇️
modules/storage/local.go	0.00% <0.00%> (-38.10%)	⬇️
modules/queue/queue.go	24.39% <0.00%> (-7.87%)	⬇️
models/attachment.go	54.86% <0.00%> (-6.25%)	⬇️
modules/webhook/matrix.go	30.30% <0.00%> (-4.67%)	⬇️
... and 208 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 54dd28f...5342029. Read the comment docs.

[techknowlogick]

please resolve conflict

[techknowlogick]

CI fail is related:

level=warning msg="[runner] Can't run linter goanalysis_metalinter: S1008: failed prerequisites: [([email protected]/gitea/modules/cron, [email protected]/gitea/modules/cron): analysis skipped: errors in package: [/go/src/code.gitea.io/gitea/modules/cron/tasks_basic.go:12:2: could not import code.gitea.io/gitea/modules/migrations (/go/src/code.gitea.io/gitea/modules/migrations/dump.go:514:29: cannot use downloader (variable of type *RepositoryRestorer) as base.Downloader value in argument to DoMigrateRepository: wrong type for method GetAsset)]]"
level=warning msg="[runner] Can't run linter unused: buildir: analysis skipped: errors in package: [/go/src/code.gitea.io/gitea/modules/migrations/dump.go:514:29: cannot use downloader (variable of type *RepositoryRestorer) as base.Downloader value in argument to DoMigrateRepository: wrong type for method GetAsset -: could not load export data: no export data for \"code.gitea.io/gitea/modules/migrations\"]"
level=error msg="Running error: buildir: analysis skipped: errors in package: [/go/src/code.gitea.io/gitea/modules/migrations/dump.go:514:29: cannot use downloader (variable of type *RepositoryRestorer) as base.Downloader value in argument to DoMigrateRepository: wrong type for method GetAsset -: could not load export data: no export data for \"code.gitea.io/gitea/modules/migrations\"]"

[6543]

I think we should stave CloneURL to yaml too
why not move Topics into a generell repo.yml ?
so the local "import" still has the right urls ...

[codecov-io]

Codecov Report

Merging #12244 (695d23e) into master (cf9d471) will decrease coverage by 0.29%.
The diff coverage is 3.25%.

@@            Coverage Diff             @@
##           master   #12244      +/-   ##
==========================================
- Coverage   42.36%   42.06%   -0.30%     
==========================================
  Files         728      733       +5     
  Lines       78015    78666     +651     
==========================================
+ Hits        33049    33093      +44     
- Misses      39540    40159     +619     
+ Partials     5426     5414      -12     

Impacted Files	Coverage Δ
cmd/dump_repo.go	0.00% <0.00%> (ø)
cmd/restore_repo.go	0.00% <0.00%> (ø)
models/admin.go	60.31% <0.00%> (-8.78%)	⬇️
models/task.go	34.95% <ø> (+0.04%)	⬆️
modules/migrations/base/downloader.go	17.51% <ø> (ø)
modules/migrations/base/pullrequest.go	0.00% <ø> (ø)
modules/migrations/dump.go	0.00% <0.00%> (ø)
modules/migrations/error.go	100.00% <ø> (ø)
modules/migrations/git.go	46.42% <ø> (+3.09%)	⬆️
modules/migrations/gitea_downloader.go	0.91% <0.00%> (+<0.01%)	⬆️
... and 29 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 212fa34...695d23e. Read the comment docs.

[6543]

@lunny "migrations/restore.go exported methods should have comment or be unexported"

[lunny]

@techknowlogick done.