Add support to import repository data from an exported data of Github

[lunny]

How to export migrations archive from Github

ref: https://docs.github.com/en/rest/reference/migrations

All the progress should be done via Github REST API v3, you can use curl or other tools to do that currently. It seems it currently only supports organization repositories but not individual repositories.

• Create a migration

curl \
  -X POST \
  -H "Accept: application/vnd.github.v3+json" \
  https://api.github.com/orgs/ORG/migrations \
  -d '{"repositories":["repositories"]}'

You can create a migration to include multiple repositories, but Gitea in this PR only supports one repository.

• Get migration status

curl \
  -H "Accept: application/vnd.github.v3+json" \
  https://api.github.com/orgs/ORG/migrations/<migration_id>

Remember the response of the first curl command and input the migration_id into the second curl command. The status maybe pending, exported.

When it's exported, you can download the archive. If it's pending, repeat the status check until it becomes exported.

• Download migration archive

curl \
  -H "Accept: application/vnd.github.v3+json" \
  https://api.github.com/orgs/ORG/migrations/<migration_id>/archive

The last step is to download the archive into your disk.

How to import the Github migrations archive into Gitea

In this PR, only a sub command of Gitea has been implemented to restore a Github migrations archive into Gitea.
i.e.

./gitea restore-repo --data_type=github --repo_filepath=./testdata/github_migration/migration_archive_test_repo.tar.gz --owner_name=lunny --repo_name=test_repo

This command will restore the Gitea's test_repo repository migration archive from the Gitea's testdata directory into the Gitea instance. It requires Gitea instance is running.

Progress and TODO

Almost features are finished, and users could help to verify it. But something left needs to do.

• Convert issue_event into comment with content, i.e. Labeled event missed Labels informations.
  • Plain comment
  • Add/Remove Label comment
  • Add/Remove Milestone comment
  • Close/Merge comment
  • Change issue title comment
  • Reopen comment
  • Assignee comment
  • Review comment
  • Review approval
  • Review rejection
• Convert pull_request data, currently fork repository branch hasn't been created.
• Replace attachments URLs in issue/comment as local attachments
• Fix pull request checking is always running when there are many pull requests

Something ignored

• For some reason, we cannot get user information for some reactions from the archive, those reactions will be ignored.
• Pasted images in the issue will not be replaced and will be kept as referent from Github
• References from current repository
• References from external repository

TODO(Not the PR)

• Projects should be migrated but currently not
• Should webhooks be migrated

[6543]

we should add some hints into our docu at the migration section ... (where to download at github and how to import ...)

[lunny]

we should add some hints into our docu at the migration section ... (where to download at github and how to import ...)

Will do that after the PR finished.

[noerw]

The archive contains a schema.json, that specifies a version of the export format (in the test data 1.0.1).
We should check that and reject (or warn for?) archives with other versions, as they may result in corrupt imports (or do they adhere to semver? then we can be smarter about rejecting only >= 2.x.x)

[lunny]

Done.

[codecov-commenter]

Codecov Report

❗ No coverage uploaded for pull request base (main@ce8eb20). Click here to learn what that means.
The diff coverage is 40.15%.

@@           Coverage Diff           @@
##             main   #18165   +/-   ##
=======================================
  Coverage        ?   46.30%           
=======================================
  Files           ?      847           
  Lines           ?   122308           
  Branches        ?        0           
=======================================
  Hits            ?    56635           
  Misses          ?    58814           
  Partials        ?     6859           

Impacted Files	Coverage Δ
cmd/restore_repo.go	0.00% <0.00%> (ø)
models/error.go	37.55% <0.00%> (ø)
models/migrate.go	35.95% <0.00%> (ø)
modules/git/diff.go	80.00% <ø> (ø)
modules/migration/comment.go	100.00% <ø> (ø)
modules/migration/issue.go	100.00% <ø> (ø)
modules/migration/pullrequest.go	0.00% <ø> (ø)
modules/migration/release.go	0.00% <ø> (ø)
modules/migration/retry_downloader.go	64.61% <0.00%> (ø)
modules/migration/review.go	0.00% <ø> (ø)
... and 19 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ce8eb20...ee2472a. Read the comment docs.

[pat-s]

@lunny I've put the code snippets from the OP into a script, maybe its helpful: https://gist.github.com/pat-s/7d8f8432c0e5afe941c28e6e345f38ea

[wolfogre]

Due to some merge errors like templates/repo/issue/view_content/comments.tmpl in 73e1a93 , I have to create a new branch to pick the necessary commits and remerge the main branch to it, so I will post a new PR to replace this one. It will take some time.

---

Update:

Due to being busy with other matters, I am currently not able to continue working on this PR. Sorry.

[duaneking]

May i respectfully ask if there is an update on this?

It looks like somebody was working on this months ago?

[6543]

could you resolve the conflicts? thanks :)