Move migrating repository from frontend to backend #6200

lunny · 2019-02-27T15:18:21Z

This PR bring migrating an external repository from frontend to backend. When migrating, you could also visit the repo home page, but it will only display a migrating gif image and text about doing. And in fact the migrating is queueing or running on the backend. Also this page will check if migrating is done, and then it will reload itself to display the code page.

main.go

codecov-io · 2019-02-28T12:22:39Z

Codecov Report

Merging #6200 into master will decrease coverage by 0.07%.
The diff coverage is 30.53%.

@@            Coverage Diff             @@
##           master    #6200      +/-   ##
==========================================
- Coverage   41.84%   41.77%   -0.08%     
==========================================
  Files         497      505       +8     
  Lines       65808    66267     +459     
==========================================
+ Hits        27539    27681     +142     
- Misses      34734    35010     +276     
- Partials     3535     3576      +41

Impacted Files	Coverage Δ
models/migrations/migrations.go	`1.3% <ø> (ø)`	⬆️
models/migrations/v99.go	`0% <0%> (ø)`
modules/structs/task.go	`0% <0%> (ø)`
modules/task/queue_redis.go	`0% <0%> (ø)`
routers/init.go	`65.33% <0%> (-2.73%)`	⬇️
routers/private/serv.go	`32.78% <0%> (-1.41%)`	⬇️
routers/repo/view.go	`40% <0%> (-2.32%)`	⬇️
models/models.go	`62.17% <100%> (+0.24%)`	⬆️
modules/migrations/gitea.go	`13.54% <100%> (+4.84%)`	⬆️
modules/setting/task.go	`100% <100%> (ø)`
... and 28 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0a96e59...badbf9e. Read the comment docs.

lunny · 2019-02-28T12:22:51Z

This is ready for review.

lafriks · 2019-02-28T13:20:53Z

Wasn't there an alternative PR for this?

public/js/index.js

jolheiser

A few small nits, but otherwise looks pretty nifty. Excited to see this, it will be perfect for longer running migrations.

public/js/index.js

templates/repo/migrating.tmpl

modules/task/task.go

lunny · 2019-03-01T00:33:33Z

@lafriks I searched migrate, migrating, backend on pulls but found none.

jolheiser · 2019-03-01T01:24:56Z

I think he's referring to #5949

lunny · 2019-03-01T02:02:09Z

@jolheiser right! it's similar. @zeripath @lafriks

zeripath · 2019-03-01T03:55:22Z

I think a task queue is likely a better approach than just spawning a goroutine every migration so I'm happy to close my pr for that. However I haven't had a proper chance to review this PR to see if ideas in my pr could be used to improve this.

jonasfranz · 2019-03-09T14:32:21Z

conflicted

lunny · 2019-04-02T05:22:35Z

It's ready to review.

custom/conf/app.ini.sample

models/models.go

models/repo.go

models/migrations/v84.go

models/repo.go

zeripath

One additional thing to these below, you should prevent cmd/Serv.go from allowing stuff to be done to a repository that is in the IsCreating.

zeripath · 2019-04-07T06:32:22Z

modules/task/migrate.go

+			var buf bytes.Buffer
+			fmt.Fprintf(&buf, "Handler crashed with error: %v", e)
+
+			for i := 1; ; i++ {


For consistency of I recommend you use log.Stack(2).

zeripath · 2019-04-07T06:45:14Z

models/repo.go

@@ -210,6 +220,11 @@ type Repository struct {
 	UpdatedUnix util.TimeStamp `xorm:"INDEX updated"`
 }

+// IsCreating indicates that repository is creating
+func (repo *Repository) IsCreating() bool {


See comments in models/repo.go.

Suggest:
IsInstantiating(), IsInitializing() or IsBeingCreated()?

zeripath · 2019-04-07T06:52:26Z

models/repo.go

+// all kinds of RepositoryStatus
+const (
+	RepositoryCreated  RepositoryStatus = iota // a normal repository
+	RepositoryCreating                         // repository is migrating or forking


Generally x is creating reads as x is creating (some unspoken) y. I suggest change these to:

RepositoryReady
RepositoryInitializing or RepositoryInstantiating or BeingCreated

(Admittedly Initializing and Instantiating are both ambiguous, but these verbs are more likely to be used reflexively in their intransitive present continuous form than create.)

Would it be sensible to make this a general lifecycle marker? We could move RepositoryArchived in here too - unless you think it should be possible to create archived repositories from scratch. Similarly RepositoryEmpty, as we don't really care if a repository is empty when it's archived.

zeripath · 2019-04-07T07:32:27Z

modules/task/migrate.go

+	"code.gitea.io/gitea/modules/util"
+)
+
+func handleCreateError(owner *models.User, err error, name string) error {


I think we need to think a bit about how we internationalise errors in general. It's sort of a shame to take what is an abstract form which could be translatable and turn it into a non-translatable string.

I agree. However, errors are special in that they should not be taken lightly: the format string needs to have the correct placeholders ('%d', '%s', etc.). If not for security reasons, at least to prevent a bigger failure; an i18n solution for this should validate the user format string and use a hard-coded default if the check fails.
Anyway, that's for another PR.

zeripath · 2019-04-07T10:21:28Z

modules/task/migrate.go

+		return err
+	}
+
+	repo, err := models.MigrateRepositoryGitData(t.Doer, t.Owner, t.Repo, *opts)


I think you probably should check that the repo is still Creating (or Initializing/Instantiating) either here or in MigrateRepositoryGitData - and either fail if not or put it into the IsCreating status before you start doing anything.

Co-Authored-By: guillep2k <[email protected]>

lunny · 2019-10-13T09:35:01Z

@guillep2k @6543 @lafriks It's ready to reveiw again.

6543

tested

* BREAKING * Remove legacy handling of drone token (go-gitea#8191) * Change repo search to use exact match for topic search. (go-gitea#7941) * Add pagination for admin api get orgs and fix only list public orgs bug (go-gitea#7742) * Implement the ability to change the ssh port to match what is in the gitea config (go-gitea#7286) * FEATURE * Org/Members: display 2FA members states + optimize sql requests (go-gitea#7621) * SetDefaultBranch on pushing to empty repository (go-gitea#7610) * Adds side-by-side diff for images (go-gitea#6784) * API method to list all commits of a repository (go-gitea#6408) * Password Complexity Checks (go-gitea#6230) * Add option to initialize repository with labels (go-gitea#6061) * Add additional password hash algorithms (go-gitea#6023) * BUGFIXES * Fix errors in create org UI regarding team access permission (go-gitea#8506) * Fix bug on FindExternalUsersByProvider (go-gitea#8504) * Create .ssh dir as necessary (go-gitea#8486) * IsBranchExist: return false if provided name is empty (go-gitea#8485) * Making openssh listen on SSH_LISTEN_PORT not SSH_PORT (go-gitea#8477) * Add check for empty set when dropping indexes during migration (go-gitea#8471) * LFS files are relative to LFS content path, ensure that when deleting they are made relative to this (go-gitea#8455) * Ensure Request Body Readers are closed in LFS server (go-gitea#8454) * Fix template bug on mirror repository setting page (go-gitea#8438) * Fix migration v96 to keep issue attachments (go-gitea#8435) * Update strk.kbt.io/projects/go/libravatar to latest (go-gitea#8429) * Singular form for files that has only one line (go-gitea#8416) * Check for either escaped or unescaped wiki filenames (go-gitea#8408) * Allow users with explicit read access to give approvals (go-gitea#8382) * Fix editor commit to new branch if PR disabled (go-gitea#8375) * readd .markdown class to all markup renderers (go-gitea#8357) * Upgrade xorm to v0.7.9 to fix some bugs (go-gitea#8354) * Fix column name ambiguity in GetUserIssueStats() (go-gitea#8347) * Change general form binding to gogs form (go-gitea#8334) * Fix pull request commit status in user dashboard list (go-gitea#8321) * Fix repo_admin_change_team_access always checked in org settings (go-gitea#8319) * Update to github.com/lafriks/[email protected] (go-gitea#8317) * Show correct commit status in PR list (go-gitea#8316) * Bugfix for image compare and minor improvements to image compare (go-gitea#8289) * Update xorm (go-gitea#8286) * Fix API for edit and delete release attachment (go-gitea#8285) * Fix nil object access in some conditions when parsing cross references (go-gitea#8281) * Fix label count (go-gitea#8267) * Only show teams access for organization repositories on collaboration setting page (go-gitea#8265) * Test more reserved usernames (go-gitea#8263) * Rewrite reference processing code in preparation for opening/closing from comment references (go-gitea#8261) * Fix assets key on release webhook (go-gitea#8253) * Allow registration when button is hidden (go-gitea#8237) * Fix release API URL generation (go-gitea#8234) * Fix milestone num_issues (go-gitea#8221) * MS Teams webhook misses commit messages (go-gitea#8209) * Fix data race (go-gitea#8204) * Fix team user api (go-gitea#8172) * Fix pull merge 500 error caused by git-fetch breaking behaviors (go-gitea#8161) * Make show private icon when repo avatar set (go-gitea#8144) * Add reviewers as participants (go-gitea#8121) * Fix Go 1.13 private repository go get issue (go-gitea#8112) * feat: highlight issue references with : (go-gitea#8101) * Make AllowedUsers configurable in sshd_config (go-gitea#8094) * Strict name matching for Repository.GetTagID() (go-gitea#8074) * Avoid ambiguity of branch/directory names for the git-diff-tree command (go-gitea#8066) * Add change title notification for issues (go-gitea#8061) * [ssh] fix the config specification in the authorized_keys template (go-gitea#8031) * Fix reading git notes from nested trees (go-gitea#8026) * Fixes synchronize tags to releases for repository - makes sure we are only getting tag refs (go-gitea#7990) * Fix adding default Telegram webhook (go-gitea#7972) * Run CORS handler first for /api routes (go-gitea#7967) * Abort synchronization from LDAP source if there is some error. (go-gitea#7960) * Fix wrong sender when send slack webhook (go-gitea#7918) * Fix bug when migrating a private repository (go-gitea#7917) * Evaluate emojis in commit messages in list view (go-gitea#7906) * Fix upload file type check (go-gitea#7890) * lfs/lock: round locked_at timestamp to second (go-gitea#7872) * fix non existent milestone with 500 error instead of 404 (go-gitea#7867) * gpg/bugfix: Use .ExpiredUnix.IsZero to display green color of forever valid gpg key (go-gitea#7846) * Fix duplicate call of webhook (go-gitea#7821) * Enable switching to a different source branch when PR already exists (go-gitea#7819) * Convert files to utf-8 for indexing (go-gitea#7814) * Do not fetch all refs in pull-request compare (go-gitea#7797) * Fix multiple bugs with statuses endpoints at API (go-gitea#7785) * Restore functionality for early gits (go-gitea#7775) * Fix Slack webhook fork message (go-gitea#7774) * Rewrite existing repo units if setting is not included in api body (go-gitea#7763) * Fix rename failed when rewrite public keys (go-gitea#7761) * Fix approvals counting (go-gitea#7757) * Add migration step to remove old repo_indexer_status orphaned records (go-gitea#7746) * Fix repo_index_status lingering when deleting a repository (go-gitea#7734) * Remove camel case tokenization from repo indexer (go-gitea#7733) * Fix milestone completness calculation when migrating (go-gitea#7725) * Regression: Include "executable" files in the index, as they are not necessarily … (go-gitea#7718) * Fixes indexed repos keeping outdated indexes when files grow too large (go-gitea#7712) * Skip non-regular files (e.g. submodules) on repo indexing (go-gitea#7711) * Fix dropTableColumns sqlite implementation (go-gitea#7710) * Update gopkg.in/src-d/go-git.v4 to v4.13.1 (go-gitea#7705) * improve branches list performance and fix protected branch icon when no-login (go-gitea#7695) * Correct wrong datetime format for git (go-gitea#7689) * Move add to hook queue for created repo to outside xorm session. (go-gitea#7675) * sugestion to use range .Branches (go-gitea#7674) * Fix bug on migrating milestone from github (go-gitea#7665) * hide delete/restore button on archived repos (go-gitea#7658) * css: use flex to fix floating paginate (go-gitea#7656) * Fix syntax highlight initialization (go-gitea#7617) * Fix panic on push at - Merging pull request causes 500 error (go-gitea#7615) * Make PKCS8, PEM and SSH2 keys work (go-gitea#7600) * Fix mistake in arc-green.less split-diff css code. (go-gitea#7587) * Handle ErrUserProhibitLogin in http git (go-gitea#7586) * Fix bug create/edit wiki pages when code master branch protected (go-gitea#7580) * Fixes Malformed URLs in API git/commits response (go-gitea#7565) * Fix file header overflow in file and blame views (go-gitea#7562) * Improve SSH key parser to handle newlines in keys (go-gitea#7522) * Fix empty commits now showing in repo overview (go-gitea#7521) * Fix repository's pull request count error (go-gitea#7518) * Fix markdown invoke sequence (go-gitea#7513) * Remove duplicated webhook trigger (go-gitea#7511) * Update User.NumRepos atomically in createRepository (go-gitea#7493) * Fix settings page of repo you aren't admin print error - Settings pages giving UnitType error message (go-gitea#7482) * Fix redirection after file edit - Handles all redirects for Web UI File CRUD (go-gitea#7478) * cmd/serv: actually exit after fatal errors (go-gitea#7458) * Fix an issue with some pages throwing 'not defined' js exceptions (go-gitea#7450) * fix Dropzone.js integration (go-gitea#7445) * Fix regex for issues in commit messages (go-gitea#7444) * Diff: Fix indentation on unhighlighted code (go-gitea#7435) * Only show "New Pull Request" button if repo allows pulls (go-gitea#7426) * Upgrade macaron/captcha to fix random error problem (go-gitea#7407) * create class for inline positioned lists (go-gitea#7393) * Fetch refs for successful testing for tag (go-gitea#7388) * add missing template variable on organisation settings (go-gitea#7385) * fix post parameter - on issue list - unset assignee (go-gitea#7380) * fix/define autochecked checkboxes on issue list in firefox (go-gitea#7320) * only return head: null if source branch was deleted (go-gitea#6705) * ENHANCEMENT * Add nofollow to sign in links (go-gitea#8509) * vendor: update mvdan.cc/xurls/v2 to v2.1.0 (go-gitea#8495) * Update milestone issues numbers when save milestone and other code improvements (go-gitea#8411) * Add extra user information when migrating release (go-gitea#8331) * Require overall success if no context is given for status check (go-gitea#8318) * Transaction-aware retry create issue to cope with duplicate keys (go-gitea#8307) * Change link on issue milestone (go-gitea#8246) * Alwaywas return local url for users avatar (go-gitea#8245) * Move some milestone functions to a standalone package (go-gitea#8213) * Move create issue comment to comments package (go-gitea#8212) * Disable max height property of comment textarea (go-gitea#8203) * Add 'Mentioning you' group to /issues page (go-gitea#8201) * oauth2 with remote Gitea (go-gitea#8149) * Reference issues from pull requests and other issues (go-gitea#8137) * Fix webhooks to use proxy from environment (go-gitea#8116) * Add merged commit id on pull view when it's merged (go-gitea#8062) * Add teams to repo on collaboration page. (go-gitea#8045) * Update swagger to 0.20.1 (go-gitea#8010) * Make link last commit massages in repository home page and commit tables (go-gitea#8006) * Add API endpoint for accessing repo topics (go-gitea#7963) * Include description in repository search (go-gitea#7942) * Use gitea forked macaron (go-gitea#7933) * Fix pull creation with empty changes (go-gitea#7920) * Allow token as authorization for accessing attachments (go-gitea#7909) * Retry create issue to cope with duplicate keys (go-gitea#7898) * Move git diff codes from models to services/gitdiff (go-gitea#7889) * migrate gplus to google oauth2 provider (go-gitea#7885) * Remove unique filter from repo indexer analyzer. (go-gitea#7878) * Detect delimiter in CSV rendering (go-gitea#7869) * Import topics during migration (go-gitea#7851) * Move CreateReview to modules/pull (go-gitea#7841) * vendor: update pdf.js to v2.1.266 (go-gitea#7834) * Support SSH_LISTEN_PORT env var in docker app.ini template (go-gitea#7829) * Add Ability for User to Customize Email Notification Frequency (go-gitea#7813) * Move database settings from models to setting (go-gitea#7806) * Display ui time with customize time location (go-gitea#7792) * Implement webhook branch filter (go-gitea#7791) * Restrict repository indexing by glob match (go-gitea#7767) * Api: advanced settings for repository (external wiki, issue tracker etc.) (go-gitea#7756) * Update migrated repositories' issues/comments/prs poster id if user has a github external user saved (go-gitea#7751) * deps: Upgrade gopkg.in/editorconfig/editorconfig-core-go.v1 (go-gitea#7749) * Apply emoji on commit graph page (go-gitea#7743) * Add a lot of extension to language mappings for syntax highlights (go-gitea#7741) * Add SQL execution on log and indexes on table repository and comment (go-gitea#7740) * Set DB connection error level to error (go-gitea#7724) * Check commit message hashes before making links (go-gitea#7713) * remove unnecessary fmt on generate bindata (go-gitea#7706) * Fix specific highlighting (CMakeLists.txt ...) (go-gitea#7686) * Add file status on API (go-gitea#7671) * Add support for DEFAULT_ORG_MEMBER_VISIBLE (go-gitea#7669) * Provide links in commit summaries in commits table/view list (go-gitea#7659) * Change length of some repository's columns (go-gitea#7652) * Move commit repo action from models to repofiles package (go-gitea#7645) * fix wrong email when use gitea as OAuth2 provider (go-gitea#7640) * [Branch View] add download button (go-gitea#7604) * Update to [email protected] (go-gitea#7596) * use 403 instead of 401 for ErrUserProhibitLogin (go-gitea#7591) * Removed unnecessary conversions (go-gitea#7557) * Un-lambda base.FileSize (go-gitea#7556) * Added missing error checks in tests (go-gitea#7554) * Move create release from models to a standalone package (go-gitea#7539) * Make default branch name link to default branch (go-gitea#7519) * Added total count of contributions to heatmap (go-gitea#7517) * Move mirror to a standalone package from models (go-gitea#7486) * Move models.PushUpdate to repofiles.PushUpdate (go-gitea#7485) * Include thread related headers in issue/coment mail (go-gitea#7484) * Refuse merge until all required status checks success (go-gitea#7481) * convert all js var to let/const (go-gitea#7464) * Only create branches for opened pull requestes when migrating from github (go-gitea#7463) * jQuery 3 (go-gitea#7425) * Add notification placeholder (go-gitea#7409) * Search Commits via Commit Hash (go-gitea#7400) * Move status table to cron package (go-gitea#7370) * wiki - page revisions list (go-gitea#7369) * Display original author and URL information when showing migrated issues/comments (go-gitea#7352) * Refactor filetype is not allowed errors (go-gitea#7309) * switch to use gliderlabs/ssh for builtin server (go-gitea#7250) * Remove settting dependency on modules/session (go-gitea#7237) * Move all mail related codes from models to services/mailer (go-gitea#7200) * Support git.PATH entry in app.ini (go-gitea#6772) * Support setting cookie domain (go-gitea#6288) * Move migrating repository from frontend to backend (go-gitea#6200) * Delete releases attachments if release is deleted (go-gitea#6068) * SECURITY * Ignore mentions for users with no access (go-gitea#8395) * Be more strict with git arguments (go-gitea#7715) * reserve .well-known username (go-gitea#7637) * TRANSLATION * Latvian translation for home page (go-gitea#8468) * Add home template italian translation (go-gitea#8352) * fix misprint (go-gitea#7452) * BUILD * use go 1.13 (go-gitea#8088) * MISC * add file line count info on UI (go-gitea#8396) * Make issues page left menu 100% width and add reponame as title attribute (go-gitea#8359) * [arc-green] white on hover for active menu items (go-gitea#8344) * Move ref (branch or tag) location on issue list page (go-gitea#8157) * apply emoji on dashboard issue list labels (go-gitea#8156) * 1148: Take up the full width when viewing the diff in split view. (go-gitea#8114) * Display description of 'make this repo private' as help text, not as tooltip (go-gitea#8097) * Fixes deformed emoji in pull request reviews (go-gitea#8047) * Add strike to old header on comment (go-gitea#8046) * Add tooltip for the visibility checkbox in /repo/create (go-gitea#8025) * Update github.com/lafriks/xormstore and tidy up mod.go (go-gitea#8020) * keep blame view buttons sequence consistent with normal view when view a file (go-gitea#8007) * Use "Pull Request" instead of "Merge Request" (go-gitea#8003) * Move line number to :before attr to hide from search on browser (go-gitea#8002) * Changed black color to white for (read) number label on issue list page (go-gitea#8000) * [Branch View] show "New Pull Request" Button only if posible (go-gitea#7977) * Fix hook problem by only setting the git environment variables if we are passed them (go-gitea#7854) * Prevent Commit Status and Message From Overflowing On Branch Page (go-gitea#7800) * Fix global search result CSS, misc CSS tweaks (go-gitea#7789) * Tweak label border CSS (go-gitea#7739) * Fix create menu item widths (go-gitea#7708) * Extract the username and password from the mirror url (go-gitea#7651) * [Branch View] Delete duplicate protection symbol (go-gitea#7624) * [Branch View] Delete Table Header (go-gitea#7622) * [Branch View] icons to buttons (go-gitea#7602) * update js dependencies (go-gitea#7462) * Add Extra Info to Branches Page (go-gitea#7461) * Bump lodash from 4.17.11 to 4.17.14 (go-gitea#7459) * wiki history improvements (go-gitea#7391) * ui fixes - compare view and archieved repo issues (go-gitea#7345) * dark theme scrollbars (go-gitea#7269) * wiki - editor - add buttons 'inline code', 'empty checkbox', 'checked checkbox' (go-gitea#7243) * Fix Statuses API only shows first 10 statuses: Add paging and extend API GetCommitStatuses (go-gitea#7141)

* BREAKING * Fix deadline on update issue or PR via API (go-gitea#8698) * Hide some user information via API if user doesn't have enough permission (go-gitea#8655) (go-gitea#8657) * Remove legacy handling of drone token (go-gitea#8191) * Change repo search to use exact match for topic search. (go-gitea#7941) * Add pagination for admin api get orgs and fix only list public orgs bug (go-gitea#7742) * Implement the ability to change the ssh port to match what is in the gitea config (go-gitea#7286) * SECURITY * Ignore mentions for users with no access (go-gitea#8395) * Be more strict with git arguments (go-gitea#7715) * reserve .well-known username (go-gitea#7637) * FEATURE * Org/Members: display 2FA members states + optimize sql requests (go-gitea#7621) * SetDefaultBranch on pushing to empty repository (go-gitea#7610) * Adds side-by-side diff for images (go-gitea#6784) * API method to list all commits of a repository (go-gitea#6408) * Password Complexity Checks (go-gitea#6230) * Add option to initialize repository with labels (go-gitea#6061) * Add additional password hash algorithms (go-gitea#6023) * BUGFIXES * Allow to merge if file path contains " or \ (go-gitea#8629) (go-gitea#8771) * On windows set core.longpaths true (go-gitea#8776) (go-gitea#8786) * Fix 500 when edit hook (go-gitea#8782) (go-gitea#8789) * Fix Checkbox at RepoSettings Protected Branch (go-gitea#8799) (go-gitea#8801) * Fix SSH2 conditional in key parsing code (go-gitea#8806) (go-gitea#8810) * Fix commit expand button to not go to commit link (go-gitea#8745) (go-gitea#8825) * Fix new user form for non-local users (go-gitea#8826) (go-gitea#8828) * Fix to close opened io resources as soon as not needed (go-gitea#8839) (go-gitea#8846) * Fix edit content button on migrated issue content (go-gitea#8877) (go-gitea#8884) * Fix require external registration password (go-gitea#8885) (go-gitea#8890) * Fix password complexity check on registration (go-gitea#8887) (go-gitea#8888) * Update Github Migration Tests (go-gitea#8896) (go-gitea#8938) (go-gitea#8945) * Fix issue with user.fullname (go-gitea#8903) * Enable punctuations ending mentions (go-gitea#8889) (go-gitea#8894) * Add Close() method to gogitRepository (go-gitea#8901) (go-gitea#8956) * Hotfix for review actions and notifications (go-gitea#8965) * Expose db.SetMaxOpenConns and allow non MySQL dbs to set conn pool params (go-gitea#8528) (go-gitea#8618) * Fix milestone close timestamp (go-gitea#8728) (go-gitea#8730) * Fix 500 when getting user as unauthenticated user (go-gitea#8653) (go-gitea#8663) * Fix 'New Issue Missing Milestone Comment' (go-gitea#8678) (go-gitea#8681) * Use AppSubUrl for more redirections (go-gitea#8647) (go-gitea#8651) * Add SubURL to redirect path (go-gitea#8632) (go-gitea#8634) * Fix template error on account page (go-gitea#8562) (go-gitea#8622) * Allow externalID to be UUID (go-gitea#8551) (go-gitea#8624) * Prevent removal of non-empty emoji panel following selection of duplicate (go-gitea#8609) (go-gitea#8623) * Update heatmap fixtures to restore tests (go-gitea#8615) (go-gitea#8616) * Ensure that diff stats can scroll independently of the diff (go-gitea#8581) (go-gitea#8621) * Webhook: set Content-Type for application/x-www-form-urlencoded (go-gitea#8600) * Fix go-gitea#8582 by handling empty repos (go-gitea#8587) (go-gitea#8594) * Fix bug on pull requests when transfer head repository (go-gitea#8564) (go-gitea#8569) * Add missed close in ServeBlobLFS (go-gitea#8527) (go-gitea#8542) * Ensure that GitRepo is set on Empty repositories (go-gitea#8539) (go-gitea#8541) * Fix migrate mirror 500 bug (go-gitea#8526) (go-gitea#8530) * Fix password complexity regex for special characters (go-gitea#8524) * Prevent .code-view from overriding font on icon fonts (go-gitea#8614) (go-gitea#8627) * Allow more than 255 characters for tokens in external_login_user table (go-gitea#8554) * Fix errors in create org UI regarding team access permission (go-gitea#8506) * Fix bug on FindExternalUsersByProvider (go-gitea#8504) * Create .ssh dir as necessary (go-gitea#8486) * IsBranchExist: return false if provided name is empty (go-gitea#8485) * Making openssh listen on SSH_LISTEN_PORT not SSH_PORT (go-gitea#8477) * Add check for empty set when dropping indexes during migration (go-gitea#8471) * LFS files are relative to LFS content path, ensure that when deleting they are made relative to this (go-gitea#8455) * Ensure Request Body Readers are closed in LFS server (go-gitea#8454) * Fix template bug on mirror repository setting page (go-gitea#8438) * Fix migration v96 to keep issue attachments (go-gitea#8435) * Update strk.kbt.io/projects/go/libravatar to latest (go-gitea#8429) * Singular form for files that has only one line (go-gitea#8416) * Check for either escaped or unescaped wiki filenames (go-gitea#8408) * Allow users with explicit read access to give approvals (go-gitea#8382) * Fix editor commit to new branch if PR disabled (go-gitea#8375) * readd .markdown class to all markup renderers (go-gitea#8357) * Upgrade xorm to v0.7.9 to fix some bugs (go-gitea#8354) * Fix column name ambiguity in GetUserIssueStats() (go-gitea#8347) * Change general form binding to gogs form (go-gitea#8334) * Fix pull request commit status in user dashboard list (go-gitea#8321) * Fix repo_admin_change_team_access always checked in org settings (go-gitea#8319) * Update to github.com/lafriks/[email protected] (go-gitea#8317) * Show correct commit status in PR list (go-gitea#8316) * Bugfix for image compare and minor improvements to image compare (go-gitea#8289) * Update xorm (go-gitea#8286) * Fix API for edit and delete release attachment (go-gitea#8285) * Fix nil object access in some conditions when parsing cross references (go-gitea#8281) * Fix label count (go-gitea#8267) * Only show teams access for organization repositories on collaboration setting page (go-gitea#8265) * Test more reserved usernames (go-gitea#8263) * Rewrite reference processing code in preparation for opening/closing from comment references (go-gitea#8261) * Fix assets key on release webhook (go-gitea#8253) * Allow registration when button is hidden (go-gitea#8237) * Fix release API URL generation (go-gitea#8234) * Fix milestone num_issues (go-gitea#8221) * MS Teams webhook misses commit messages (go-gitea#8209) * Fix data race (go-gitea#8204) * Fix team user api (go-gitea#8172) * Fix pull merge 500 error caused by git-fetch breaking behaviors (go-gitea#8161) * Make show private icon when repo avatar set (go-gitea#8144) * Add reviewers as participants (go-gitea#8121) * Fix Go 1.13 private repository go get issue (go-gitea#8112) * feat: highlight issue references with : (go-gitea#8101) * Make AllowedUsers configurable in sshd_config (go-gitea#8094) * Strict name matching for Repository.GetTagID() (go-gitea#8074) * Avoid ambiguity of branch/directory names for the git-diff-tree command (go-gitea#8066) * Add change title notification for issues (go-gitea#8061) * [ssh] fix the config specification in the authorized_keys template (go-gitea#8031) * Fix reading git notes from nested trees (go-gitea#8026) * Fixes synchronize tags to releases for repository - makes sure we are only getting tag refs (go-gitea#7990) * Fix adding default Telegram webhook (go-gitea#7972) * Run CORS handler first for /api routes (go-gitea#7967) * Abort synchronization from LDAP source if there is some error. (go-gitea#7960) * Fix wrong sender when send slack webhook (go-gitea#7918) * Fix bug when migrating a private repository (go-gitea#7917) * Evaluate emojis in commit messages in list view (go-gitea#7906) * Fix upload file type check (go-gitea#7890) * lfs/lock: round locked_at timestamp to second (go-gitea#7872) * fix non existent milestone with 500 error instead of 404 (go-gitea#7867) * gpg/bugfix: Use .ExpiredUnix.IsZero to display green color of forever valid gpg key (go-gitea#7846) * Fix duplicate call of webhook (go-gitea#7821) * Enable switching to a different source branch when PR already exists (go-gitea#7819) * Convert files to utf-8 for indexing (go-gitea#7814) * Do not fetch all refs in pull-request compare (go-gitea#7797) * Fix multiple bugs with statuses endpoints at API (go-gitea#7785) * Restore functionality for early gits (go-gitea#7775) * Fix Slack webhook fork message (go-gitea#7774) * Rewrite existing repo units if setting is not included in api body (go-gitea#7763) * Fix rename failed when rewrite public keys (go-gitea#7761) * Fix approvals counting (go-gitea#7757) * Add migration step to remove old repo_indexer_status orphaned records (go-gitea#7746) * Fix repo_index_status lingering when deleting a repository (go-gitea#7734) * Remove camel case tokenization from repo indexer (go-gitea#7733) * Fix milestone completness calculation when migrating (go-gitea#7725) * Regression: Include "executable" files in the index, as they are not necessarily … (go-gitea#7718) * Fixes indexed repos keeping outdated indexes when files grow too large (go-gitea#7712) * Skip non-regular files (e.g. submodules) on repo indexing (go-gitea#7711) * Fix dropTableColumns sqlite implementation (go-gitea#7710) * Update gopkg.in/src-d/go-git.v4 to v4.13.1 (go-gitea#7705) * improve branches list performance and fix protected branch icon when no-login (go-gitea#7695) * Correct wrong datetime format for git (go-gitea#7689) * Move add to hook queue for created repo to outside xorm session. (go-gitea#7675) * sugestion to use range .Branches (go-gitea#7674) * Fix bug on migrating milestone from github (go-gitea#7665) * hide delete/restore button on archived repos (go-gitea#7658) * css: use flex to fix floating paginate (go-gitea#7656) * Fix syntax highlight initialization (go-gitea#7617) * Fix panic on push at - Merging pull request causes 500 error (go-gitea#7615) * Make PKCS8, PEM and SSH2 keys work (go-gitea#7600) * Fix mistake in arc-green.less split-diff css code. (go-gitea#7587) * Handle ErrUserProhibitLogin in http git (go-gitea#7586) * Fix bug create/edit wiki pages when code master branch protected (go-gitea#7580) * Fixes Malformed URLs in API git/commits response (go-gitea#7565) * Fix file header overflow in file and blame views (go-gitea#7562) * Improve SSH key parser to handle newlines in keys (go-gitea#7522) * Fix empty commits now showing in repo overview (go-gitea#7521) * Fix repository's pull request count error (go-gitea#7518) * Fix markdown invoke sequence (go-gitea#7513) * Remove duplicated webhook trigger (go-gitea#7511) * Update User.NumRepos atomically in createRepository (go-gitea#7493) * Fix settings page of repo you aren't admin print error - Settings pages giving UnitType error message (go-gitea#7482) * Fix redirection after file edit - Handles all redirects for Web UI File CRUD (go-gitea#7478) * cmd/serv: actually exit after fatal errors (go-gitea#7458) * Fix an issue with some pages throwing 'not defined' js exceptions (go-gitea#7450) * fix Dropzone.js integration (go-gitea#7445) * Fix regex for issues in commit messages (go-gitea#7444) * Diff: Fix indentation on unhighlighted code (go-gitea#7435) * Only show "New Pull Request" button if repo allows pulls (go-gitea#7426) * Upgrade macaron/captcha to fix random error problem (go-gitea#7407) * create class for inline positioned lists (go-gitea#7393) * Fetch refs for successful testing for tag (go-gitea#7388) * add missing template variable on organisation settings (go-gitea#7385) * fix post parameter - on issue list - unset assignee (go-gitea#7380) * fix/define autochecked checkboxes on issue list in firefox (go-gitea#7320) * only return head: null if source branch was deleted (go-gitea#6705) * ENHANCEMENT * Add nofollow to sign in links (go-gitea#8509) * vendor: update mvdan.cc/xurls/v2 to v2.1.0 (go-gitea#8495) * Update milestone issues numbers when save milestone and other code improvements (go-gitea#8411) * Add extra user information when migrating release (go-gitea#8331) * Require overall success if no context is given for status check (go-gitea#8318) * Transaction-aware retry create issue to cope with duplicate keys (go-gitea#8307) * Change link on issue milestone (go-gitea#8246) * Alwaywas return local url for users avatar (go-gitea#8245) * Move some milestone functions to a standalone package (go-gitea#8213) * Move create issue comment to comments package (go-gitea#8212) * Disable max height property of comment textarea (go-gitea#8203) * Add 'Mentioning you' group to /issues page (go-gitea#8201) * oauth2 with remote Gitea (go-gitea#8149) * Reference issues from pull requests and other issues (go-gitea#8137) * Fix webhooks to use proxy from environment (go-gitea#8116) * Add merged commit id on pull view when it's merged (go-gitea#8062) * Add teams to repo on collaboration page. (go-gitea#8045) * Update swagger to 0.20.1 (go-gitea#8010) * Make link last commit massages in repository home page and commit tables (go-gitea#8006) * Add API endpoint for accessing repo topics (go-gitea#7963) * Include description in repository search (go-gitea#7942) * Use gitea forked macaron (go-gitea#7933) * Fix pull creation with empty changes (go-gitea#7920) * Allow token as authorization for accessing attachments (go-gitea#7909) * Retry create issue to cope with duplicate keys (go-gitea#7898) * Move git diff codes from models to services/gitdiff (go-gitea#7889) * migrate gplus to google oauth2 provider (go-gitea#7885) * Remove unique filter from repo indexer analyzer. (go-gitea#7878) * Detect delimiter in CSV rendering (go-gitea#7869) * Import topics during migration (go-gitea#7851) * Move CreateReview to modules/pull (go-gitea#7841) * vendor: update pdf.js to v2.1.266 (go-gitea#7834) * Support SSH_LISTEN_PORT env var in docker app.ini template (go-gitea#7829) * Add Ability for User to Customize Email Notification Frequency (go-gitea#7813) * Move database settings from models to setting (go-gitea#7806) * Display ui time with customize time location (go-gitea#7792) * Implement webhook branch filter (go-gitea#7791) * Restrict repository indexing by glob match (go-gitea#7767) * Api: advanced settings for repository (external wiki, issue tracker etc.) (go-gitea#7756) * Update migrated repositories' issues/comments/prs poster id if user has a github external user saved (go-gitea#7751) * deps: Upgrade gopkg.in/editorconfig/editorconfig-core-go.v1 (go-gitea#7749) * Apply emoji on commit graph page (go-gitea#7743) * Add a lot of extension to language mappings for syntax highlights (go-gitea#7741) * Add SQL execution on log and indexes on table repository and comment (go-gitea#7740) * Set DB connection error level to error (go-gitea#7724) * Check commit message hashes before making links (go-gitea#7713) * remove unnecessary fmt on generate bindata (go-gitea#7706) * Fix specific highlighting (CMakeLists.txt ...) (go-gitea#7686) * Add file status on API (go-gitea#7671) * Add support for DEFAULT_ORG_MEMBER_VISIBLE (go-gitea#7669) * Provide links in commit summaries in commits table/view list (go-gitea#7659) * Change length of some repository's columns (go-gitea#7652) * Move commit repo action from models to repofiles package (go-gitea#7645) * fix wrong email when use gitea as OAuth2 provider (go-gitea#7640) * [Branch View] add download button (go-gitea#7604) * Update to [email protected] (go-gitea#7596) * use 403 instead of 401 for ErrUserProhibitLogin (go-gitea#7591) * Removed unnecessary conversions (go-gitea#7557) * Un-lambda base.FileSize (go-gitea#7556) * Added missing error checks in tests (go-gitea#7554) * Move create release from models to a standalone package (go-gitea#7539) * Make default branch name link to default branch (go-gitea#7519) * Added total count of contributions to heatmap (go-gitea#7517) * Move mirror to a standalone package from models (go-gitea#7486) * Move models.PushUpdate to repofiles.PushUpdate (go-gitea#7485) * Include thread related headers in issue/coment mail (go-gitea#7484) * Refuse merge until all required status checks success (go-gitea#7481) * convert all js var to let/const (go-gitea#7464) * Only create branches for opened pull requestes when migrating from github (go-gitea#7463) * jQuery 3 (go-gitea#7425) * Add notification placeholder (go-gitea#7409) * Search Commits via Commit Hash (go-gitea#7400) * Move status table to cron package (go-gitea#7370) * wiki - page revisions list (go-gitea#7369) * Display original author and URL information when showing migrated issues/comments (go-gitea#7352) * Refactor filetype is not allowed errors (go-gitea#7309) * switch to use gliderlabs/ssh for builtin server (go-gitea#7250) * Remove settting dependency on modules/session (go-gitea#7237) * Move all mail related codes from models to services/mailer (go-gitea#7200) * Support git.PATH entry in app.ini (go-gitea#6772) * Support setting cookie domain (go-gitea#6288) * Move migrating repository from frontend to backend (go-gitea#6200) * Delete releases attachments if release is deleted (go-gitea#6068) * TRANSLATION * Latvian translation for home page (go-gitea#8468) * Add home template italian translation (go-gitea#8352) * fix misprint (go-gitea#7452) * BUILD * use go 1.13 (go-gitea#8088) * MISC * add file line count info on UI (go-gitea#8396) * Make issues page left menu 100% width and add reponame as title attribute (go-gitea#8359) * [arc-green] white on hover for active menu items (go-gitea#8344) * Move ref (branch or tag) location on issue list page (go-gitea#8157) * apply emoji on dashboard issue list labels (go-gitea#8156) * 1148: Take up the full width when viewing the diff in split view. (go-gitea#8114) * Display description of 'make this repo private' as help text, not as tooltip (go-gitea#8097) * Fixes deformed emoji in pull request reviews (go-gitea#8047) * Add strike to old header on comment (go-gitea#8046) * Add tooltip for the visibility checkbox in /repo/create (go-gitea#8025) * Update github.com/lafriks/xormstore and tidy up mod.go (go-gitea#8020) * keep blame view buttons sequence consistent with normal view when view a file (go-gitea#8007) * Use "Pull Request" instead of "Merge Request" (go-gitea#8003) * Move line number to :before attr to hide from search on browser (go-gitea#8002) * Changed black color to white for (read) number label on issue list page (go-gitea#8000) * [Branch View] show "New Pull Request" Button only if posible (go-gitea#7977) * Fix hook problem by only setting the git environment variables if we are passed them (go-gitea#7854) * Prevent Commit Status and Message From Overflowing On Branch Page (go-gitea#7800) * Fix global search result CSS, misc CSS tweaks (go-gitea#7789) * Tweak label border CSS (go-gitea#7739) * Fix create menu item widths (go-gitea#7708) * Extract the username and password from the mirror url (go-gitea#7651) * [Branch View] Delete duplicate protection symbol (go-gitea#7624) * [Branch View] Delete Table Header (go-gitea#7622) * [Branch View] icons to buttons (go-gitea#7602) * update js dependencies (go-gitea#7462) * Add Extra Info to Branches Page (go-gitea#7461) * Bump lodash from 4.17.11 to 4.17.14 (go-gitea#7459) * wiki history improvements (go-gitea#7391) * ui fixes - compare view and archieved repo issues (go-gitea#7345) * dark theme scrollbars (go-gitea#7269) * wiki - editor - add buttons 'inline code', 'empty checkbox', 'checked checkbox' (go-gitea#7243) * Fix Statuses API only shows first 10 statuses: Add paging and extend API GetCommitStatuses (go-gitea#7141)

During a large code move in go-gitea#6200 the OriginalURL field was accidentially changed to be populated with the CloneAddr field which will contain the username and/or password provided during a migration. This behavior was fixed in previous PR go-gitea#9097 and this migration will remove any authentication details that were stored in the database between those two.

* Add migration to sanitize repository original_url During a large code move in #6200 the OriginalURL field was accidentially changed to be populated with the CloneAddr field which will contain the username and/or password provided during a migration. This behavior was fixed in previous PR #9097 and this migration will remove any authentication details that were stored in the database between those two. * use net/url to rebuild URL instead of strings.Replace * Update models/migrations/migrations.go * changes per lunny * make fmt

lunny added the type/enhancement An improvement of existing functionality label Feb 27, 2019

techknowlogick reviewed Feb 27, 2019

View reviewed changes

main.go Outdated Show resolved Hide resolved

GiteaBot added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label Feb 27, 2019

yasuokav mentioned this pull request Feb 28, 2019

Duplicate post request #6189

Closed

7 tasks

lunny mentioned this pull request Feb 28, 2019

Gitea hosted Gitea #1029

Open

20 tasks

lunny force-pushed the lunny/task_migrate branch from 222a54e to a4a2a95 Compare February 28, 2019 08:29

lunny changed the title ~~WIP: move migrating to backend~~ Move migrating repository from frontend to backend Feb 28, 2019

lunny added this to the 1.9.0 milestone Feb 28, 2019

lunny force-pushed the lunny/task_migrate branch from c240072 to c4217b8 Compare February 28, 2019 12:36

lafriks reviewed Feb 28, 2019

View reviewed changes

public/js/index.js Outdated Show resolved Hide resolved

jolheiser requested changes Feb 28, 2019

View reviewed changes

public/js/index.js Outdated Show resolved Hide resolved

templates/repo/migrating.tmpl Outdated Show resolved Hide resolved

modules/task/task.go Outdated Show resolved Hide resolved

lunny force-pushed the lunny/task_migrate branch from 4a3ea2c to b7f4ee9 Compare March 1, 2019 03:40

lunny force-pushed the lunny/task_migrate branch from 5ffaa9b to a3205ad Compare March 9, 2019 14:58

lunny mentioned this pull request Mar 19, 2019

Unable to migrate large repository #6361

Closed

7 tasks

lunny force-pushed the lunny/task_migrate branch from 5e5d72d to 5b91019 Compare April 2, 2019 02:11

lafriks requested changes Apr 2, 2019

View reviewed changes

custom/conf/app.ini.sample Outdated Show resolved Hide resolved

models/models.go Show resolved Hide resolved

models/repo.go Outdated Show resolved Hide resolved

lunny force-pushed the lunny/task_migrate branch from 5b91019 to ce2fb09 Compare April 3, 2019 14:35

lafriks requested changes Apr 4, 2019

View reviewed changes

models/migrations/v84.go Outdated Show resolved Hide resolved

models/repo.go Outdated Show resolved Hide resolved

lunny force-pushed the lunny/task_migrate branch from b9a4f1c to 1abf66e Compare April 7, 2019 00:31

zeripath reviewed Apr 7, 2019

View reviewed changes

lunny force-pushed the lunny/task_migrate branch from ff20474 to 381cf14 Compare April 12, 2019 08:20

lunny and others added 8 commits October 12, 2019 23:49

Update docs/content/doc/advanced/config-cheat-sheet.en-us.md

6c6bde7

Co-Authored-By: guillep2k <[email protected]>

rename ErrTaskIsNotExist to ErrTaskDoesNotExist

220de6f

delete release after add one on tests to make it run happy

a1a1358

fix tests

45ab0cd

fix tests

5e1888f

improve codes

fb4e7f3

fix lint

80c9e8d

fix lint

8edb889

lunny force-pushed the lunny/task_migrate branch from 9ebad04 to 8edb889 Compare October 12, 2019 15:49

fix migrations

badbf9e

6543 approved these changes Oct 13, 2019

View reviewed changes

GiteaBot added lgtm/need 1 This PR needs approval from one additional maintainer to be merged. and removed lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. labels Oct 13, 2019

lafriks approved these changes Oct 13, 2019

View reviewed changes

GiteaBot added lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. and removed lgtm/need 1 This PR needs approval from one additional maintainer to be merged. labels Oct 13, 2019

lafriks requested a review from jolheiser October 13, 2019 12:44

lunny merged commit f2a3abc into go-gitea:master Oct 13, 2019

lunny deleted the lunny/task_migrate branch October 13, 2019 13:23

6543 mentioned this pull request Oct 13, 2019

Update migrated repositories' issues/comments/prs poster id if user has a github external user saved #7751

Merged

3 tasks

guillep2k mentioned this pull request Oct 18, 2019

[Branch View] [README] Screenshot Update #7986

Closed

6543 mentioned this pull request Oct 24, 2019

Repositories are broken after unsuccessful clone/mirror creation #1249

Closed

7 tasks

mrsdizzie mentioned this pull request Dec 19, 2019

Add migration to sanitize repository original_url #9423

Merged

alexanderadam mentioned this pull request Jun 10, 2020

GitHub Migrations for Enterprise GitHub servers #11815

Closed

go-gitea locked and limited conversation to collaborators Nov 24, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move migrating repository from frontend to backend #6200

Move migrating repository from frontend to backend #6200

lunny commented Feb 27, 2019 •

edited

Loading

codecov-io commented Feb 28, 2019 •

edited

Loading

lunny commented Feb 28, 2019

lafriks commented Feb 28, 2019

jolheiser left a comment

lunny commented Mar 1, 2019

jolheiser commented Mar 1, 2019

lunny commented Mar 1, 2019

zeripath commented Mar 1, 2019

jonasfranz commented Mar 9, 2019

lunny commented Apr 2, 2019

zeripath left a comment

zeripath Apr 7, 2019

zeripath Apr 7, 2019

zeripath Apr 7, 2019

zeripath Apr 7, 2019

guillep2k Oct 8, 2019

zeripath Apr 7, 2019

lunny commented Oct 13, 2019

6543 left a comment

Move migrating repository from frontend to backend #6200

Move migrating repository from frontend to backend #6200

Conversation

lunny commented Feb 27, 2019 • edited Loading

codecov-io commented Feb 28, 2019 • edited Loading

Codecov Report

lunny commented Feb 28, 2019

lafriks commented Feb 28, 2019

jolheiser left a comment

Choose a reason for hiding this comment

lunny commented Mar 1, 2019

jolheiser commented Mar 1, 2019

lunny commented Mar 1, 2019

zeripath commented Mar 1, 2019

jonasfranz commented Mar 9, 2019

lunny commented Apr 2, 2019

zeripath left a comment

Choose a reason for hiding this comment

zeripath Apr 7, 2019

Choose a reason for hiding this comment

zeripath Apr 7, 2019

Choose a reason for hiding this comment

zeripath Apr 7, 2019

Choose a reason for hiding this comment

zeripath Apr 7, 2019

Choose a reason for hiding this comment

guillep2k Oct 8, 2019

Choose a reason for hiding this comment

zeripath Apr 7, 2019

Choose a reason for hiding this comment

lunny commented Oct 13, 2019

6543 left a comment

Choose a reason for hiding this comment

lunny commented Feb 27, 2019 •

edited

Loading

codecov-io commented Feb 28, 2019 •

edited

Loading