Not able to mirror gitlab to github #200

defyjoy · 2024-01-24T03:49:44Z

I have tried whatever combination

Given all permissions to token
Used username + token
Used token file /gickup/gitlab-token.txt

I keep getting error as below . I am not sure what am I doing wrong here

Here is my configuration

source:
  gitlab:
    # - token_file: /gickup/gitlab-token.txt
    # - token_file: gitlab-token.txt
    - ssh: false # can be true or false
      # username: [email protected]
      token: glpat-abczyx
      includeorgs: 
        - myapp
      include: # this includes the repo "foobar"
        - myapp
destination:
  github:    
    - token_file: /gickup/github-token.txt # alternatively, specify token in a file
    # - token_file: github-token.txt # alternatively, specify token in a file
      organization: defyjoy   
      force: true
      visibility:
        repositories: private # private, public, default: private

log: # optional
  timeformat: 2006-01-02 15:04:05 
  file-logging: # optional
    dir: log # directory to log into
    file: gickup.log # file to log into
    maxage: 7 # keep logs for 7 days

Here are the gitlab token permissions -

Here are the error logs -

myapp-gickup-runner-1  | 2024-01-24T09:11:59+05:30 INF Reading /gickup/conf.yml file=/gickup/conf.yml
myapp-gickup-runner-1  | 2024-01-24 09:11:59 INF Configuration loaded destinations=1 pairs=1 sources=1
myapp-gickup-runner-1  | 2024-01-24 09:11:59 INF Backup run starting
myapp-gickup-runner-1  | 2024-01-24 09:11:59 INF grabbing repositories from  stage=gitlab url=
myapp-gickup-runner-1  | 2024-01-24 09:12:05 INF starting backup for https://gitlab.com/myapp/myapp.git stage=backup
myapp-gickup-runner-1  | 2024-01-24 09:12:05 INF mirroring myapp to https://github.com stage=github url=https://github.com
myapp-gickup-runner-1  | 2024-01-24 09:12:48 ERR authentication required stage=tempclone url=https://gitlab.com/myapp/myapp.git
myapp-gickup-runner-1  | 2024-01-24 09:12:52 INF Backup run complete duration=53.866037817s

The text was updated successfully, but these errors were encountered:

cooperspencer · 2024-01-24T06:30:28Z

Hi,
I'll have a look and get back to you on that.

cooperspencer · 2024-01-24T15:45:43Z

I had a look and ran into the same issue as you did. Solution for me was to add the user parameter to my gitlab config.
For the next release I'll try to find the tokens user, so that you don't need to add that in the future anymore.

source:
  gitlab:
    # - token_file: /gickup/gitlab-token.txt
    # - token_file: gitlab-token.txt
    - ssh: false # can be true or false
      user: your-username
      # username: [email protected]
      token: glpat-abczyx
      includeorgs: 
        - myapp
      include: # this includes the repo "foobar"
        - myapp

defyjoy · 2024-01-25T01:08:08Z

OK so I tried to fill in user with both the username in gitlab eg: defyjoy1212343 or my email . Both didn't work . What is the user we should be putting there ?

cooperspencer · 2024-01-25T06:44:07Z

It should be your gitlab username. I just committed code to the main branch to automatically get your username, via the token, if not specifically mentioned in the config file.
You can try to run that via the docker containers buddyspencer/gickup:dev or buddyspencer/gickup:ubuntu-dev

defyjoy · 2024-01-25T10:58:54Z

So this time it got weirder . It did receive the username from buddyspencer/gickup:dev but look at that duration. It took almost 19 minutes and was doing nothing -

╰─(base) ○ docker compose up
[+] Building 0.0s (0/0)                                                                                                                  docker:desktop-linux
[+] Running 1/0
 ✔ Container myapp-gickup-runner-1  Created                                                                                                           0.0s
Attaching to myapp-gickup-runner-1
myapp-gickup-runner-1  | 2024-01-25T16:02:37+05:30 INF Reading /gickup/conf.yml file=/gickup/conf.yml
myapp-gickup-runner-1  | 2024-01-25 16:02:37 INF Configuration loaded destinations=1 pairs=1 sources=1
myapp-gickup-runner-1  | 2024-01-25 16:02:37 INF Backup run starting
myapp-gickup-runner-1  | 2024-01-25 16:02:38 INF grabbing repositories from myusername1234 stage=gitlab url=https://gitlab.com
myapp-gickup-runner-1  | 2024-01-25 16:02:45 INF starting backup for https://gitlab.com/myapp/myapp.git stage=backup
myapp-gickup-runner-1  | 2024-01-25 16:02:45 INF mirroring myapp to https://github.com stage=github url=https://github.com

myapp-gickup-runner-1  | 2024-01-25 16:21:37 ERR authentication required stage=tempclone url=https://gitlab.com/myapp/myapp.git
myapp-gickup-runner-1  | 2024-01-25 16:21:41 INF Backup run complete duration=19m3.891281938s
myapp-gickup-runner-1 exited with code 0

The conf.yaml . I have removed the user since it's fetched automatically -

source:
  gitlab:
    - ssh: false # can be true or false
      token: "glpat-abcxyz"
      includeorgs: 
        - myapp
      include: # this includes the repo "foobar"
        - myapp
destination:
  github:    
    - token_file: /gickup/github-token.txt 
      organization: myapp   
      force: true
      visibility:
        repositories: private 
log:
  timeformat: 2006-01-02 15:04:05 
  file-logging: # optional
    dir: log # directory to log into
    file: gickup.log # file to log into
    maxage: 7

If a gitlab has a project , do we include the same in includeorgs ?

I am not very sure where am I wrong .

cooperspencer · 2024-01-25T12:12:53Z

the step "mirrorin myapp to https://github.com" contains the local clone and then the push to Github. Weird that it takes so long, is it a very large repo?

The authentication required seems to be coming from github directly. I will do more investigation on that, but I can reproduce the problem.

defyjoy · 2024-01-25T12:14:50Z

If the authentication issue is coming from github ( destination ) I can create a new PAT token and see if it works . I was confused if the authentication failed on gitlab (source) side.

defyjoy · 2024-01-25T12:21:01Z

Ok so I tried with a new token on github and gitlab both tokens does not work for auth still . Yes the repo is on larger side but this time it took around 1minute around . So probably my intermittent internet issue .

I suspect the issue is more likely towards gitlab

cooperspencer · 2024-01-25T15:33:13Z

I found the error... I didn't add the authentication during the git fetch. You can retry it with the dev containers

defyjoy · 2024-01-26T02:03:14Z

Great work !! However one last issue. I am able to work through other smaller repos but this one a pretty heavy one and receiving the message -

myapp-gickup-runner-1  | 2024-01-26 07:23:09 INF mirroring myapp to https://github.com stage=github url=https://github.com
myapp-gickup-runner-1  | 2024-01-26 07:24:25 ERR some refs were not updated stage=tempclone url=https://gitlab.com/myapp/myapp.git
myapp-gickup-runner-1  | 2024-01-26 07:24:37 INF Backup run complete duration=1m32.857118001s

Anyway to get around this ? When I clone it on my local it doesn't throw tantrums of course .

Here are the logs when I clone it on my local

remote: Enumerating objects: 510981, done.
remote: Counting objects: 100% (391/391), done.
remote: Compressing objects: 100% (384/384), done.
remote: Total 510981 (delta 228), reused 12 (delta 0), pack-reused 510590
Receiving objects: 100% (510981/510981), 369.51 MiB | 35.60 MiB/s, done.
Resolving deltas: 100% (388574/388574), done.

The downloaded size is arounf 450 MB on disk

Looking at docker memory stats I didn't find any spike which might also cause this to fail

cooperspencer · 2024-01-26T06:45:58Z

That is a great use case because I don't have such big repositories, so thanks a lot for testing it so extensively.
I created a new container for this issue: buddyspencer/gickup:issue_200

After some digging, I found that someone had a similar issue with go-git and there solution was to add Force: true to git fetch. I added that to this branch. Can you please use buddyspencer/gickup:issue_200 and test if it works?

defyjoy · 2024-01-26T07:15:30Z

Went quite far but not quite . I received this error now -

myapp-gickup-runner-1 ERR command error on refs/heads/release-myapp-one: pre-receive hook declined stage=github url=https://gitlab.com/myapp/myapp.git
myapp-gickup-runner-1  | 2024-01-26 12:42:50 INF Backup run complete duration=9m34.106607763s

cooperspencer · 2024-01-26T07:22:42Z

damn, this is a hard one. Is gickup able to backup this repo locally? So instead of using github as destination can you try to use local and tell me if that works?

defyjoy · 2024-01-26T07:53:44Z

Thanks for responding so quick . So yes there is absolutely no issue in backing up locally .

myapp-gickup-runner-1  | 2024-01-26T13:20:41+05:30 INF Reading /gickup/conf.yml file=/gickup/conf.yml
myapp-gickup-runner-1  | 2024-01-26 13:20:41 INF Configuration loaded destinations=1 pairs=1 sources=1
myapp-gickup-runner-1  | 2024-01-26 13:20:41 INF Backup run starting
myapp-gickup-runner-1  | 2024-01-26 13:20:42 INF grabbing repositories from joydeep7840104 stage=gitlab url=https://gitlab.com
myapp-gickup-runner-1  | 2024-01-26 13:20:46 INF starting backup for https://gitlab.com/myapp/myapp.git stage=backup
myapp-gickup-runner-1  | 2024-01-26 13:20:46 INF cloning gitlab.com/myapp/myapp.git/1706255446 path=/gickup/backup stage=locally
myapp-gickup-runner-1  | 2024-01-26 13:21:48 INF zipping gitlab.com/myapp/myapp.git/1706255446 path=/gickup/backup stage=locally
myapp-gickup-runner-1  | 2024-01-26 13:21:53 INF Backup run complete duration=1m12.177087158s

The destination config

destination:
  local:
    - path: /gickup/backup
      structured: true
      zip: true
      keep: 5
      bare: true

Docker compose mounted volume

    volumes:
      - ${PWD}/conf.yml:/gickup/conf.yml # Change the path of your local config ${PWD} is your current directory (where the docker-compose.yml is located)
      - ${PWD}/gitlab-token.txt:/gickup/gitlab-token.txt
      - ${PWD}/github-token.txt:/gickup/github-token.txt
      - ${PWD}/tmp:/tmp/
      - ${PWD}/backup:/gickup/backup

And the zipped file got created

cooperspencer · 2024-01-26T08:26:05Z

I pushed a new container on buddyspencer/gickup:issue_200. It won't fix the issue but hopefully tells me where it comes from. Can you please post the error message afterwards?

defyjoy · 2024-01-26T09:01:40Z

Ok so I removed the stale docker container and rerun to pull the new image however the issue remains the same and no information provided too .

myapp-gickup-runner-1  | 2024-01-26 14:30:29 ERR command error on refs/heads/release-myapp-one: pre-receive hook declined git=push stage=github url=https://gitlab.com/myapp/myapp.git
myapp-gickup-runner-1  | 2024-01-26 14:30:42 INF Backup run complete duration=9m43.378001265s

cooperspencer · 2024-01-26T09:06:59Z

It comes from the push. Can you check if you have any pre-receive hooks on your repo?

defyjoy · 2024-01-26T09:08:32Z

It comes from the push. Can you check if you have any pre-receive hooks on your repo?

Which repo ? Github or Gitlab ?

On github side the repo was created by gickup so no webhooks are there

However on gitlab side there is a sentry webhook . I believe that should not cause any issue as that just listens to gitlab events

cooperspencer · 2024-01-26T09:18:05Z

doesn't matter actually. does your repo have a .git/hooks directory?

defyjoy · 2024-01-26T09:42:07Z

You're right . The repo folder indeed does have .git/hooks directory

cooperspencer · 2024-01-26T09:45:23Z

can you check if any of the hooks in there give you issues regarding pushing the code? also try to remove the force parameter from the github push.

defyjoy · 2024-01-26T09:51:09Z

I tried creating one branch and pushing it . It doesn't seem to fail . If thats what you wanted to ask

cooperspencer · 2024-01-26T09:57:49Z

destination:
  github:    
    - token_file: /gickup/github-token.txt # alternatively, specify token in a file
    # - token_file: github-token.txt # alternatively, specify token in a file
      organization: defyjoy   
      force: false
      visibility:
        repositories: private # private, public, default: private

the other question was if there is actually anything in the hooks directory

defyjoy · 2024-01-26T10:01:33Z

Yes there are some sample stuff but not sure if that is causing the issue

cooperspencer · 2024-01-26T10:05:39Z

the pre-receive is most likey the culprit

cooperspencer · 2024-01-26T10:07:16Z

damn, no all them are just samples, nothing is active...

defyjoy · 2024-01-26T10:21:10Z

Okay so another thing I did is to remove the force parameter from github destination but yet the issue remains the same .

defyjoy · 2024-01-27T01:52:30Z

Few good news I found from another tool .

remote: warning: File release-admin/.meteor/local/db/journal/WiredTigerPreplog.0000000001 is 100.00 MB; this is larger than GitHub's recommended maximum file size of 50.00 MB
remote: warning: File src/private/export-2019-07-25T09_00_07.167Z.json is 92.02 MB; this is larger than GitHub's recommended maximum file size of 50.00 MB
remote: warning: File release-admin/.meteor/local/db/journal/WiredTigerLog.0000000001 is 100.00 MB; this is larger than GitHub's recommended maximum file size of 50.00 MB
remote: warning: File idrodata/dji-logs/dji_go_4.apk is 97.20 MB; this is larger than GitHub's recommended maximum file size of 50.00 MB
remote: error: Trace: 115082b1f60349ef7f2be39747de59735528ee23f89d6e9d0f491dc0610115d0
remote: error: See https://gh.io/lfs for more information.
remote: error: File src/imports/situation/server/tests/data/data-zones.json is 130.65 MB; this exceeds GitHub's file size limit of 100.00 MB
remote: error: GH001: Large files detected. You may want to try Git Large File Storage - https://git-lfs.github.com/.

So there are some huge files which is why it was rejecting for push . But gickup doesn't show this issue.

cooperspencer · 2024-01-27T06:50:34Z

Which tool were you using? Maybe I can check what they do differently than me.

defyjoy · 2024-01-29T02:26:03Z

I used this - https://github.com/Enteee/git-sync-mirror

cooperspencer · 2024-01-29T11:33:54Z

seems like go-git doesn't give me the full error, but thanks a lot for testing it so much. I will add git lfs to the sync option, then it should be possible to mirror your big repository to github.

cooperspencer · 2024-02-01T06:17:43Z

Quick question, is the repository on Gitlab already using LFS?

defyjoy · 2024-02-01T13:41:06Z

I think so . But I am not sure . But I do know that Gitlab is actually storing those huge files . But can't identify which branch right now

defyjoy · 2024-02-04T04:43:50Z

Hi any updates on this. ?

cooperspencer · 2024-02-04T08:24:54Z

Not yet. I'll try to do it tomorrow.

cooperspencer · 2024-02-05T14:24:04Z

I started working on it. Already had some success, but for now it only pushes your default branch. Also I am not fully happy with the error messages for it.
You can track the progress right here:
#204

cooperspencer · 2024-02-09T10:48:51Z

can you try to run it on buddyspencer/gickup:issue_200?

It worked in my tests. I had an LFS repository on Gitlab. Had two branches and mirrored the whole repo to GitHub.

defyjoy · 2024-02-13T13:59:58Z

Hi Sorry for the delay . I will test this and report tomorrow on status for this .

cooperspencer · 2024-02-15T14:26:31Z

Did you already test it?

defyjoy · 2024-02-21T14:13:43Z

I will keep you posted tomorrow on this

petarov · 2024-09-16T20:12:57Z

Hi, I guess I have a similar problem but the other way around - GitHub to GitLab. gickup 0.10.36

INF mirroring repo-name to https://gitlab.com stage=gitlab url=https://gitlab.com
ERR command error on refs/heads/master: pre-receive hook declined git=push stage=gitlab url=https://github.com/petarov/repo-name.git

I've noticed that this only happens when syncing LFS-enabled repositories.

cooperspencer · 2024-09-17T15:27:46Z

I will run some tests on this and get back to you

cooperspencer · 2024-09-17T15:33:32Z

Mind if I ask how large the repository, you try to mirror, is?

petarov · 2024-09-17T18:34:07Z

Sure, so I got maybe 10 lfs repos of different sizes, but I would say the smallest is around 1.5 MiB and the biggest around 38 MiB. Thanks for replying.

petarov · 2024-10-27T20:38:48Z

@cooperspencer Just wanted to add that my problem seems to be fixed by enabling GitLab mirroring. Previously I was using local checkout+push (for whatever reason).

      mirror:
        enabled: false

Not able to mirror gitlab to github #200

Not able to mirror gitlab to github #200

Comments

defyjoy commented Jan 24, 2024 • edited Loading

cooperspencer commented Jan 24, 2024

cooperspencer commented Jan 24, 2024

defyjoy commented Jan 25, 2024

cooperspencer commented Jan 25, 2024 • edited Loading

defyjoy commented Jan 25, 2024 • edited Loading

cooperspencer commented Jan 25, 2024

defyjoy commented Jan 25, 2024 • edited Loading

defyjoy commented Jan 25, 2024 • edited Loading

cooperspencer commented Jan 25, 2024

defyjoy commented Jan 26, 2024 • edited Loading

cooperspencer commented Jan 26, 2024

defyjoy commented Jan 26, 2024

cooperspencer commented Jan 26, 2024

defyjoy commented Jan 26, 2024 • edited Loading

cooperspencer commented Jan 26, 2024

defyjoy commented Jan 26, 2024

cooperspencer commented Jan 26, 2024

defyjoy commented Jan 26, 2024 • edited Loading

cooperspencer commented Jan 26, 2024

defyjoy commented Jan 26, 2024

cooperspencer commented Jan 26, 2024

defyjoy commented Jan 26, 2024

cooperspencer commented Jan 26, 2024

defyjoy commented Jan 26, 2024

cooperspencer commented Jan 26, 2024

cooperspencer commented Jan 26, 2024

defyjoy commented Jan 26, 2024

defyjoy commented Jan 27, 2024

cooperspencer commented Jan 27, 2024

defyjoy commented Jan 29, 2024

cooperspencer commented Jan 29, 2024 • edited Loading

cooperspencer commented Feb 1, 2024

defyjoy commented Feb 1, 2024

defyjoy commented Feb 4, 2024

cooperspencer commented Feb 4, 2024

cooperspencer commented Feb 5, 2024

cooperspencer commented Feb 9, 2024 • edited Loading

defyjoy commented Feb 13, 2024

cooperspencer commented Feb 15, 2024 • edited Loading

defyjoy commented Feb 21, 2024

petarov commented Sep 16, 2024

cooperspencer commented Sep 17, 2024

cooperspencer commented Sep 17, 2024

petarov commented Sep 17, 2024

petarov commented Oct 27, 2024

defyjoy commented Jan 24, 2024 •

edited

Loading

cooperspencer commented Jan 25, 2024 •

edited

Loading

defyjoy commented Jan 25, 2024 •

edited

Loading

defyjoy commented Jan 25, 2024 •

edited

Loading

defyjoy commented Jan 25, 2024 •

edited

Loading

defyjoy commented Jan 26, 2024 •

edited

Loading

defyjoy commented Jan 26, 2024 •

edited

Loading

defyjoy commented Jan 26, 2024 •

edited

Loading

cooperspencer commented Jan 29, 2024 •

edited

Loading

cooperspencer commented Feb 9, 2024 •

edited

Loading

cooperspencer commented Feb 15, 2024 •

edited

Loading