Support self-hosted Go Modules in GitLab

[davidgwcurve]

What Renovate type, platform and version are you using?

• Self-hosted Renovate
• Version: renovate/renovate:24.63.0@sha256:b79ce7d09a53f0f4b1ffe7ec1e91a0c8f088e1858d045c83dcf8eeba5a306a46
• Gitlab EE

Describe the bug

Renovate is unable to lookup dependencies that exist in our private, self hosted Gitlab EE repositories. When an onboarding MR is raised we have warnings only for our own go mod dependencies, everything else (docker, github, etc) is fine.

Relevant debug logs

Appropriate log from renovate (i've removed domain name):

DEBUG: Failed to look up dependency [domain_name]/go/driver (repository=go/driver, packageFile=go.mod, dependency=[domain]/go/driver)
DEBUG: Go lookup source url (repository=go/driver)
       "goModule": "[domain]/go/go-curve/v2",
       "goSourceUrl": "https://[domain]/go/go-curve"
DEBUG: Datasource unknown error (repository=go/driver)
       "datasource": "go",
       "lookupName": "[domain]/go/go-curve/v2",
       "err": {
         "message": "Cannot read property 'datasource' of null",
         "stack": "TypeError: Cannot read property 'datasource' of null\n    at Object.getReleases (/usr/src/app/node_modules/renovate/lib/datasource/go/index.ts:128:18)\n    at processTicksAndRejections (internal/process/task_queues.js:93:5)\n    at fetchReleases (/usr/src/app/node_modules/renovate/lib/datasource/index.ts:218:13)\n    at Object.getPkgReleases (/usr/src/app/node_modules/renovate/lib/datasource/index.ts:272:7)\n    at Object.lookupUpdates (/usr/src/app/node_modules/renovate/lib/workers/repository/process/lookup/index.ts:55:30)\n    at fetchDepUpdates (/usr/src/app/node_modules/renovate/lib/workers/repository/process/fetch.ts:42:26)\n    at /usr/src/app/node_modules/p-map/index.js:57:22"
       }

Have you created a minimal reproduction repository?

Please read the minimal reproductions documentation to learn how to make a good minimal reproduction repository.

• This is a really small bug, it does not need a reproduction (think small typo)
• I have provided a minimal reproduction repository
• I don't have time for that, but it happens in a public repository I have linked to
• I don't have time for that, and cannot share my private repository
• The nature of this bug means it's impossible to reproduce publicly

Additional context

Looking at the logs, I could see a particularly interesting line DEBUG: Go lookup source url. I found that message in the source here,

renovate/lib/datasource/go/index.ts

Line 61 in 6b15f6e

logger.debug({ goModule, goSourceUrl }, 'Go lookup source url');

and determined the following.

The if (sourceMatch) { is true as when calling https://${goModule}?go-get=1 you do get back the go-import and go-source meta html.

What seems to happen though is that the following two if statements are not satisfied as the URL is not a github.com URL, and nor is it /^(https:\/\/[^/]*gitlab.[^/]*)\/(.*)$/. This causes getDatasource to return null.

f (goSourceUrl?.startsWith('https://github.com/')) {
      return {
        datasource: github.id,
        lookupName: goSourceUrl
          .replace('https://github.com/', '')
          .replace(/\/$/, ''),
      };
    }
    const gitlabRes = gitlabRegExp.exec(goSourceUrl);
    if (gitlabRes) {
      return {
        datasource: gitlab.id,
        registryUrl: gitlabRes[1],
        lookupName: gitlabRes[2].replace(/\/$/, ''),
      };
    }

Perhaps a resolution here could be to default to the platform that is specified in config, in our case gitlab? Or use the RENOVATE_ENDPOINT to configure the gitlabRegExp for a best guess attempt at resolving the dependency? Obviously once that is working the call to Gitlab needs to be made with the PAT to be able to read from the private repository.

So far i'm really impressed with Renovate and the flexibility it provides, we have hundreds of repositories we want to automatically manage our dependencies in but the main use case we have is allowing us to manage internal dependencies updates which it is currently failing to do. Any guidance / suggestions would be really appreciated :)

[rarkins]

We recently implemented similar logic for changelog lookups for GitLab where we check if the hostname matches with the configured endpoint hostname and if so then can know it's a GitLab platform to look up. The challenge here is that we don't really want to pass the endpoint to datasource, so maybe we could look for a solution that queries hostRules.

[github-actions]

This issue has been labeled with status:requirements. This indicates that we don't know enough to start work and further requirements or a reproduction are needed first.

This label will be replaced with status:ready once all requirements and reproductions necessary to start work have been met.

If it's not clear what is missing to move this issue forward, ask for clarification in a new comment. If you think we already have what we need to move forward, mention this in a new comment.

[davidgwcurve]

thanks @rarkins for picking this up so quickly. What are timelines are we looking like at the moment? What can I do to help get this moving? It would be amazing for us to have this as soon as is possible, it's something we have an active interest in solving due to our sheer number of repos that need better dependency management. Alternatives such as dependabot don't fit the bill so would love to get a solution here with Renovate!

[viceice]

I'll try to fix the exception now, so at least all non private deps should work.

[rarkins]

I would add the following logic alongside the existing gitlab regex one:

• Query hostRules for hostType=gitlab, url=https://[go module]
• If the answer back from hostRules contains a token field, then we know it's a GitLab host and can redirect the lookup to a gitlab tags one

[davidgwcurve]

thanks @rarkins, we can take a look if you think this is a community fix. Are you okay if I use this issue as a discussion ground if we need some support please?

[renovate-release]

🎉 This issue has been resolved in version 24.70.0 🎉

The release is available on:

• GitHub release
• 24.70.0

Your semantic-release bot 📦🚀