Unable to add self-hosted git repo as Data source

[sonicfather]

NetBox version

v3.5.1

Python version

3.9

Steps to Reproduce

1. add git repo as Data source, with credentials and https url
2. try to sync and watch logs

Expected Behavior

Synchro of self-hosted git repo as Data source.

Observed Behavior

Logs return an ssl error whereas custom CA certificate are installed. I try to git clone directly from the netbox worker and it is OK, only git clone using Data source does not work.

In the log we got this error:

17:14:24 default: core.jobs.sync_datasource(job=<Job: 64d84b92-962a-4f1c-8a21-0d03cc1f5670>) (64d84b92-962a-4f1c-8a21-0d03cc1f5670)
ERROR:root:Fetching remote data failed (MaxRetryError): HTTPSConnectionPool(host='mygitlab.mydomain', port=443): Max retries exceeded with url: /myproject.git/info/refs?service=git-upload-pack (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1123)')))
INFO:rq.worker:default: Job OK (64d84b92-962a-4f1c-8a21-0d03cc1f5670)
17:14:24 default: Job OK (64d84b92-962a-4f1c-8a21-0d03cc1f5670)
INFO:rq.worker:Result is kept for 500 seconds

[candlerb]

The simple and secure solution would be to give your local gitlab server a valid certificate signed by LetsEncrypt. If it has a public IP (v4 or v6) and accepts inbound HTTP connections on port 80, then it's easy - use a client like certbot or dehydrated with the HTTP01 challenge. If not, it becomes a bit more difficult as you have to set up the DNS01 challenge, either by integrating with your authoritative nameserver's API or by running a separate one like acme-dns.

Of course, once you have this working, you can sign all the certs you like for your domains - and automatically fetch new ones every 60-90 days. Also consider certgrinder which is good if you want a single host to perform all the certificate fetching work on behalf of your other hosts; those other hosts don't need any Internet connectivity at all.

The other option would be to attempt to get your Netbox installation to trust the (private) certificate authority that you used to sign your gitlab server's certificate - which may be the server certificate itself, if it's self-signed.

For these use cases it might be nice if the "data sources" in Netbox had the option to specify a different CA certificate for each data source when using https URLs.

Since Netbox doesn't have this feature, the only other option would be to try and install the root CA certificate, but there are several places you might need to put the CA cert to get it trusted, including your system certificate store, and the python certifi package inside the Netbox virtualenv. Certificate stores vary significantly between OSes and I'm not going to guess how it might be done on yours.

[sonicfather]

Thanks for your answer.

To precise my issue, it is not a self-signed certificate but an internal PKI signed certificate. The ertificate authority is already trusted by the netbox server. From the server I can git clone successfully and I can also git clone from the venv.

The only "git clone" that does not worked is from the Data source feature. I was thinking about trust the certificate authority inside the certifi store but I am not confident on how to do it. Can you confirm it is the good way to resolve this issue ?

But I think by default the Data source features should trust the system certificate store.

[candlerb]

The ertificate authority is already trusted by the netbox server.

Apparently not by Python though. Perhaps it needs adding to the certifi store, or made available to Python some other way.

On my system I see that /opt/netbox/venv/lib/python3.8/site-packages/certifi/cacert.pem exists with all the root certs concatenated, and you could try adding your cert to that file as a test.

However, according to https://pypi.org/project/certifi/ :

Addition/Removal of Certificates

Certifi does not support any addition/removal or other modification of the CA trust store content. This project is intended to provide a reliable and highly portable root of trust to python deployments. Look to upstream projects for methods to use alternate trust.

[sonicfather]

I added all my certificate chain (Root + sub CA) into certifi trust store, I got the same error.

[sonicfather]

Does anyone have an idea how to fix this problem?

I still do not know if I am doing somthing wrong or if it is just not possible in this netbox release.

[ChrisdAutume]

Hi There,

porcelain lib read the sslCAInfo in the gitconfig of the user to operate.

So you should use it to set your custum bundle CA:
git config http."https://xxxxx.fr/".sslCAInfo /my/path/cacert.pem

[hhopjh]

I believe this command should be executed from inside particular git repo. Correct? Where git repositories are located on Netbox internal disk?

[Wellyas]

Hi,
I encounter the same issue with a self hosted gitlab server sign by an internal CA.
I think a way to resolve it, will be to add an other parameter for git backend, a boolean for skipping TLS verify. Base on that parameter we could add to the porcelain config dict the key http.sslVerify to match this parameter.

[candlerb]

I think a way to resolve it, will be to add an other parameter for git backend, a boolean for skipping TLS verify.

A more secure option would be to be able to specify the CA cert - which would be the server's own cert, if it's self-signed.

e.g. for the requests library, you can either specify verify="/path/to/cert.pem" or verify=False

It would also be good to log if this is in fact the error.

[Wellyas]

You can do the same with git config httpsslCAinfo.
The issue is how you can configure this with netbox. Because netbox use porcelain library to make git call. And each repository server have a specific git config.

in the data_backends.py we use this method to configure a proxy.

So I think it will be nice if there is a way to add this config when we configure the data source.

If you want I can propose a PR for this.

I try to edit the data_backends to allow the TLS insecure and there is a warning log. it is not catch as standard warning but we can find it in the worker log.

Regards

[hendrikbl]

It would be nice to have the option to configure the porcelain git config when setting up the git data source.

Maintaining a custom version of data_backends.py will we very tedious, especially when using netbox with docker containers.

[teixemf]

I am also facing this problem and opened the feature proposal Data Sources | Git | Allow to select customized CA Cert file for HTTPS URLs #13989

[stanislav-zaprudskiy]

As under the hood dulwich with urllib3 (and without certifi) is used, one could tell it use a "proper" CA store using SSL_CERT_FILE environment variable.

ENV SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt

[FloMeyer]

Hi @stanislav-zaprudskiy can you tell where to put this string?
I tried with /etc/environment with content: "SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt" But this is not working for netbox (no docker).

[markkuleinio]

My try would be to add

Environment="SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt"

inside the [Service] section in /etc/systemd/system/netbox-rq.service.

Based on this idea: #15394 (comment)

[FloMeyer]

Thank you @markkuleinio! This is working. You made my day! :)

[ITTV-tools]

Unfortunatly this does not work for me.
I even tried to add the file in the Dockerfile:

COPY zertifikatskette.crt /usr/local/share/ca-certificates
RUN update-ca-certificates

An then I used the docker-override to add the Variable:

  netbox-worker:
    image: netbox:latest-plugins
    environment:
      SSL_CERT_FILE: /etc/ssl/certs/ca-certificates.crt

Still a ssl error.
I tried to open the page with curl inside of the container. That is working without issues

[fknorn]

So, was this ever resolved? I'm facing the same situation where my data source refuses to work because of certificate verification issues.