Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failing to get BG from railway/nightscout #1446

Open
rossens opened this issue Jan 15, 2023 · 11 comments
Open

Failing to get BG from railway/nightscout #1446

rossens opened this issue Jan 15, 2023 · 11 comments

Comments

@rossens
Copy link

rossens commented Jan 15, 2023

Describe the bug
OpenAPS rig no longer able to loop since migrating Nightscout from Heroku to Railway.

The rig is unable to get BG data from railway/nightscout (however it is able to send pump data to nightscout just fine).

/var/log/openaps/pump-loop.log: "Waiting up to 4 minutes for new BG.........." (but never gets BG)
/var/log/openaps/ns-loop.log: "Loading CGM data from Nightscout failed {"code":"CERT_HAS_EXPIRED"}

BG url: https://melaniejellis.up.railway.app/api/v1/entries/sgv.json?count=288&token=

Note: From Edison bash console I can get the data no problem using curl. Only seems to be an issue from node js
node js version: nodejs/unknown,now 8.17.0-1nodesource1 i386 [installed]

Note: railway uses "Lets Encrypt" for certification. Is node version too old to support "Lets Encrypt"? (I was reluctant to try upgrading node js in case it breaks something else.)

To Reproduce
Steps to reproduce the behavior:

  1. Migrate nightscout from heroku to railway: https://nightscout.github.io/vendors/railway/migration/?fbclid=IwAR3PVqKdfU7rVO8n41LHfi83FgupH7ZO8rHvZs_ECIinnuYZZ5CJoKKaBRk
  2. Check if rig is still looping

Expected behavior
Rig should continue looping after upgrading nightscout from heroku to railway

Smartphone:
model: Samsung S21
xdrip: 344deb4-2022.11.16

Setup Information:

  • Pump type: Medtronic 722, v. 2.4A firmware
  • CGM type: "xDrip-DexcomG6" with anubis transmitter
  • Rig type: Edison/Explorer Board rig
  • Linux version: 3.10.98-jubilinux-edison (robin@robin-i7) (gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.3) ) #3 SMP PREEMPT Sun Aug 13 04:22:45 EDT 2017
  • oref0 version: 0.7.1-dev (we updated from github to latest dev on 14/01/2023)

Additional context
I'm creating this bug report on behalf of @melaniejellis - who I've been helping get her openAPS rig looping again.
I'm not completely familiar with the openAPS system so apologies if this turns out this is not be a bug (but rather something we've missed in the setup.)

I implemented a temporary work around to get melanie looping again... which was to add the following lines to /usr/bin/oref0-get-ns-entries (https://github.com/openaps/oref0/blob/dev/bin/oref0-get-ns-entries.js). This allows the ns script to get the BG data, ignoring the certificate issue:

    headers["User-Agent"] = 'openaps';
    var uri = nsurl + '/api/v1/entries/sgv.json?count=' + records + tokenAuth;
    var options = {
      uri: uri
      , json: true
      , timeout: 90000
      , headers: headers
      , gzip : true
+     , agentOptions: {
+         rejectUnauthorized: false
+       }
    };

Look forward to hearing (if this is a real bug) what is the proper fix.

@scottleibrand
Copy link
Contributor

According to ChatGPT, The main difference between certificate validation in curl and Node.js is that curl uses the operating system's root CA store, while Node.js uses its own copy of the root CA store, which is managed by the Node.js tls module.

Our working rigs are using Linux edison-eb 3.10.98-jubilinux-edison #3 SMP PREEMPT Sun Aug 13 04:22:45 EDT 2017 i686 GNU/Linux, which looks the same as yours. For node we have:

# nodejs -v
v8.17.0
# node -v
v9.11.2

Do you have a second Edison available that you could do a fresh flash and OpenAPS install on? I suspect that would fix the issue.

@sulkaharo
Copy link
Collaborator

Wonder what Node version is the latest that runs comfortably on the Edison. The chip is an x86 one, so theoretically even the latest LTS releases should work.

@melaniejellis
Copy link

Hi, thanks for your reply, I do have a few back ups. Ill do the flash and openAPS and report back.

@melaniejellis
Copy link

I've got :

nodejs -v

v8.17.0

node -v

v8.17.0

@sulkaharo
Copy link
Collaborator

sulkaharo commented Jan 16, 2023

Hmm I just tested loading the data and I don't think the issue is with Node. My browsers also fail to connect to the site and testing this with wget, the utility reports the certificate that's published doesn't match the domain name, which is a hard error and blocks SSL from connecting:

--2023-01-16 12:08:35--  https://melaniejellis.up.railway.app/
Resolving melaniejellis.up.railway.app (melaniejellis.up.railway.app)... 52.166.12.23
Connecting to melaniejellis.up.railway.app (melaniejellis.up.railway.app)|52.166.12.23|:443... connected.
ERROR: no certificate subject alternative name matches
	requested host name ‘melaniejellis.up.railway.app’.
To connect to melaniejellis.up.railway.app insecurely, use `--no-check-certificate'.

@rossens
Copy link
Author

rossens commented Jan 16, 2023

Hmm yeah agreed something strange is going on then.
I just checked myself accessing https://melaniejellis.up.railway.app/ from chrome (windows 10) and had no issues.
Also tried wget on a linux box, which also worked:

$ wget https://melaniejellis.up.railway.app
--2023-01-16 18:35:48--  https://melaniejellis.up.railway.app/
Resolving melaniejellis.up.railway.app (melaniejellis.up.railway.app)... 104.196.232.237
Connecting to melaniejellis.up.railway.app (melaniejellis.up.railway.app)|104.196.232.237|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 42973 (42K) [text/html]
Saving to: ‘index.html’

index.html          100%[===================>]  41.97K   263KB/s    in 0.2s

2023-01-16 18:35:49 (263 KB/s) - ‘index.html’ saved [42973/42973]

Notice a different ip address though... wondering if that changed over night (legit), or there is something odd going on with 2 versions of the same site or something?

@sulkaharo
Copy link
Collaborator

Right, so looks like Railway.app is very widely blacklisted in a variety of software intended for cybersecurity purposes. So if you're having issues loading data from Railway, please check the network you're using is not blocking Railway. I just discovered the corporate security solution my work has in place was blocking 100% of Railway.

@rossens
Copy link
Author

rossens commented Jan 17, 2023

Same issue on a different rig:

root@meleAPS:~# node /usr/bin/oref0-get-ns-entries ns-glucose.json https://melaniejellis.up.railway.app xxx

Connecting to 192.168.1.1, testing for xDrip API availability
Load from local xDrip timed out, likely not connected to xDrip hotspot
{"errno":-2,"code":"ENOENT","syscall":"open","path":"/root/ns-glucose.json"}
Loading CGM data from Nightscout failed {"code":"CERT_HAS_EXPIRED"}

Whereas curl, wget etc work fine.
Versions:

root@meleAPS:~# node -v
v8.17.0
root@meleAPS:~# nodejs -v
v8.17.0
root@meleAPS:~# oref0-version
0.7.0 [dev]
root@meleAPS:~# cd /root/src/oref0/ && git status
On branch dev
Your branch is up-to-date with 'origin/dev'.
nothing to commit, working tree clean

Note: I attempted re-flashing the rig as well - but failed miserably. The flashall.bat never detected the edison USB OTG device. (I noticed the OTG seemed flaky - it would constantly connect/disconnect in device manager every few seconds).

@rossens
Copy link
Author

rossens commented Jan 21, 2023

Update on this issue:
I use a simpler script to rule out as much other noise as possible:

#!/usr/bin/env node
'use strict';

const https = require('https');
https.get('https://melaniejellis.up.railway.app/api/v1/entries/sgv.json?count=1&token=xxx', (resp) => {
  let data = '';
  resp.on('data', (chunk) => { data += chunk; });
  resp.on('end', () => { console.log("Success: " + data); });

}).on("error", (err) => { console.log("Error: " + err.message); });

I tested this works on Ubuntu 22.04 (node v12.22.9)
Success: [{"_id":"...<trimmed>...}]
I tested this does not work on Edison openAPS rig (node v8.17.0)
Error: certificate has expired
I also tried updating node on Edison to "latest" (v9.11.2) - but this version also not working.

I had just about run out of ideas when I found this: node-fetch/node-fetch#568
Apparently these guys had encountered the identical looking issue with their node-fetch module vs Lets Encrypt certificates, for one user it was even also on a Heroku app.
They used the same temporary workaround as I suggested (rejectUnauthorized: false), and have since made a proper fix: "They merged https://github.com/electron/electron/pull/31218/files" which patches boringssl

I'm not yet sure if/how this same fix would be applicable in oref0 (does it use boring ssl?) - but passing on my find in the hope you guys may know.
Any thoughts?

@sulkaharo
Copy link
Collaborator

Sounds like the easy quick to deploy fix is to not host Nightscout in a service that uses Let's Encrypt certificates. Edisons haven't been supported for years in terms of software (by vendors) and if we can't get a newer Node version to run, there's relatively little we can do. Disabling the certificate checks is technically possible but sounds like something that shouldn't be done in the mainline version of openaps.

@rossens
Copy link
Author

rossens commented Jan 23, 2023

Could another option be to avoid node for the https work? e.g. I tried this which works:

#!/usr/bin/env node

const { exec } = require('child_process');
exec('curl https://melaniejellis.up.railway.app/api/v1/entries/sgv.json?count=1&token=xxx',
(error, stdout, stderr) => {
  console.log(stdout);
  if (error !== null) {
      console.log(`exec error: ${error}`);
  }
});

Interestingly when I first tried this it failed with the same certificate error - and I realised both curl & wget had the same issue as node. But this was an older Edison which hadn't been apt upgraded in a while. After an apt upgrade - curl & wget were both fixed (but node still broken).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants