MiniMed 715 history parsing fails

[sulkaharo]

I'm trying to get a 715 working with OpenAPS and it's clear the decocare history page parsing doesn't actually support this pump. Some of the events do parse correctly but not all. The two stack traces I've seen so far are:

settings/pumphistory-24h.json  raised  month must be in 1..12
Traceback (most recent call last):
  File "/usr/local/bin/openaps-report", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/home/edison/src/openaps/bin/openaps-report", line 82, in <module>
    app( )
  File "/home/edison/src/openaps/openaps/cli/__init__.py", line 51, in __call__
    self.run(self.args)
  File "/home/edison/src/openaps/bin/openaps-report", line 75, in run
    output = app(args, self)
  File "/home/edison/src/openaps/openaps/cli/subcommand.py", line 52, in __call__
    return self.method.main(args, app)
  File "/home/edison/src/openaps/openaps/reports/invoke.py", line 40, in main
    output = task.method(args, app)
  File "/home/edison/src/openaps/openaps/uses/use.py", line 45, in __call__
    output = self.main(args, app)
  File "/home/edison/src/openaps/openaps/vendors/medtronic.py", line 639, in main
    for rec in self.range( ):
  File "/usr/local/lib/python2.7/dist-packages/decocare/models/__init__.py", line 67, in __call__
    for record in page:
  File "/usr/local/lib/python2.7/dist-packages/decocare/models/__init__.py", line 46, in download_page
    for record in self.find_records(page):
  File "/usr/local/lib/python2.7/dist-packages/decocare/models/__init__.py", line 158, in find_records
    records = decoder.decode( )
  File "/usr/local/lib/python2.7/dist-packages/decocare/history.py", line 851, in decode
    record = parse_record(self.stream, B, larger=larger, model=self.model)
  File "/usr/local/lib/python2.7/dist-packages/decocare/history.py", line 782, in parse_record
    record.parse( head + date + body )
  File "/usr/local/lib/python2.7/dist-packages/decocare/records/base.py", line 66, in parse
    return self.decode( )
  File "/usr/local/lib/python2.7/dist-packages/decocare/history.py", line 73, in decode
    return (dict(valid_date=date(*mid).isoformat()))
ValueError: month must be in 1..12

and

Traceback (most recent call last):
  File "/usr/local/bin/openaps-report", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/home/edison/src/openaps/bin/openaps-report", line 82, in <module>
    app( )
  File "/home/edison/src/openaps/openaps/cli/__init__.py", line 51, in __call__
    self.run(self.args)
  File "/home/edison/src/openaps/bin/openaps-report", line 75, in run
    output = app(args, self)
  File "/home/edison/src/openaps/openaps/cli/subcommand.py", line 52, in __call__
    return self.method.main(args, app)
  File "/home/edison/src/openaps/openaps/reports/invoke.py", line 40, in main
    output = task.method(args, app)
  File "/home/edison/src/openaps/openaps/uses/use.py", line 45, in __call__
    output = self.main(args, app)
  File "/home/edison/src/openaps/openaps/vendors/medtronic.py", line 639, in main
    for rec in self.range( ):
  File "/home/edison/src/decocare/decocare/models/__init__.py", line 67, in __call__
    for record in page:
  File "/home/edison/src/decocare/decocare/models/__init__.py", line 46, in download_page
    for record in self.find_records(page):
  File "/home/edison/src/decocare/decocare/models/__init__.py", line 158, in find_records
    records = decoder.decode( )
  File "/home/edison/src/decocare/decocare/history.py", line 855, in decode
    record = parse_record(self.stream, B, larger=larger, model=self.model)
  File "/home/edison/src/decocare/decocare/history.py", line 786, in parse_record
    record.parse( head + date + body )
  File "/home/edison/src/decocare/decocare/records/base.py", line 66, in parse
    return self.decode( )
  File "/home/edison/src/decocare/decocare/history.py", line 598, in decode
    temp = { 0: 'absolute', 1: 'percent' }[self.head[1]]
KeyError: 81

I logged some of the packets and the date error seems to be caused by the date parser being handled bytes that cannot possibly be a date, resulting in the parsing completely failing or the date being bogus (year deciphered anywhere between 1970 to 2056).

[sulkaharo]

Things that would help:

• Point out how to log the full history page bytes on an Explorer and re-run parsing with fixed code so we don't need to test against a moving target (the pump memory)
• Confirm if anyone is actually using a 715 successfully with OpenAPS - I suspect not

[sulkaharo]

Have tried filling the pump memory with known to work events and that seems to result in the pump working for a few hours, after which some event appears that breaks the event parsing fully. Another observed symptom is, sometimes an event is parsed in a manner where fetching the history only produces partial history, causing IOB to be calculated wrong. I'm observing the latter is due to the date parsing fail, where the parser stops returning events after hitting an event that's mis-parsed as too old.

[kenstack]

@sulkaharo we had a fix for some date parser issues a few months ago - what version of decocare are you using? Just making sure its not a version problem. Ill try to check later that the change made it into master too

make sure your mm-decode file has this after the datetime import - that helped me tremendously on time parsing issues

import datetime
json.JSONEncoder.default = lambda self,obj: (obj.isoformat() if isinstance(obj, datetime.datetime) else None)

the other thing that is possible to do - but a bit risky - is to trap errors in mm-decode and still spit out the rest that do decode - problem is you could potentially miss one ...

[sulkaharo]

@kenstack Latest master from this repo.

Another thing that could help is porting the data dump tools in decocare to work on radios newer than the CareLink. Realized I currently only have working computers with USB 3 ports, which are incompatible with the old USB 1 CareLink stick.

[kenstack]

@sulkaharo sorry added a comment above :)

[kenstack]

@sulkaharo I did the PR for json.JSONEncoder.default = lambda self,obj: (obj.isoformat() if isinstance(obj, datetime.datetime) else None) - could it have broken things on the 715? Perhaps comment it out and try it again - I tested with a 723 and a 722 not a 715 .... I doubt it but something to try

[sulkaharo]

Will try, thanks! I also found a spare Pi & CareLink stick, will try to run Ben's data gathering status_quo.sh on the problematic pump tomorrow.

[sulkaharo]

Ok read through the decocare documentation with more thought and found the instructions how to dump the memory into files, which are now here: https://github.com/sulkaharo/715debugging

Based on trying to run the decoder on the files, some files do produce errors. The nice this is, this is now reproducible. :)

[sulkaharo]

I used list_history.py and list_opcodes.py to process the files and put the results to the repo. Looking at the output, many of the files parse ok (or at least produce some results), while 20 of the 37 pages immediately terminate the processing with MISSING DATETIME error. This is an old pump that's been in use for years, so I doubt any of the pages are actually empty.

[sulkaharo]

@kenstack mm-decode-history-page.py has the line and it doesn't make a difference to the decoding whether it's there or commented out.

The two errors that appear during decoding are

Traceback (most recent call last):
  File "../../decoding-carelink/list_history.py", line 128, in <module>
    main( )
  File "../../decoding-carelink/list_history.py", line 107, in main
    records = find_records(stream, opts)
  File "../../decoding-carelink/list_history.py", line 86, in find_records
    record = parse_record( stream, B, larger=opts.larger )
  File "/Users/sulka/Documents/nightscout/decoding-carelink/decocare/history.py", line 774, in parse_record
    record  = suggest(head, larger, model=model)
  File "/Users/sulka/Documents/nightscout/decoding-carelink/decocare/history.py", line 761, in suggest
    record = klass(head, model)
  File "/Users/sulka/Documents/nightscout/decoding-carelink/decocare/records/bolus.py", line 88, in __init__
    self.MMOL_DEFAULT = model.MMOL_DEFAULT
AttributeError: 'NoneType' object has no attribute 'MMOL_DEFAULT'

Traceback (most recent call last):
  File "../../decoding-carelink/list_history.py", line 128, in <module>
    main( )
  File "../../decoding-carelink/list_history.py", line 107, in main
    records = find_records(stream, opts)
  File "../../decoding-carelink/list_history.py", line 86, in find_records
    record = parse_record( stream, B, larger=opts.larger )
  File "/Users/sulka/Documents/nightscout/decoding-carelink/decocare/history.py", line 774, in parse_record
    record  = suggest(head, larger, model=model)
  File "/Users/sulka/Documents/nightscout/decoding-carelink/decocare/history.py", line 761, in suggest
    record = klass(head, model)
  File "/Users/sulka/Documents/nightscout/decoding-carelink/decocare/history.py", line 673, in __init__
    self.body_length = model.old6cBody + 3
AttributeError: 'NoneType' object has no attribute 'old6cBody'

[sulkaharo]

If @bewest feels like looking at the data files mentioned above, that'd be amazing.

[jmcrawford]

I'm not sure about this, but it seems each page stores a different set of records. I've done some testing, I get this error on any page above 6, using "openaps use pump read_history_data #":

Traceback (most recent call last):
File "/usr/local/bin/openaps-use", line 63, in
app( )
File "/usr/local/lib/python2.7/dist-packages/openaps/cli/init.py", line 51, in call
self.run(self.args)
File "/usr/local/bin/openaps-use", line 57, in run
output = app(args, self)
File "/usr/local/lib/python2.7/dist-packages/openaps/uses/init.py", line 92, in call
return self.method.selected(args)(args, app)
File "/usr/local/lib/python2.7/dist-packages/openaps/uses/init.py", line 31, in call
return self.method(args, app)
File "/usr/local/lib/python2.7/dist-packages/openaps/uses/use.py", line 45, in call
output = self.main(args, app)
File "/usr/local/lib/python2.7/dist-packages/openaps/vendors/medtronic.py", line 578, in main
history = self.pump.model.read_history_data(**self.get_params(args))
File "/usr/local/lib/python2.7/dist-packages/decocare/models/init.py", line 27, in call
return types.MethodType(self.func, inst)(self.response)
File "/usr/local/lib/python2.7/dist-packages/decocare/models/init.py", line 166, in read_history_data
decoder = history.HistoryPage(response.data, self)
File "/usr/local/lib/python2.7/dist-packages/decocare/history.py", line 778, in init
assert lib.BangInt(crc) == computed, "CRC does not match page data"
AssertionError: CRC does not match page data

[sulkaharo]

Small update: looks like official Medtronic software can read this pump without glitches, which indicates the issue is with Decocare.