Timestamps are sometimes negative or nonsensical

[MarinManuel]

Hi,
I have been investigating the issue that I originally raised in this issue in open-ephys-python-tools. Unfortunately, I have not been able to isolate exactly what causes the issue, but I wrote a little script that generate a bunch of sessions/experiments/recordings in OpenEphys GUI and I can consistently get this weird behavior. Hopefully this might help track down the problem.

The issue can be summarized like this:
I record Neuropixels data and other electrophysiological data in Open Ephys GUI. to synchronize both streams, I have a pulser box that generate random pulses that are sent to both the NPX card (line 1) and the DAQ card (line 8). I have a merger in my signal chain and I have set up the synchronization in the record node.

sometimes, but not always, the timestamps associated with the data are just plain wrong. Sometimes, all the timestamps are negative, but they still increment regularly, and I would argue that as long as they are the same across both streams, it doesn't matter what their actual absolute value is.
On the other hand, though, sometimes the timestamps are just nonsensical.

Consider Trial 1 (data available here)
the script trial_run.py generates 6 sessions (6 recording folders) each containing 2 experiments, each containing 2 recordings.

The recordings in Session 1 experiment 1 are fine:

the two waveform are recordings from the first channel of each of the streams. The vertical dashed line are stimulation time extracted from events sent to line 1 of the DAQ.

However, recordings in experiments 2 have negative timestamps, although the structure of the recording is preserved

I can work with this, but the fact that the timestamps come up negative is probably a symptom of the underlying issue.

More troubling though, is that some of the recording have completely nonsensical timestamps, such as this one, taken from the 3rd session, experiment 1, recording 1

Curiously, Session 3/experiment 2/recording 1 is back to (almost) normal

but other subsequent recordings show the same nonsensical timestamps.

I have run a different trial using the exact same configuration (after closing and reopening the GUI, if that matters), and I see the same pattern, but not exactly the same files are affected. The data is available here.

I have also run a trial with just the NPX stream, and no merging and no synchronization, and all the files are correct then. (Data available here).

[MarinManuel]

Here is a plot of the timestamps as a function of the sample_numbers for the first recording

while this is the same plot for the first nonsensical recording shown above (Trial 1/Session 3/experiment 1/recording 1)

I have tried to recompute the synchronization of the timestamps using open-ephys-python-tools (as per these instructions).
Although it restores a somewhat plausible timebase, the two streams are not synchronized:

rec.add_sync_line(line=1, processor_id=100, stream_name='ProbeA', main=False)
rec.add_sync_line(line=8, processor_id=102, stream_name='PXIe-6341', main=False)

rec.add_sync_line(line=1, processor_id=100, stream_name='ProbeA', main=True)
rec.add_sync_line(line=8, processor_id=102, stream_name='PXIe-6341', main=False)
rec.compute_global_timestamps(overwrite=True)

[MarinManuel]

@medengineer
I encountered this issue again yesterday doing a real experiment. I am really keen on fixing this as it is causing some real headaches to analyze the data.

Am I the only one experiencing this issue? I am sure it would have come up before?
Anyway, let me know how I can assist find the root of the problem. I am able to recompile the GUI if needed to enable some debug messages or anything that could be useful.

Thanks!

[medengineer]

Hi @MarinManuel

The offline misalignment is due to an extra scaling factor getting applied when computing the continuous global timestamps. Removing the extra division by sync_line[‘scaling’] here gives the proper alignment offline.

I’m replicating your exact setup locally now to investigate online synchronization, will follow up here as soon as I find anything.

Thanks for reporting this!

[medengineer]

Re: online synchronization, the current algorithm is not robust to random/short pulses. If you can switch from using random pulses to a stable 1Hz square wave that should solve online synchronization without changing the GUI code.

We are working on overhauling the online algorithm to support arbitrary sync signals for the next major GUI release.

[MarinManuel]

Thanks I will switch to fixed 1Hz pulses and report back. I just want to point out that the documentation specifically said

For that reason, the ideal signal is one that has an average frequency of around 1 Hz, but randomized intervals between the pulses. Such a signal can be generated by an Arduino or other programmable hardware device capable of emitting 5V digital signals.

[medengineer]

To clarify, random intervals between rising edges are currently supported online. More important is ensuring pulse widths between rising and falling edges are sufficiently long. I believe that restriction should be updated to 200ms instead of the current 100ms in that same section of documentation.

[MarinManuel]

Thanks.
I have tried both fixed 1Hz pulses (from the NPX basestation) and 250ms-long pulses with random interpulses intervals. This fixes the nonsensical timestamps, but I still get some files where the timestamps are negative.

[MarinManuel]

Hi @MarinManuel

The offline misalignment is due to an extra scaling factor getting applied when computing the continuous global timestamps. Removing the extra division by sync_line[‘scaling’] here gives the proper alignment offline.

I’m replicating your exact setup locally now to investigate online synchronization, will follow up here as soon as I find anything.

Thanks for reporting this!

this seems to work well to fix the files offline