Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modify some calls to allow for FIFO support #17

Draft
wants to merge 1 commit into
base: v2.3-branch
Choose a base branch
from

Conversation

David-McKenna
Copy link
Contributor

Hey Dave,

We've been looking to use FIFOs as an input to rawspec at the LOFAR stations given we're performing a packet-to-GUPPI raw conversion, so preventing a write to disk would save a good chunk of time while processing data.

Due to the use of seeks (via lseek) in rawspec, it can't handle FIFOs as an input in the current version. This PR introduces an alternative code path where rawspec will detect if the input is a FIFO or PIPE, and now either

  • Read only the data required from FIFOs, rather than reading and reverse seeking (in rawspec_raw_read_header)
  • Read data from FIFOs into a discard buffer to seek into the future (for handling schan, in main)

To achieve the latter, I've created a new function, read_to_skip based on calls to read_fully, this has an 8k buffer that it repeatedly reads data into until the requested data has been consumed. I tried benchmarking a few options for the output array (the array of constant length, a variable length array and a malloc'd array) and found the fixed 8k array was the best option (VLA was faster in some cases, but segfaulted for larger read lengths).

I've tried to follow your coding style and approach while implementing this, but let me know if you want anything changed or if there's way you feel I could improve the implementation.

Cheers,
David

@David-McKenna David-McKenna changed the base branch from master to v2.3-branch July 7, 2021 10:29
@texadactyl texadactyl marked this pull request as draft November 5, 2021 18:43
@texadactyl
Copy link
Contributor

@David-McKenna
This PR is pending your analysis of the hashpipe technique - direct call vs using a pipe.

cc: @david-macmahon

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants