feat: File row splits #1709
feat: File row splits #1709
Conversation
| @@ -26,11 +26,11 @@ pub(crate) trait ReadMasked { | |||
|
|
|||
| /// Read an array with a [`RowMask`]. | |||
| pub(crate) struct ReadArray { | |||
| layout: Box<dyn LayoutReader>, | |||
| layout: Arc<dyn LayoutReader>, | |||
There was a problem hiding this comment.
So LayoutReaders are no longer stateful?
There was a problem hiding this comment.
Seems like I overlooked some layers of of state/locks in the readers, that will take some thinking and probably additional refactoring.
There was a problem hiding this comment.
ok after a more careful reading and a small test, as far as I can tell while the LayoutReader implementations are stateful (with interior mutability), they can read independently from each other as the LayoutBufferedReader is the part that actually controls the overall flow and each stream initializes a new one.
There was a problem hiding this comment.
There can be multiple concurrent reads being tracked in the layout reader but are they reusable?
I thought something encoded assumptions that splits were accessed monotonically but maybe that was in the BufferedReader
There was a problem hiding this comment.
I thought so too, but that does seem to be in the BufferedLayoutReader. The LayoutReader implementation read through poll_read which takes a RowMask to read try and read a specific range on each call, and the BufferedReader tracks that.
There was a problem hiding this comment.
You should be able to access the layoutreaders out of order so we can read multiple at the same time irrespective of their children.
|
|
||
| // Set up a stream of RowMask that result from applying a filter expression over the file. | ||
| let mask_iterator = if let Some(fr) = &self.filter_reader { | ||
| Box::new(BufferedLayoutReader::new( |
There was a problem hiding this comment.
Ideally we pull the splits_stream out of this thing and just stream::iter(self.splits().iter()).map(|s| self.read_range(s).boxed()).buffered(64) or whatever?
Or is that not possible?
There was a problem hiding this comment.
Honestly worried about using buffered everywhere
There was a problem hiding this comment.
Buffered is a bit too naive. You want more control over coalescing
Introduces a new
VortexReadHandlethat can be used cloned and used to either generate a full stream over the underlying file OR read a specific row range.The handle also implements
Cloneso it can be re-used over the same file.DataFusion support will be in a followup PR.