Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor VortexFileWriter #1676

Open
6 tasks
gatesn opened this issue Dec 13, 2024 · 0 comments
Open
6 tasks

Refactor VortexFileWriter #1676

gatesn opened this issue Dec 13, 2024 · 0 comments

Comments

@gatesn
Copy link
Contributor

gatesn commented Dec 13, 2024

Tracking a bunch of breaking changes to the format and refactoring of the writer

  • Consistent naming for buffers / messages. I'm thinking "blocks".
  • Ensure we don't write extra padding / framing, e.g. we sometimes use IPCMessage and MessageWriter for writing to the file, even though this adds framing for a streaming format.
  • VortexFileWriter to pass chunks into an abstract LayoutWriter.
  • Impl LayoutWriter for struct, chunked, flat, and later dict layouts.
  • Layout flat buffer to hold BlockIds instead of offset/length. This allows Layout to be abstract over the storage format and not assume linear bytes (e.g. can store in an arbitrary block device).
  • Postscript to point to DType + FileLayout and not assume contiguous bytes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant