New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Separate runtime for IO / CPU bound tasks #197

Closed

matthewmturner opened this issue Oct 14, 2024 · 0 comments · Fixed by #247

Collaborator

matthewmturner commented Oct 14, 2024

At least for the FlightSQL server implementation this would probably make sense. As usual, throw it as an option in the config.

matthewmturner mentioned this issue

[DISCUSSION] 2024 Q4 / 2025 Q1 Roadmap apache/datafusion#13274

Open

matthewmturner linked a pull request

that will close this issue

Add DedicatedExecutor to FlightSQL Server #247

Merged

matthewmturner mentioned this issue

Add DedicatedExecutor to FlightSQL Server #247

Merged

matthewmturner added a commit that referenced this issue


          Add DedicatedExecutor to FlightSQL Server (#247)

e8c8e20

Add's a dedicated executor for running CPU bound work on the FlightSQL
server.

There is interest from the [DataFusion
community](apache/datafusion#13274 (comment))
for this, it was already on our
[roadmap](#197)
and I think the DFT FlightSQL server is a great place to have a
reference implementation.

Initial inspiration and context can be found
[here](https://thenewstack.io/using-rustlangs-async-tokio-runtime-for-cpu-bound-tasks/).

Most of the initial implementation was copied from
[here](https://github.com/influxdata/influxdb3_core/blob/6fcbb004232738d55655f32f4ad2385523d10696/executor/src/lib.rs)
with some tweaks for our current setup. In particular we dont have
metrics yet in the FlightSQL server implementation (but it is on the
[roadmap](#210))
- I expect to do a follow on where metrics are integrated.

matthewmturner closed this as completed in #247

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment