Document DataFusion Threading / tokio runtimes (how to separate IO and CPU bound work)

[tustvold]

Is your feature request related to a problem or challenge?

DataFusion performs CPU bound work within async closures. This causes issues if running IO on the same async runtime, as the cooperative nature of such schedulers allows the CPU bound work to starve servicing of IO. This leads to errors such as apache/arrow-rs#5882.

Describe the solution you'd like

I think at the very least this needs to be better documented, I couldn't find any mention of this in the DataFusion documentation following a cursory search.

I also think more holistic approach would be valuable to this, as it stands the use of async within DataFusion acts as a massive footgun that encourages users to intermix IO and CPU work in a way that is at best inefficient, but this can be tracked as a separate follow on task.

Describe alternatives you've considered

No response

Additional context

No response

[alamb]

I recommend two things:

1. Write a blog with background and explanation of why using two threadpools is important with DataFusion and examples of how to do it
2. Add additional documentation (ideally linking to the blog) with a summary and linking to the blog with content.

[ozankabak]

I think it'd be great to have a good documentation on this.

[alamb]

I think it'd be great to have a good documentation on this.

100% agree -- @itsjunetime and @tustvold are working on a bit of it in apache/arrow-rs#6612. I'll try and help with the documentation as well

[alamb]

Documentation

• Improve documentation (and ASCII art) about streaming execution, and thread pools #13423

I hope to work on the example a bit more shortly

[adriangb]

I was thinking, beyond examples, could we include a basic implementation in DataFusion? I'm not saying it's wired up by default, but put it behind a feature flag and make it provisional but ship it with the source code? Maybe that + docs on how to use it is enough for a lot of use cases?

I think we have a working version of this in 2 files and not that many LOC. We'd be happy to donate it.

[alamb]

I was thinking, beyond examples, could we include a basic implementation in DataFusion? I'm not saying it's wired up by default, but put it behind a feature flag and make it provisional but ship it with the source code? Maybe that + docs on how to use it is enough for a lot of use cases?

Yes I agree this would be ideal. Thank you @adriangb

I think we have a working version of this in 2 files and not that many LOC. We'd be happy to donate it.

That would be awesome -- thank you! I can potentially adapt my example to use what you have

Any chance you can make a PR?

[adriangb]

@alamb #13634

[alamb]

Update here:

• I have an example here: Add example for using a separate threadpool for CPU bound work #13424
• @adriangb had an example in add cross rt execution code #13634 that takes a similar approach
• I tried to implement approach suggested by @tustvold on Alternate example of using two thread pools to run DataFusion IO and CPU operations #13690 but a not (yet) convinced it is feasible to implement