[dask] hold ports until training

[jmoralez]

Contributes to #5865 by holding on to the ports that LightGBM will use during training for as long as possible on the Python side, thus decreasing the chance of a race condition (but not eliminating it).

[jmoralez]

Seems like the actor approach won't work here due to dask/distributed#7842. Feel free to try the alternative here @jameslamb.

EDIT:
I've tried a similar approach without using the actor interface directly, it isn't 100% clean but it does the job, let me know what you think. I still have some details to fix like some types and docstrings, but I'd like to get your opinion on the approach first.

[jameslamb]

Very nice, thanks so much for working on this! This is a much better version of what I was kind of envisioning in #5865 (comment).

I'm unsure about exactly how it works, but mainly because of my own ignorance of Dask and TCP. If you tell me this is working for you on both LocalCluster and a multi-machine distributed Dask cluster, then I think it's worth the time for you to add the remaining type hints, tests, docs, etc. and I'll commit to testing it myself on both LocalCluster and some different configurations of multi-machine clusters from Coiled or with something like dask-cloudprovider.

[jameslamb]

Based on #5890 (comment), I'm feeling pretty optimistic about this approach! Awesome work.

Some time this week, I'll try to test this on a multi-machine cluster using Coiled or dask-cloudprovider.

[jmoralez]

Seems like coiled doesn't provide free credits to use their infra anymore, so I haven't been able to test this on a real cluster. I'll try to find an alternative because I don't have an AWS account.

[jameslamb]

I can test on my AWS account.

[jameslamb]

I tested this tonight on AWS Elastic Container Service (ECS) cluster, using dask-cloudprovider: https://github.com/jameslamb/lightgbm-dask-testing/blob/main/notebooks/demo-aws.ipynb.

Happy to say it worked well!

I left one small comment, do what you want with it.

Great work, @jmoralez !!!

[jameslamb]

Looks great to me!

You should be the one to click the button, @jmoralez 😁

[adfea9c0]

Hey only saw this now, but thanks for looking into it !

[github-actions]

This pull request has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.