[python-package] add 'pandas' extra

[jameslamb]

While working on #5936, I discovered that lightgbm is incompatible with pandas<0.24.0. This PR proposes adding a [pandas] extra to the Python package to track that floor, so users can run pip install 'lightgbm[pandas]' to be guaranteed to either get a compatible version or a big, loud, clear error.

Other changes:

• moves configuration for extras to pyproject.toml
  • [python-package] [ci] switch to PEP 517 / 518 builds (remove setup.py) (fixes #5061) #5759 broke pip install 'lightgbm[dask]', sorry 😬
• moves docs for the extras up under "Install from PyPI" section in the docs
• consolidates the three dask dependencies into one

How I found that version floor

details (click me)

lightgbm can only be used with pandas if CategoricalDType can be imported...

LightGBM/python-package/lightgbm/compat.py

Lines 6 to 17 in ac57d5a

	"""pandas"""
	try:
	from pandas import DataFrame as pd_DataFrame
	from pandas import Series as pd_Series
	from pandas import concat
	try:
	from pandas import CategoricalDtype as pd_CategoricalDtype
	except ImportError:
	from pandas.api.types import CategoricalDtype as pd_CategoricalDtype
	PANDAS_INSTALLED = True
	except ImportError:
	PANDAS_INSTALLED = False

... which was introduced in pandas==0.21.0: pandas-dev/pandas#16015.

lightgbm also relies on pd.DataFrame.to_numpy() and pd.Series.to_numpy()...

LightGBM/python-package/lightgbm/basic.py

Line 704 in ac57d5a

data = data.to_numpy(dtype=target_dtype, copy=False)

LightGBM/python-package/lightgbm/basic.py

Line 711 in ac57d5a

data = data.to_numpy(dtype=target_dtype, na_value=np.nan)

LightGBM/python-package/lightgbm/basic.py

Line 2594 in ac57d5a

label = label.to_numpy(dtype=np.float32, copy=False)

LightGBM/python-package/lightgbm/basic.py

Line 2601 in ac57d5a

label = label.to_numpy(dtype=np.float32, na_value=np.nan)

LightGBM/python-package/lightgbm/basic.py

Lines 2964 to 2965 in ac57d5a

	elif isinstance(other.data, pd_DataFrame):
	self.data = dt_DataTable(np.hstack((self.data.to_numpy(), other.data.values)))

... which was introduced in pandas==0.24.0: pandas-dev/pandas#23623

How I discovered that extras were broken (and then tested that this fixes them)

details (click me)

# build the wheel
sh ./build-python.sh bdist_wheel

# try to install
pip install \
    --find-links=./dist \
    'lightgbm[pandas]'

On master:

WARNING: lightgbm 3.3.5.99 does not provide the extra 'pandas'

On this branch:

Looking in links: ./dist
Processing ./dist/lightgbm-3.3.5.99-py3-none-macosx_12_0_x86_64.whl
Collecting pandas>=0.24.0
  Downloading pandas-2.0.2-cp39-cp39-macosx_10_9_x86_64.whl (11.8 MB)
...

[jameslamb]

The format where these are split out as individual items

dask[array]>=2.0.0
dask[dataframe]>=2.0.0
dask[distributed]>=2.0.0

Is confusing. It makes it seem like 3 separate versions of 3 separate things, which just happen to be the same... but that's not really what it means.

That is 3 different ways to say dask >= 2.0.0.

We should just use a single number, to make that clearer.

[github-actions]

This pull request has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.