[BUG] Attribute error when importing DeepSpeed Engine

[Chris-hughes10]

Describe the bug
When I run the line from deepspeed import DeepSpeedEngine an attribute error is raised. Stacktrace provided below.

To Reproduce
Steps to reproduce the behavior:

1. In the terminal, run python
2. Run from deepspeed import DeepSpeedEngine
3. See error

Expected behavior
The import works successfully, or steps are given to resolve the error.

ds_report output

[2023-07-14 15:00:23,990] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/opt/conda/envs/core/lib/python3.10/site-packages/pydantic/_internal/_config.py:261: UserWarning: Valid config keys have changed in V2:
* 'allow_population_by_field_name' has been renamed to 'populate_by_name'
* 'validate_all' has been renamed to 'validate_default'
  warnings.warn(message, UserWarning)
/opt/conda/envs/core/lib/python3.10/site-packages/pydantic/_internal/_fields.py:126: UserWarning: Field "model_persistence_threshold" has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
  warnings.warn(
Traceback (most recent call last):
  File "/opt/conda/envs/core/bin/ds_report", line 3, in <module>
    from deepspeed.env_report import cli_main
  File "/opt/conda/envs/core/lib/python3.10/site-packages/deepspeed/__init__.py", line 16, in <module>
    from . import module_inject
  File "/opt/conda/envs/core/lib/python3.10/site-packages/deepspeed/module_inject/__init__.py", line 6, in <module>
    from .replace_module import replace_transformer_layer, revert_transformer_layer, ReplaceWithTensorSlicing, GroupQuantizer, generic_injection
  File "/opt/conda/envs/core/lib/python3.10/site-packages/deepspeed/module_inject/replace_module.py", line 792, in <module>
    from ..pipe import PipelineModule
  File "/opt/conda/envs/core/lib/python3.10/site-packages/deepspeed/pipe/__init__.py", line 6, in <module>
    from ..runtime.pipe import PipelineModule, LayerSpec, TiedLayerSpec
  File "/opt/conda/envs/core/lib/python3.10/site-packages/deepspeed/runtime/pipe/__init__.py", line 6, in <module>
    from .module import PipelineModule, LayerSpec, TiedLayerSpec
  File "/opt/conda/envs/core/lib/python3.10/site-packages/deepspeed/runtime/pipe/module.py", line 19, in <module>
    from ..activation_checkpointing import checkpointing
  File "/opt/conda/envs/core/lib/python3.10/site-packages/deepspeed/runtime/activation_checkpointing/checkpointing.py", line 25, in <module>
    from deepspeed.runtime.config import DeepSpeedConfig
  File "/opt/conda/envs/core/lib/python3.10/site-packages/deepspeed/runtime/config.py", line 29, in <module>
    from .zero.config import get_zero_config, ZeroStageEnum
  File "/opt/conda/envs/core/lib/python3.10/site-packages/deepspeed/runtime/zero/__init__.py", line 6, in <module>
    from .partition_parameters import ZeroParamType
  File "/opt/conda/envs/core/lib/python3.10/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 616, in <module>
    class Init(InsertPostInitMethodToModuleSubClasses):
  File "/opt/conda/envs/core/lib/python3.10/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 618, in Init
    param_persistence_threshold = get_config_default(DeepSpeedZeroConfig, "param_persistence_threshold")
  File "/opt/conda/envs/core/lib/python3.10/site-packages/deepspeed/runtime/config_utils.py", line 116, in get_config_default
    field_name).required, f"'{field_name}' is a required field and does not have a default value"
AttributeError: 'FieldInfo' object has no attribute 'required'. Did you mean: 'is_required'?

System info (please complete the following information):

• OS: [Ubuntu:22.04]
• GPU count and types: 1xv100
• Python version: 3.10.6

Launcher context
Are you launching your experiment with the deepspeed launcher, MPI, or something else?
Unable to launch

Docker context
Are you using a specific docker image that you can share?

Dockerfile:

# Builds GPU-compatible docker image (works on CPU too)
# Uses multi-staged approach to reduce size
FROM condaforge/mambaforge:latest AS compile-image

# Create a conda environment called `core` that will always be activated
COPY environment.yaml .
RUN mamba env create --name core --file environment.yaml && mamba clean -y --all -f

# Stage 2, allows us to select base OS
FROM ubuntu:22.04 AS build-image
COPY --from=compile-image /opt/conda /opt/conda

# Ensure conda/mamba can be called and pip uses the core environment
ENV PATH /opt/conda/envs/core/bin:/opt/conda/bin:$PATH

# Make bash the default shell
RUN chsh -s /bin/bash
SHELL ["/bin/bash", "-c"]

# Ensure `core` env is always activated
RUN echo "source activate core" >> ~/.bashrc

environment.yaml:

channels:
  - pytorch
  - nvidia
  - conda-forge
dependencies:
  - python=3.10.6
  - pandas=1.5.3
  - datasets=2.13.1
  - transformers=4.30.2
  - pytorch=2.0.0
  - gcc=13.1.0
  - pytorch-cuda=11.7
  - deepspeed=0.9.5
  - pip:
    - peft==0.3.0

Additional context
Stacktrace:

[2023-07-14 14:55:23,710] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/opt/conda/envs/core/lib/python3.10/site-packages/pydantic/_internal/_config.py:261: UserWarning: Valid config keys have changed in V2:
* 'allow_population_by_field_name' has been renamed to 'populate_by_name'
* 'validate_all' has been renamed to 'validate_default'
  warnings.warn(message, UserWarning)
/opt/conda/envs/core/lib/python3.10/site-packages/pydantic/_internal/_fields.py:126: UserWarning: Field "model_persistence_threshold" has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
  warnings.warn(
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/conda/envs/core/lib/python3.10/site-packages/deepspeed/__init__.py", line 16, in <module>
    from . import module_inject
  File "/opt/conda/envs/core/lib/python3.10/site-packages/deepspeed/module_inject/__init__.py", line 6, in <module>
    from .replace_module import replace_transformer_layer, revert_transformer_layer, ReplaceWithTensorSlicing, GroupQuantizer, generic_injection
  File "/opt/conda/envs/core/lib/python3.10/site-packages/deepspeed/module_inject/replace_module.py", line 792, in <module>
    from ..pipe import PipelineModule
  File "/opt/conda/envs/core/lib/python3.10/site-packages/deepspeed/pipe/__init__.py", line 6, in <module>
    from ..runtime.pipe import PipelineModule, LayerSpec, TiedLayerSpec
  File "/opt/conda/envs/core/lib/python3.10/site-packages/deepspeed/runtime/pipe/__init__.py", line 6, in <module>
    from .module import PipelineModule, LayerSpec, TiedLayerSpec
  File "/opt/conda/envs/core/lib/python3.10/site-packages/deepspeed/runtime/pipe/module.py", line 19, in <module>
    from ..activation_checkpointing import checkpointing
  File "/opt/conda/envs/core/lib/python3.10/site-packages/deepspeed/runtime/activation_checkpointing/checkpointing.py", line 25, in <module>
    from deepspeed.runtime.config import DeepSpeedConfig
  File "/opt/conda/envs/core/lib/python3.10/site-packages/deepspeed/runtime/config.py", line 29, in <module>
    from .zero.config import get_zero_config, ZeroStageEnum
  File "/opt/conda/envs/core/lib/python3.10/site-packages/deepspeed/runtime/zero/__init__.py", line 6, in <module>
    from .partition_parameters import ZeroParamType
  File "/opt/conda/envs/core/lib/python3.10/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 616, in <module>
    class Init(InsertPostInitMethodToModuleSubClasses):
  File "/opt/conda/envs/core/lib/python3.10/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 618, in Init
    param_persistence_threshold = get_config_default(DeepSpeedZeroConfig, "param_persistence_threshold")
  File "/opt/conda/envs/core/lib/python3.10/site-packages/deepspeed/runtime/config_utils.py", line 116, in get_config_default
    field_name).required, f"'{field_name}' is a required field and does not have a default value"
AttributeError: 'FieldInfo' object has no attribute 'required'. Did you mean: 'is_required'?

[loadams]

@Chris-hughes10 - what version of pydantic do you have installed in this environment? We aren't compatible with pydantic 2.0 yet, so we've pinned to <2.0.0

[ElliotYechanKim]

This is likely to be a problem with pydrantic as reported by #3280 . This can be resolved by downgrading your pydrantic version or upgrading your deepspeed version(it seems like they solved it in #3290 )

[loadams]

Hi @Chris-hughes10 - were you able to confirm if this was due to the pydantic version in the environment?

[loadams]

Hi @Chris-hughes10 - closing this for now, I suspect it is the pydantic issue. If that doesn't fix it, please re-open the issue and let me know.

[mohman313]

I had the same issue and by downgrading the Pydantic version, it was resolved!

[loadams]

Thanks for confirming @mohman313. We should have this fixed so we support pydantic 2+ shortly.

[chintan-ushur]

Agreed, downgrading the pydantic version helped me as well.

[gshimansky]

Is it possible to correct conda-forge deepspeed build to reflect requirement of pydantic<2.0.0? Currently conda isntalls pydantic 2.3.0 as a dependency of deepspeed and it is necessary to google the exception to find this bug report and figure that version 2.3.0 is incompatible.

[loadams]

Hi @gshimansky - we don't publish those packages, I believe @hadim is the maintainer of this package from here: https://github.com/conda-forge/deepspeed-feedstock.

Meanwhile, we are updating the yml that we maintain in our repo here if you are able to test this to confirm, thanks!

[gshimansky]

Ok create a bug conda-forge/deepspeed-feedstock#23

[kwanhoP]

in my case, i have a another function that use latest version of pydantic. So, solved it upgrade the deepspeed.

pydantic 2.6.*
deepspeed 0.13*