Cluster setup for client fails on requirements install

[tarockey]

When attempting to run the command pip3 install "git+https://github.com/Azure/feast-azure.git#subdirecto ry=cluster/sdk/python" on a new Azure Ubuntu VM, the pip install fails with the attached error messages.

pip error.txt
.

[tarockey]

I tried manually installing protobuf, and the error at the end changed to:

` ERROR: setuptools==39.0.1 is used in combination with setuptools_scm>=6.x

Your build configuration is incomplete and previously worked by accident!


This happens as setuptools is unable to replace itself when a activated build dependency
requires a more recent setuptools version
(it does not respect "setuptools>X" in setup_requires).


setuptools>=31 is required for setup.cfg metadata support
setuptools>=42 is required for pyproject.toml configuration support

Suggested workarounds if applicable:
 - preinstalling build dependencies like setuptools_scm before running setup.py
 - installing setuptools_scm using the system package manager to ensure consistency
 - migrating from the deprecated setup_requires mechanism to pep517/518
   and using a pyproject.toml to declare build dependencies
   which are reliably pre-installed before running the build tools

  """
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/pip-4iipkrtg-build/cluster/sdk/python/setup.py", line 161, in <module>
    "develop": DevelopCommand,
  File "/usr/lib/python3/dist-packages/setuptools/__init__.py", line 129, in setup
    return distutils.core.setup(**attrs)
  File "/usr/lib/python3.6/distutils/core.py", line 108, in setup
    _setup_distribution = dist = klass(attrs)
  File "/usr/lib/python3/dist-packages/setuptools/dist.py", line 372, in __init__
    _Distribution.__init__(self, attrs)
  File "/usr/lib/python3.6/distutils/dist.py", line 281, in __init__
    self.finalize_options()
  File "/usr/lib/python3/dist-packages/setuptools/dist.py", line 528, in finalize_options
    ep.load()(self, ep.name, value)
  File "/tmp/pip-4iipkrtg-build/cluster/sdk/python/.eggs/setuptools_scm-6.3.2-py3.6.egg/setuptools_scm/integration.py", line 65, in version_keyword
    dist.metadata.version = _get_version(config)
  File "/tmp/pip-4iipkrtg-build/cluster/sdk/python/.eggs/setuptools_scm-6.3.2-py3.6.egg/setuptools_scm/__init__.py", line 177, in _get_version
    parsed_version = _do_parse(config)
  File "/tmp/pip-4iipkrtg-build/cluster/sdk/python/.eggs/setuptools_scm-6.3.2-py3.6.egg/setuptools_scm/__init__.py", line 143, in _do_parse
    "use git+https://github.com/user/proj.git#egg=proj" % config.absolute_root
LookupError: setuptools-scm was unable to detect version for '/tmp/pip-4iipkrtg-build/cluster/sdk'.

Make sure you're either building from a fully intact git repository or PyPI tarballs. Most other sources (such as GitHub's tarballs, a git checkout without the .git folder) don't contain the necessary metadata and will not work.

For example, if you're using pip, instead of https://github.com/user/proj/archive/master.zip use git+https://github.com/user/proj.git#egg=proj

`

[tarockey]

Update since the latest commit to the cluster folder, the error has changed.
Now when trying to run pip install "git+https://github.com/Azure/feast-azure.git#subdirectory=cluster/sdk/python", the output is:

Collecting git+https://github.com/Azure/feast-azure.git#subdirectory=cluster/sdk/python
  Cloning https://github.com/Azure/feast-azure.git to /tmp/pip-req-build-l8fwresa
  Running command git clone --filter=blob:none -q https://github.com/Azure/feast-azure.git /tmp/pip-req-build-l8fwresa
  Resolved https://github.com/Azure/feast-azure.git to commit 2f87d2a891827a36861ab558532ce6925899c516
  Running command git submodule update --init --recursive -q
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/azureuser/feastenv/bin/python /home/azureuser/feastenv/lib/python3.6/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpzf558e0d
       cwd: /tmp/pip-req-build-l8fwresa/cluster/sdk/python
  Complete output (39 lines):
  WARNING: '.' not a valid package name; please use only .-separated package names in setup.py
  Traceback (most recent call last):
    File "/home/azureuser/feastenv/lib/python3.6/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
      main()
    File "/home/azureuser/feastenv/lib/python3.6/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/home/azureuser/feastenv/lib/python3.6/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 164, in prepare_metadata_for_build_wheel
      return hook(metadata_directory, config_settings)
    File "/tmp/pip-build-env-sfa19zun/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 174, in prepare_metadata_for_build_wheel
      self.run_setup()
    File "/tmp/pip-build-env-sfa19zun/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 268, in run_setup
      self).run_setup(setup_script=setup_script)
    File "/tmp/pip-build-env-sfa19zun/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 158, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 161, in <module>
      "develop": DevelopCommand,
    File "/tmp/pip-build-env-sfa19zun/overlay/lib/python3.6/site-packages/setuptools/__init__.py", line 153, in setup
      return distutils.core.setup(**attrs)
    File "/usr/lib/python3.6/distutils/core.py", line 108, in setup
      _setup_distribution = dist = klass(attrs)
    File "/tmp/pip-build-env-sfa19zun/overlay/lib/python3.6/site-packages/setuptools/dist.py", line 457, in __init__
      for k, v in attrs.items()
    File "/usr/lib/python3.6/distutils/dist.py", line 281, in __init__
      self.finalize_options()
    File "/tmp/pip-build-env-sfa19zun/overlay/lib/python3.6/site-packages/setuptools/dist.py", line 831, in finalize_options
      ep(self)
    File "/tmp/pip-build-env-sfa19zun/overlay/lib/python3.6/site-packages/setuptools/dist.py", line 852, in _finalize_setup_keywords
      ep.load()(self, ep.name, value)
    File "/tmp/pip-build-env-sfa19zun/normal/lib/python3.6/site-packages/setuptools_scm/integration.py", line 65, in version_keyword
      dist.metadata.version = _get_version(config)
    File "/tmp/pip-build-env-sfa19zun/normal/lib/python3.6/site-packages/setuptools_scm/__init__.py", line 177, in _get_version
      parsed_version = _do_parse(config)
    File "/tmp/pip-build-env-sfa19zun/normal/lib/python3.6/site-packages/setuptools_scm/__init__.py", line 143, in _do_parse
      "use git+https://github.com/user/proj.git#egg=proj" % config.absolute_root
  LookupError: setuptools-scm was unable to detect version for '/tmp/pip-req-build-l8fwresa/cluster/sdk'.

  Make sure you're either building from a fully intact git repository or PyPI tarballs. Most other sources (such as GitHub's tarballs, a git checkout without the .git folder) don't contain the necessary metadata and will not work.

  For example, if you're using pip, instead of https://github.com/user/proj/archive/master.zip use git+https://github.com/user/proj.git#egg=proj
  ----------------------------------------
WARNING: Discarding git+https://github.com/Azure/feast-azure.git#subdirectory=cluster/sdk/python. Command errored out with exit status 1: /home/azureuser/feastenv/bin/python /home/azureuser/feastenv/lib/python3.6/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpzf558e0d Check the logs for full command output.
ERROR: Command errored out with exit status 1: /home/azureuser/feastenv/bin/python /home/azureuser/feastenv/lib/python3.6/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpzf558e0d Check the logs for full command output.

[xiaoyongzhumsft]

@tarockey thanks for trying out! Looks like for some reason it's not working. Can you try this command? I just verified with Python 3.8.
pip install avro confluent_kafka git+https://github.com/xiaoyongzhu/feast-spark.git#subdirectory=python

[nithin-nk]

@snowmanmsft It's still failing. How to install the feast SDK?

[delta0123]

@snowmanmsft tried the above command but the issue still exists with the SDK files.

[xiaoyongzhumsft]

@nithinkanil1 @delta0123
Can you try to install using this command with Python 3.8/Linux? I just tested it should work:

pip install git+https://github.com/xiaoyongzhu/feast-spark.git#subdirectory=python

Also I'm trying to fix the above issue for other python versions. Looks like it's a dependency issue.

[nithin-nk]

@snowmanmsft I tried to install using python 3.8. I am getting below mentioned error

    Installed /tmp/pip-cpm3nhjn-build/python/.eggs/cachetools-4.2.4-py3.8.egg
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-cpm3nhjn-build/python/setup.py", line 129, in <module>
        setup(
      File "/usr/lib/python3/dist-packages/setuptools/__init__.py", line 128, in setup
        _install_setup_requires(attrs)
      File "/usr/lib/python3/dist-packages/setuptools/__init__.py", line 123, in _install_setup_requires
        dist.fetch_build_eggs(dist.setup_requires)
      File "/usr/lib/python3/dist-packages/setuptools/dist.py", line 510, in fetch_build_eggs
        resolved_dists = pkg_resources.working_set.resolve(
      File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 783, in resolve
        raise VersionConflict(dist, req).with_context(dependent_req)
    pkg_resources.ContextualVersionConflict: (protobuf 4.0.0rc2 (/tmp/pip-cpm3nhjn-build/python/.eggs/protobuf-4.0.0rc2-py3.8.egg), Requirement.parse('protobu
f<4.0dev,>=3.5.0.post1'), {'grpcio-tools'})

[xiaoyongzhumsft]

@snowmanmsft I tried to install using python 3.8. I am getting below mentioned error

    Installed /tmp/pip-cpm3nhjn-build/python/.eggs/cachetools-4.2.4-py3.8.egg
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-cpm3nhjn-build/python/setup.py", line 129, in <module>
        setup(
      File "/usr/lib/python3/dist-packages/setuptools/__init__.py", line 128, in setup
        _install_setup_requires(attrs)
      File "/usr/lib/python3/dist-packages/setuptools/__init__.py", line 123, in _install_setup_requires
        dist.fetch_build_eggs(dist.setup_requires)
      File "/usr/lib/python3/dist-packages/setuptools/dist.py", line 510, in fetch_build_eggs
        resolved_dists = pkg_resources.working_set.resolve(
      File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 783, in resolve
        raise VersionConflict(dist, req).with_context(dependent_req)
    pkg_resources.ContextualVersionConflict: (protobuf 4.0.0rc2 (/tmp/pip-cpm3nhjn-build/python/.eggs/protobuf-4.0.0rc2-py3.8.egg), Requirement.parse('protobu
f<4.0dev,>=3.5.0.post1'), {'grpcio-tools'})

hmm looks like you have some environment conflict. Do you mind creating a new python environment (with conda or virtual env) and give it another shot?

[delta0123]

@snowmanmsft I had followed below steps:
install conda: https://docs.conda.io/en/latest/miniconda.html
->conda create -n azure_38 python=3.8
->conda activate azure_38
->pip install git+https://github.com/xiaoyongzhu/feast-spark.git#subdirectory=python

and then edited test_featurestore.py with Ip address, but later when executed : python3 test_featurestore.py, got following error:

Ip used: 10.10.151.15:6565
Traceback (most recent call last):
File "/home/produser/softwares/anaconda3/envs/azure_38/lib/python3.8/site-packages/feast/grpc/grpc.py", line 48, in create_grpc_channel
grpc.channel_ready_future(channel).result(timeout=timeout)
File "/home/produser/softwares/anaconda3/envs/azure_38/lib/python3.8/site-packages/grpc/_utilities.py", line 140, in result
self._block(timeout)
File "/home/produser/softwares/anaconda3/envs/azure_38/lib/python3.8/site-packages/grpc/_utilities.py", line 86, in _block
raise grpc.FutureTimeoutError()
grpc.FutureTimeoutError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "test_featurestore.py", line 71, in
client.apply(driver_id)
File "/home/produser/softwares/anaconda3/envs/azure_38/lib/python3.8/site-packages/feast/client.py", line 516, in apply
log_usage("apply", self._telemetry_id, datetime.utcnow(), self.version())
File "/home/produser/softwares/anaconda3/envs/azure_38/lib/python3.8/site-packages/feast/client.py", line 372, in version
serving_version = self._serving_service.GetFeastServingInfo(
File "/home/produser/softwares/anaconda3/envs/azure_38/lib/python3.8/site-packages/feast/client.py", line 184, in _serving_service
channel = create_grpc_channel(
File "/home/produser/softwares/anaconda3/envs/azure_38/lib/python3.8/site-packages/feast/grpc/grpc.py", line 51, in create_grpc_channel
raise ConnectionError(
ConnectionError: Connection timed out while attempting to connect to localhost:6566

[delta0123]

@snowmanmsft
=>when tried executing python3 test_featurestore.py with (azure_38) using conda activate azure_38 following error is displayed:

Traceback (most recent call last):
File "test_featurestore.py", line 2, in
from feast import Client
ImportError: cannot import name 'Client' from 'feast' (/home/produser/softwares/anaconda3/envs/azure_38/lib/python3.8/site-packages/feast/init.py)

=>and when tired executing python3 test_featurestore.py with (base) following error was displayed:

Ip used: 10.10.151.7:6565
Traceback (most recent call last):
File "/home/produser/softwares/anaconda3/lib/python3.7/site-packages/feast/grpc/grpc.py", line 48, in create_grpc_channel
grpc.channel_ready_future(channel).result(timeout=timeout)
File "/home/produser/softwares/anaconda3/lib/python3.7/site-packages/grpc/_utilities.py", line 139, in result
self._block(timeout)
File "/home/produser/softwares/anaconda3/lib/python3.7/site-packages/grpc/_utilities.py", line 85, in _block
raise grpc.FutureTimeoutError()
grpc.FutureTimeoutError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "test_featurestore.py", line 14, in
client.list_feature_tables()
File "/home/produser/softwares/anaconda3/lib/python3.7/site-packages/feast/client.py", line 716, in list_feature_tables
feature_table_protos = self._core_service.ListFeatureTables(
File "/home/produser/softwares/anaconda3/lib/python3.7/site-packages/feast/client.py", line 169, in _core_service
timeout=self._config.getint(opt.GRPC_CONNECTION_TIMEOUT),
File "/home/produser/softwares/anaconda3/lib/python3.7/site-packages/feast/grpc/grpc.py", line 52, in create_grpc_channel
f"Connection timed out while attempting to connect to {url}"
ConnectionError: Connection timed out while attempting to connect to 10.10.151.7:6565

[Rakesh-Raushan]

@delta0123 were you able to get it resolved? I mean ConnectionError? I'm facing this exact error