Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[multi-asic]: load minigraph is failing on multi-asic vs with latest master branch image #11269

Closed
SuvarnaMeenakshi opened this issue Jun 28, 2022 · 3 comments

Comments

@SuvarnaMeenakshi
Copy link
Contributor

Description

The multi-asic image from Jun19th does not see any issue: https://dev.azure.com/mssonic/build/_build/results?buildId=112200&view=results
The multi-asic image from Jun 22 has this issue https://dev.azure.com/mssonic/build/_build/results?buildId=113368&view=results

Steps to reproduce the issue:

  1. Take sonic-4asic-vs.img.gz from https://dev.azure.com/mssonic/build/_build/results?buildId=113368&view=results pipeline, copy this as sonic-vs.img.gz if ~/sonic-vm/images/
  2. Bring up 4-asic vs testbed cd /data/sonic-mgmt/ansible
    ./testbed-cli.sh -t vtestbed.yaml -m veos_vtb -k ceos add-topo vms-kvm-four-asic-t1-lag password.txt
  3. deploy-mg:
  4. ./testbed-cli.sh -t vtestbed.yaml -m veos_vtb deploy-mg vms-kvm-four-asic-t1-lag veos_vtb password.txt

Describe the results you received:

sudo config load_minigraph -y
Stopping SONiC target ...
Running command: /usr/local/bin/sonic-cfggen -H -m -j /etc/sonic/init_cfg.json   --write-to-db
Running command: /usr/local/bin/sonic-cfggen -H -m -j /etc/sonic/init_cfg.json  -n asic0 --write-to-db
Warning: Failed to get device ID from asic.conf file for asic0
Running command: /usr/local/bin/sonic-cfggen -H -m -j /etc/sonic/init_cfg.json  -n asic1 --write-to-db
Warning: Failed to get device ID from asic.conf file for asic1
Running command: /usr/local/bin/sonic-cfggen -H -m -j /etc/sonic/init_cfg.json  -n asic2 --write-to-db
Warning: Failed to get device ID from asic.conf file for asic2
Running command: /usr/local/bin/sonic-cfggen -H -m -j /etc/sonic/init_cfg.json  -n asic3 --write-to-db
Warning: Failed to get device ID from asic.conf file for asic3
Running command: /usr/local/bin/sonic-cfggen -d -y /etc/sonic/sonic_version.yml -t /usr/share/sonic/templates/sonic-environment.j2,/etc/sonic/sonic-environment
Running command: config qos reload --no-dynamic-buffer
Running command: /usr/local/bin/sonic-cfggen -n asic0 -d --write-to-db -t /usr/share/sonic/device/x86_64-kvm_x86_64-r0/msft_four_asic_vs/0/buffers.json.j2,config-db -t /usr/share/sonic/device/x86_64-kvm_x86_64-r0/msft_four_asic_vs/0/qos.json.j2,config-db -y /etc/sonic/sonic_version.yml
Running command: /usr/local/bin/sonic-cfggen -n asic1 -d --write-to-db -t /usr/share/sonic/device/x86_64-kvm_x86_64-r0/msft_four_asic_vs/1/buffers.json.j2,config-db -t /usr/share/sonic/device/x86_64-kvm_x86_64-r0/msft_four_asic_vs/1/qos.json.j2,config-db -y /etc/sonic/sonic_version.yml
Running command: /usr/local/bin/sonic-cfggen -n asic2 -d --write-to-db -t /usr/share/sonic/device/x86_64-kvm_x86_64-r0/msft_four_asic_vs/2/buffers.json.j2,config-db -t /usr/share/sonic/device/x86_64-kvm_x86_64-r0/msft_four_asic_vs/2/qos.json.j2,config-db -y /etc/sonic/sonic_version.yml
Running command: /usr/local/bin/sonic-cfggen -n asic3 -d --write-to-db -t /usr/share/sonic/device/x86_64-kvm_x86_64-r0/msft_four_asic_vs/3/buffers.json.j2,config-db -t /usr/share/sonic/device/x86_64-kvm_x86_64-r0/msft_four_asic_vs/3/qos.json.j2,config-db -y /etc/sonic/sonic_version.yml
Running command: pfcwd start_default
Traceback (most recent call last):
  File "/usr/local/bin/pfcwd", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/click/decorators.py", line 57, in new_func
    obj = ctx.ensure_object(object_type)
  File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 477, in ensure_object
    self.obj = rv = object_type()
  File "/usr/local/lib/python3.9/dist-packages/utilities_common/db.py", line 36, in __init__
    multi_asic.connect_config_db_for_ns(ns)
  File "/usr/local/lib/python3.9/dist-packages/sonic_py_common/multi_asic.py", line 43, in connect_config_db_for_ns
    config_db = swsscommon.ConfigDBConnector(namespace=namespace)
  File "/usr/lib/python3/dist-packages/swsscommon/swsscommon.py", line 1945, in __init__
    super(ConfigDBConnector, self).__init__(use_unix_socket_path = use_unix_socket_path, namespace = namespace)
  File "/usr/lib/python3/dist-packages/swsscommon/swsscommon.py", line 1575, in __init__
    for db_name in self.get_db_list():
  File "/usr/lib/python3/dist-packages/swsscommon/swsscommon.py", line 1511, in get_db_list
    return _swsscommon.SonicV2Connector_Native_get_db_list(self)
RuntimeError: :- validateNamespace: Initialize global DB config using API SonicDBConfig::initializeGlobalConfig

Describe the results you expected:

config load_minigraph should go through without an error:
pfcwd start_default -- was not causing error earlier, but is showing issue now.

Output of show version:

(paste your output here)

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

@SuvarnaMeenakshi
Copy link
Contributor Author

SuvarnaMeenakshi commented Jun 28, 2022

Do not see this issue without #11177

@SuvarnaMeenakshi SuvarnaMeenakshi changed the title [mutli-asic]: load minigraph is failing on multi-asic vs with latest master branch image [multi-asic]: load minigraph is failing on multi-asic vs with latest master branch image Jun 28, 2022
@liuh-80
Copy link
Contributor

liuh-80 commented Jun 28, 2022

The issue may relate with this PR:
sonic-net/sonic-utilities#2183

This PR change InterfaceAliasConverter() to lazy initialize. and InterfaceAliasConverter() will try load DB config if it's not initialized.

Then pfcwd seems not initialize DB config by itself, so when InterfaceAliasConverter() change to lazy initialize, pfcwd breaked.

To fix this issue, pfcwd need initialize config with load_db_config() method in it's main() function.

qiluo-msft pushed a commit to sonic-net/sonic-utilities that referenced this issue Jul 4, 2022
#### What I did
Fix pfcwd connect DB with exception issue: sonic-net/sonic-buildimage#11269

pfcwd implicit depends on InterfaceAliasConverter() to initialize DB config, however following PR change InterfaceAliasConverter() behavior to lazy initialize, then pfcwd failed when try connect to DB without initialize DB config:

#2183

#### How I did it
Load  DB config in pfcwd.

#### How to verify it
Pass all UT.
@liuh-80
Copy link
Contributor

liuh-80 commented Jul 4, 2022

Close because fixed by this PR: sonic-net/sonic-utilities#2238

@liuh-80 liuh-80 closed this as completed Jul 4, 2022
gechiang pushed a commit to gechiang/sonic-utilities that referenced this issue Sep 14, 2022
#### What I did
Fix pfcwd connect DB with exception issue: sonic-net/sonic-buildimage#11269

pfcwd implicit depends on InterfaceAliasConverter() to initialize DB config, however following PR change InterfaceAliasConverter() behavior to lazy initialize, then pfcwd failed when try connect to DB without initialize DB config:

sonic-net#2183

#### How I did it
Load  DB config in pfcwd.

#### How to verify it
Pass all UT.
gechiang added a commit to sonic-net/sonic-utilities that referenced this issue Sep 15, 2022
#### What I did
Fix pfcwd connect DB with exception issue: sonic-net/sonic-buildimage#11269

pfcwd implicit depends on InterfaceAliasConverter() to initialize DB config, however following PR change InterfaceAliasConverter() behavior to lazy initialize, then pfcwd failed when try connect to DB without initialize DB config:

#2183

#### How I did it
Load  DB config in pfcwd.

#### How to verify it
Pass all UT.

Co-authored-by: Hua Liu <[email protected]>
malletvapid23 added a commit to malletvapid23/Sonic-Utility that referenced this issue Aug 3, 2023
#### What I did
Fix pfcwd connect DB with exception issue: sonic-net/sonic-buildimage#11269

pfcwd implicit depends on InterfaceAliasConverter() to initialize DB config, however following PR change InterfaceAliasConverter() behavior to lazy initialize, then pfcwd failed when try connect to DB without initialize DB config:

sonic-net/sonic-utilities#2183

#### How I did it
Load  DB config in pfcwd.

#### How to verify it
Pass all UT.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants