Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[tlm_teamd]: sporadic segfault is observed during config reload #5433

Closed
nazariig opened this issue Sep 22, 2020 · 1 comment · Fixed by sonic-net/sonic-swss#1490
Closed

[tlm_teamd]: sporadic segfault is observed during config reload #5433

nazariig opened this issue Sep 22, 2020 · 1 comment · Fixed by sonic-net/sonic-swss#1490
Labels
Bug 🐛 team 👭

Comments

@nazariig
Copy link
Collaborator

Description
Sporadic segfault in tlm_teamd is observed during config reload:

root@sonic:/home/admin# show int po
Flags: A - active, I - inactive, Up - up, Dw - Down, N/A - not available,
       S - selected, D - deselected, * - not synced
  No.  Team Dev         Protocol    Ports
-----  ---------------  ----------  -------
 0002  PortChannel0002  N/A
 0005  PortChannel0005  N/A
 0008  PortChannel0008  N/A
 0011  PortChannel0011  N/A
 0014  PortChannel0014  N/A
 0017  PortChannel0017  N/A
 0020  PortChannel0020  N/A
 0023  PortChannel0023  N/A

Steps to reproduce the issue:

  1. config reload -y

Describe the results you received:

Sep 22 18:16:52.902251 arc-switch1025 NOTICE teamd#teammgrd: :- addLag: Start port channel PortChannel0023 with teamd
Sep 22 18:16:52.945422 arc-switch1025 NOTICE teamd#teammgrd: :- setLagAdminStatus: Set port channel PortChannel0023 admin status to up
Sep 22 18:16:52.962378 arc-switch1025 NOTICE teamd#teammgrd: :- setLagMtu: Set port channel PortChannel0023 MTU to 9100
Sep 22 18:16:53.059705 arc-switch1025 NOTICE teamd#teammgrd: :- addLagMember: Add Ethernet0 to port channel PortChannel0002
Sep 22 18:16:53.153798 arc-switch1025 NOTICE teamd#teammgrd: :- addLagMember: Add Ethernet4 to port channel PortChannel0002
Sep 22 18:16:53.237006 arc-switch1025 NOTICE teamd#teammgrd: :- addLagMember: Add Ethernet12 to port channel PortChannel0005
Sep 22 18:16:53.320528 arc-switch1025 NOTICE teamd#teammgrd: :- addLagMember: Add Ethernet8 to port channel PortChannel0005
Sep 22 18:16:53.428405 arc-switch1025 NOTICE teamd#teammgrd: :- addLagMember: Add Ethernet16 to port channel PortChannel0008
Sep 22 18:16:53.556186 arc-switch1025 NOTICE teamd#teammgrd: :- addLagMember: Add Ethernet20 to port channel PortChannel0008
Sep 22 18:16:53.654836 arc-switch1025 NOTICE teamd#teammgrd: :- addLagMember: Add Ethernet24 to port channel PortChannel0011
Sep 22 18:16:53.769752 arc-switch1025 NOTICE teamd#teammgrd: :- addLagMember: Add Ethernet28 to port channel PortChannel0011
Sep 22 18:16:53.866892 arc-switch1025 NOTICE teamd#teammgrd: :- addLagMember: Add Ethernet32 to port channel PortChannel0014
Sep 22 18:16:53.868438 arc-switch1025 INFO kernel: [11286.555858] tlm_teamd[10276]: segfault at 0 ip 0000559d266bab58 sp 00007fffbbb02e90 error 4 in tlm_teamd[559d266b7000+9000]
Sep 22 18:16:54.063472 arc-switch1025 NOTICE teamd#teammgrd: :- addLagMember: Add Ethernet36 to port channel PortChannel0014
Sep 22 18:16:54.246999 arc-switch1025 NOTICE teamd#teammgrd: :- addLagMember: Add Ethernet40 to port channel PortChannel0017
Sep 22 18:16:54.365263 arc-switch1025 NOTICE teamd#teammgrd: :- addLagMember: Add Ethernet44 to port channel PortChannel0017
Sep 22 18:16:54.488597 arc-switch1025 NOTICE teamd#teammgrd: :- addLagMember: Add Ethernet48 to port channel PortChannel0020
Sep 22 18:16:54.590859 arc-switch1025 NOTICE teamd#teammgrd: :- addLagMember: Add Ethernet52 to port channel PortChannel0020
Sep 22 18:16:54.715252 arc-switch1025 NOTICE teamd#teammgrd: :- addLagMember: Add Ethernet56 to port channel PortChannel0023
Sep 22 18:16:54.837019 arc-switch1025 NOTICE teamd#teammgrd: :- addLagMember: Add Ethernet60 to port channel PortChannel0023

Describe the results you expected:
No segmentation fault is expected

Additional information you deem important (e.g. issue happens only occasionally):

Output of show version:

SONiC Software Version: SONiC.master.411-03f82a01
Distribution: Debian 10.5
Kernel: 4.19.0-9-2-amd64
Build commit: 03f82a01
Build date: Tue Sep 22 09:49:47 UTC 2020
Built by: johnar@jenkins-worker-7

Platform: x86_64-mlnx_msn2700-r0
HwSKU: ACS-MSN2700
ASIC: mellanox

Attach debug file sudo generate_dump:
sonic_dump_arc-switch1025_20200922_182305.tar.gz
@abdosi
Copy link
Contributor

abdosi commented Sep 30, 2020

can you check this issue with PR sonic-net/sonic-swss#1450 ?
we need to still update submodule

@pavel-shirshov pavel-shirshov mentioned this issue Oct 30, 2020
Merged
lguohan pushed a commit to sonic-net/sonic-swss that referenced this issue Nov 1, 2020
@pavel-shirshov
[tlm_teamd]: Make the destionation for std::transform() to use std::b…
d7643f2 
…ack_inserter() for allocating new space for the copied objects (#1490)

Fixes: sonic-net/sonic-buildimage#5755
Fixes: sonic-net/sonic-buildimage#5433

Make the destination for std::transform() to use std::back_inserter() for allocating new space for the copied objects
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug 🐛 team 👭
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[tlm_teamd]: Fix segfault, when retrying to do teamd_connect() again pavel-shirshov/sonic-swss
2 participants
@nazariig @abdosi