[fast-reboot] Orchagent crash after "addFdbEntry: Failed to create dynamic FDB"

[vaibhavhd]

Description

Orchagent crashes as FDB was learnt before it could be programmed.

SAI version: 5.0.0.6-1

Steps to reproduce the issue:

1. Run test platform_tests/test_advanced_reboot.py::test_fast_reboot
2. The test will fail with "FAILED:dut:DUT hasn't booted up in 300 seconds",
3. Check syslog for real errors.

Describe the results you received:

Jul 31 00:58:14.503729 str2-7050cx3-acs-01 NOTICE admin: Enabling Watchdog before fast-reboot
Jul 31 00:58:14.834209 str2-7050cx3-acs-01 NOTICE admin: Rebooting with /sbin/kexec -e to SONiC-OS-master.25538-5e435e05a ...

Jul 31 00:59:20.308026 str2-7050cx3-acs-01 ERR swss#orchagent: :- meta_sai_validate_fdb_entry: object key SAI_OBJECT_TYPE_FDB_ENTRY:{"bvid":"oid:0x2600000000063d","mac":"72:06:00:01:00:00","switch_id":"oid:0x21000000000000"} already exists
Jul 31 00:59:20.308026 str2-7050cx3-acs-01 ERR swss#orchagent: :- addFdbEntry: Failed to create dynamic FDB 72:06:00:01:00:00 in Vlan1000 on Ethernet12, rv:-6
Jul 31 00:59:20.308026 str2-7050cx3-acs-01 ERR swss#orchagent: :- handleSaiCreateStatus: Encountered failure in create operation, exiting orchagent, SAI API: SAI_API_FDB, status: SAI_STATUS_ITEM_ALREADY_EXISTS
Jul 31 00:59:20.308426 str2-7050cx3-acs-01 NOTICE swss#orchagent: :- uninitialize: begin
Jul 31 00:59:20.308426 str2-7050cx3-acs-01 NOTICE swss#orchagent: :- uninitialize: begin
Jul 31 00:59:20.308465 str2-7050cx3-acs-01 NOTICE swss#orchagent: :- uninitialize: end server thread begin
Jul 31 00:59:20.308630 str2-7050cx3-acs-01 NOTICE swss#orchagent: :- serverThreadFunction: end
Jul 31 00:59:20.308630 str2-7050cx3-acs-01 NOTICE swss#orchagent: :- uninitialize: end server thread end
Jul 31 00:59:20.308630 str2-7050cx3-acs-01 NOTICE swss#orchagent: :- uninitialize: begin
Jul 31 00:59:20.308770 str2-7050cx3-acs-01 NOTICE swss#orchagent: :- uninitialize: begin
Jul 31 00:59:20.308825 str2-7050cx3-acs-01 NOTICE swss#orchagent: :- ~RedisChannel: join ntf thread begin
Jul 31 00:59:20.308853 str2-7050cx3-acs-01 NOTICE swss#orchagent: :- ~RedisChannel: join ntf thread end
Jul 31 00:59:20.308911 str2-7050cx3-acs-01 NOTICE swss#orchagent: :- clear_local_state: clearing local state
Jul 31 00:59:20.308911 str2-7050cx3-acs-01 NOTICE swss#orchagent: :- meta_init_db: begin
Jul 31 00:59:20.312134 str2-7050cx3-acs-01 NOTICE swss#orchagent: :- meta_init_db: end
Jul 31 00:59:20.312134 str2-7050cx3-acs-01 NOTICE swss#orchagent: :- uninitialize: end
Jul 31 00:59:20.312246 str2-7050cx3-acs-01 NOTICE swss#orchagent: :- stopRecording: stopped recording
Jul 31 00:59:20.312318 str2-7050cx3-acs-01 NOTICE swss#orchagent: :- stopRecording: closed recording file: sairedis.rec
Jul 31 00:59:20.312386 str2-7050cx3-acs-01 NOTICE swss#orchagent: :- uninitialize: end
Jul 31 00:59:20.312671 str2-7050cx3-acs-01 NOTICE swss#orchagent: :- uninitialize: end
Jul 31 00:59:20.312671 str2-7050cx3-acs-01 NOTICE swss#orchagent: :- zmqPollThread: ending pool thread, since run is false
Jul 31 00:59:20.312699 str2-7050cx3-acs-01 NOTICE swss#orchagent: :- zmqPollThread: end
Jul 31 00:59:20.312721 str2-7050cx3-acs-01 NOTICE swss#orchagent: :- ~ZeroMQSelectableChannel: ending zmq poll thread for channel ipc:///tmp/saiServer
Jul 31 00:59:20.312778 str2-7050cx3-acs-01 NOTICE swss#orchagent: :- ~ZeroMQSelectableChannel: ended zmq poll thread for channel ipc:///tmp/saiServer
Jul 31 00:59:20.313074 str2-7050cx3-acs-01 NOTICE swss#orchagent: :- uninitialize: end

Describe the results you expected:

Output of show version:

SONiC Software Version: SONiC.master.25538-5e435e05a
Distribution: Debian 10.10
Kernel: 4.19.0-12-2-amd64
Build commit: 5e435e05a
Build date: Mon Jul 26 14:39:47 UTC 2021
Built by: AzDevOps@sonic-build-workers-000J8D

Platform: x86_64-arista_7050cx3_32s
HwSKU: Arista-7050CX3-32S-C32
ASIC: broadcom
ASIC Count: 1
Serial Number: JPE20432334
Model Number: DCS-7050CX3-32S-SSD
Hardware Revision: N/A
Uptime: 01:22:33 up 24 min,  1 user,  load average: 0.46, 0.40, 0.31

# docker exec -it syncd dpkg -s libsaibcm | head
Package: libsaibcm
Status: install ok installed
Priority: extra
Section: libs
Installed-Size: 245525
Maintainer: Guohan Lu <[email protected]>
Architecture: amd64
Source: saibcm
Version: 5.0.0.6-1
Provides: libsai

Additional information you deem important (e.g. issue happens only occasionally):

[vaibhavhd]

This is same as an older issue that was fixed:

The fix (sonic-net/sonic-swss#1815) is still not part of master branch.

Updating the master branch head with this PR should fix this issue: #8313